|
--- |
|
license: apache-2.0 |
|
library_name: transformers |
|
--- |
|
# Laser-Dolphin-Mixtral-2x7b-dpo |
|
|
|
![laser_dolphin_image](./dolphin_moe.png) |
|
|
|
**New Version will be uploaded soon** |
|
|
|
Credit to Fernando Fernandes and Eric Hartford for their project [laserRMT](https://github.com/cognitivecomputations/laserRMT) |
|
|
|
This model is a medium-sized MoE implementation based on [cognitivecomputations/dolphin-2.6-mistral-7b-dpo-laser](https://huggingface.co/cognitivecomputations/dolphin-2.6-mistral-7b-dpo-laser) |
|
|
|
A 2x7b configuration offers better performance than a standard 7b model even if loaded in 4 bit. (9G VRAM) |
|
|
|
If this 2x7b model is loaded in 4 bit the hellaswag score is .8270 which is higher than the base model achieves on its own in full precision. |
|
|
|
The process is outlined in this [notebook](https://github.com/cognitivecomputations/laserRMT/blob/main/examples/laser-dolphin-mixtral-2x7b.ipynb) |
|
|
|
**These Quants will result in unpredicted behavior and I am working on new Quants as I have updated the model** |
|
|
|
Quatizations provided by [TheBloke](https://huggingface.co/TheBloke/laser-dolphin-mixtral-2x7b-dpo-GGUF) |
|
|
|
|
|
|
|
## Code Example |
|
Switch the commented model definition to use in 4-bit. Should work with 9GB and still exceed the single 7B model by 5-6 points roughly |
|
|
|
```python |
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
|
def generate_response(prompt): |
|
""" |
|
Generate a response from the model based on the input prompt. |
|
|
|
Args: |
|
prompt (str): Prompt for the model. |
|
|
|
Returns: |
|
str: The generated response from the model. |
|
""" |
|
# Tokenize the input prompt |
|
inputs = tokenizer(prompt, return_tensors="pt") |
|
|
|
# Generate output tokens |
|
outputs = model.generate(**inputs, max_new_tokens=256, eos_token_id=tokenizer.eos_token_id, pad_token_id=tokenizer.pad_token_id) |
|
|
|
# Decode the generated tokens to a string |
|
response = tokenizer.decode(outputs[0], skip_special_tokens=True) |
|
|
|
return response |
|
|
|
# Load the model and tokenizer |
|
model_id = "macadeliccc/laser-dolphin-mixtral-2x7b-dpo" |
|
tokenizer = AutoTokenizer.from_pretrained(model_id) |
|
model = AutoModelForCausalLM.from_pretrained(model_id, load_in_4bit=True) |
|
|
|
prompt = "Write a quicksort algorithm in python" |
|
|
|
# Generate and print responses for each language |
|
print("Response:") |
|
print(generate_response(prompt), "\n") |
|
``` |
|
|
|
[colab](https://colab.research.google.com/drive/1cmRhAkDWItV7utHNqNANVZnqDqQNsTUr?usp=sharing) with usage example |
|
|
|
## Eval |
|
|
|
evaluation [colab](https://colab.research.google.com/drive/1FpwgsGzCR4tORTxAwUxpN3PcP22En2xk?usp=sharing) |
|
|
|
| Model |AGIEval|GPT4All|TruthfulQA|Bigbench|Average| |
|
|---------------------------------------------------------------------------------------------------|------:|------:|---------:|-------:|------:| |
|
|[laser-dolphin-mixtral-2x7b-dpo](https://huggingface.co/macadeliccc/laser-dolphin-mixtral-2x7b-dpo)| 41.31| 73.67| 61.69| 42.79| 54.87| |
|
|
|
### AGIEval |
|
| Task |Version| Metric |Value| |Stderr| |
|
|------------------------------|------:|--------|----:|---|-----:| |
|
|agieval_aqua_rat | 0|acc |22.44|± | 2.62| |
|
| | |acc_norm|21.26|± | 2.57| |
|
|agieval_logiqa_en | 0|acc |34.87|± | 1.87| |
|
| | |acc_norm|35.79|± | 1.88| |
|
|agieval_lsat_ar | 0|acc |22.17|± | 2.75| |
|
| | |acc_norm|23.04|± | 2.78| |
|
|agieval_lsat_lr | 0|acc |43.14|± | 2.20| |
|
| | |acc_norm|45.10|± | 2.21| |
|
|agieval_lsat_rc | 0|acc |57.25|± | 3.02| |
|
| | |acc_norm|55.76|± | 3.03| |
|
|agieval_sat_en | 0|acc |71.84|± | 3.14| |
|
| | |acc_norm|71.84|± | 3.14| |
|
|agieval_sat_en_without_passage| 0|acc |44.17|± | 3.47| |
|
| | |acc_norm|41.75|± | 3.44| |
|
|agieval_sat_math | 0|acc |40.91|± | 3.32| |
|
| | |acc_norm|35.91|± | 3.24| |
|
|
|
Average: 41.31% |
|
|
|
### GPT4All |
|
| Task |Version| Metric |Value| |Stderr| |
|
|-------------|------:|--------|----:|---|-----:| |
|
|arc_challenge| 0|acc |58.02|± | 1.44| |
|
| | |acc_norm|60.58|± | 1.43| |
|
|arc_easy | 0|acc |85.48|± | 0.72| |
|
| | |acc_norm|82.62|± | 0.78| |
|
|boolq | 1|acc |87.16|± | 0.59| |
|
|hellaswag | 0|acc |65.04|± | 0.48| |
|
| | |acc_norm|83.63|± | 0.37| |
|
|openbookqa | 0|acc |35.60|± | 2.14| |
|
| | |acc_norm|45.00|± | 2.23| |
|
|piqa | 0|acc |81.99|± | 0.90| |
|
| | |acc_norm|83.51|± | 0.87| |
|
|winogrande | 0|acc |73.16|± | 1.25| |
|
|
|
Average: 73.67% |
|
|
|
### TruthfulQA |
|
| Task |Version|Metric|Value| |Stderr| |
|
|-------------|------:|------|----:|---|-----:| |
|
|truthfulqa_mc| 1|mc1 |44.31|± | 1.74| |
|
| | |mc2 |61.69|± | 1.50| |
|
|
|
Average: 61.69% |
|
|
|
### Bigbench |
|
| Task |Version| Metric |Value| |Stderr| |
|
|------------------------------------------------|------:|---------------------|----:|---|-----:| |
|
|bigbench_causal_judgement | 0|multiple_choice_grade|59.47|± | 3.57| |
|
|bigbench_date_understanding | 0|multiple_choice_grade|66.67|± | 2.46| |
|
|bigbench_disambiguation_qa | 0|multiple_choice_grade|36.05|± | 3.00| |
|
|bigbench_geometric_shapes | 0|multiple_choice_grade|20.33|± | 2.13| |
|
| | |exact_str_match | 7.52|± | 1.39| |
|
|bigbench_logical_deduction_five_objects | 0|multiple_choice_grade|27.80|± | 2.01| |
|
|bigbench_logical_deduction_seven_objects | 0|multiple_choice_grade|19.86|± | 1.51| |
|
|bigbench_logical_deduction_three_objects | 0|multiple_choice_grade|48.67|± | 2.89| |
|
|bigbench_movie_recommendation | 0|multiple_choice_grade|49.60|± | 2.24| |
|
|bigbench_navigate | 0|multiple_choice_grade|53.20|± | 1.58| |
|
|bigbench_reasoning_about_colored_objects | 0|multiple_choice_grade|68.50|± | 1.04| |
|
|bigbench_ruin_names | 0|multiple_choice_grade|41.74|± | 2.33| |
|
|bigbench_salient_translation_error_detection | 0|multiple_choice_grade|16.23|± | 1.17| |
|
|bigbench_snarks | 0|multiple_choice_grade|64.09|± | 3.58| |
|
|bigbench_sports_understanding | 0|multiple_choice_grade|70.69|± | 1.45| |
|
|bigbench_temporal_sequences | 0|multiple_choice_grade|37.70|± | 1.53| |
|
|bigbench_tracking_shuffled_objects_five_objects | 0|multiple_choice_grade|23.44|± | 1.20| |
|
|bigbench_tracking_shuffled_objects_seven_objects| 0|multiple_choice_grade|17.60|± | 0.91| |
|
|bigbench_tracking_shuffled_objects_three_objects| 0|multiple_choice_grade|48.67|± | 2.89| |
|
|
|
Average: 42.79% |
|
|
|
Average score: 54.87% |
|
|
|
Elapsed time: 02:53:28 |
|
|
|
## Citations |
|
|
|
Fernando Fernandes Neto and Eric Hartford. "Optimizing Large Language Models Using Layer-Selective Rank Reduction and Random Matrix Theory." 2024. |
|
|
|
```bibtex |
|
@article{sharma2023truth, |
|
title={The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction}, |
|
author={Sharma, Pratyusha and Ash, Jordan T and Misra, Dipendra}, |
|
journal={arXiv preprint arXiv:2312.13558}, |
|
year={2023} } |
|
``` |
|
|
|
```bibtex |
|
@article{gao2021framework, |
|
title={A framework for few-shot language model evaluation}, |
|
author={Gao, Leo and Tow, Jonathan and Biderman, Stella and Black, Sid and DiPofi, Anthony and Foster, Charles and Golding, Laurence and Hsu, Jeffrey and McDonell, Kyle and Muennighoff, Niklas and others}, |
|
journal={Version v0. 0.1. Sept}, |
|
year={2021} |
|
} |
|
``` |