|
--- |
|
license: llama2 |
|
language: |
|
- en |
|
tags: |
|
- summary |
|
|
|
--- |
|
# Bubo Bubo 13B |
|
|
|
 |
|
|
|
# Prompting |
|
|
|
## Prompt Template for alpaca style |
|
|
|
``` |
|
### Instruction: |
|
|
|
<prompt> (without the <>) |
|
|
|
### Response: |
|
``` |
|
|
|
## Sample Code |
|
|
|
```python |
|
import torch |
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
|
torch.set_default_device("cuda") |
|
|
|
model = AutoModelForCausalLM.from_pretrained("ibivibiv/bubo-bubo-13b", torch_dtype="auto", device_config='auto') |
|
tokenizer = AutoTokenizer.from_pretrained("ibivibiv/bubo-bubo-13b") |
|
|
|
inputs = tokenizer("### Instruction: Summarize this email chain : <email chain stuff here>.\n### Response:\n", return_tensors="pt", return_attention_mask=False) |
|
|
|
outputs = model.generate(**inputs, max_length=200) |
|
text = tokenizer.batch_decode(outputs)[0] |
|
print(text) |
|
``` |
|
|
|
# Model Details |
|
* **Trained by**: [ibivibiv](https://huggingface.co/ibivibiv) |
|
* **Library**: [HuggingFace Transformers](https://github.com/huggingface/transformers) |
|
* **Model type:** **bubo-bubo-13b** is an auto-regressive language model fine tuned on the Llama 2 transformer architecture. |
|
* **Language(s)**: English |
|
* **Purpose**: Has specific training for summary tasks. This model is targeted towards summarizing communication chains specifically. |
|
|
|
# Benchmark Scores |
|
|
|
I ran the benchmark harness, for curiousity, but this model is completely geared towards summarizing. |
|
|
|
| Test Name | Accuracy | |
|
|------------------------------------------------------|----------------------| |
|
| all | 0.579149139810157 | |
|
| arc:challenge | 0.5631399317406144 | |
|
| hellaswag | 0.6317466640111532 | |
|
| hendrycksTest-abstract_algebra | 0.32 | |
|
| hendrycksTest-anatomy | 0.5481481481481482 | |
|
| hendrycksTest-astronomy | 0.5657894736842105 | |
|
| hendrycksTest-business_ethics | 0.55 | |
|
| hendrycksTest-clinical_knowledge | 0.6 | |
|
| hendrycksTest-college_biology | 0.6388888888888888 | |
|
| hendrycksTest-college_chemistry | 0.38 | |
|
| hendrycksTest-college_computer_science | 0.43 | |
|
| hendrycksTest-college_mathematics | 0.34 | |
|
| hendrycksTest-college_medicine | 0.5260115606936416 | |
|
| hendrycksTest-college_physics | 0.3431372549019608 | |
|
| hendrycksTest-computer_security | 0.71 | |
|
| hendrycksTest-conceptual_physics | 0.49361702127659574 | |
|
| hendrycksTest-econometrics | 0.35964912280701755 | |
|
| hendrycksTest-electrical_engineering | 0.5586206896551724 | |
|
| hendrycksTest-elementary_mathematics | 0.3439153439153439 | |
|
| hendrycksTest-formal_logic | 0.3333333333333333 | |
|
| hendrycksTest-global_facts | 0.42 | |
|
| hendrycksTest-high_school_biology | 0.6903225806451613 | |
|
| hendrycksTest-high_school_chemistry | 0.45320197044334976 | |
|
| hendrycksTest-high_school_computer_science | 0.58 | |
|
| hendrycksTest-high_school_european_history | 0.6787878787878788 | |
|
| hendrycksTest-high_school_geography | 0.7424242424242424 | |
|
| hendrycksTest-high_school_government_and_politics | 0.8341968911917098 | |
|
| hendrycksTest-high_school_macroeconomics | 0.558974358974359 | |
|
| hendrycksTest-high_school_mathematics | 0.3 | |
|
| hendrycksTest-high_school_microeconomics | 0.5672268907563025 | |
|
| hendrycksTest-high_school_physics | 0.33112582781456956 | |
|
| hendrycksTest-high_school_psychology | 0.7577981651376147 | |
|
| hendrycksTest-high_school_statistics | 0.4212962962962963 | |
|
| hendrycksTest-high_school_us_history | 0.8186274509803921 | |
|
| hendrycksTest-high_school_world_history | 0.759493670886076 | |
|
| hendrycksTest-human_aging | 0.6547085201793722 | |
|
| hendrycksTest-human_sexuality | 0.6412213740458015 | |
|
| hendrycksTest-international_law | 0.6776859504132231 | |
|
| hendrycksTest-jurisprudence | 0.75 | |
|
| hendrycksTest-logical_fallacies | 0.6993865030674846 | |
|
| hendrycksTest-machine_learning | 0.41964285714285715 | |
|
| hendrycksTest-management | 0.7281553398058253 | |
|
| hendrycksTest-marketing | 0.8504273504273504 | |
|
| hendrycksTest-medical_genetics | 0.6 | |
|
| hendrycksTest-miscellaneous | 0.7624521072796935 | |
|
| hendrycksTest-moral_disputes | 0.6560693641618497 | |
|
| hendrycksTest-moral_scenarios | 0.4346368715083799 | |
|
| hendrycksTest-nutrition | 0.673202614379085 | |
|
| hendrycksTest-philosophy | 0.7009646302250804 | |
|
| hendrycksTest-prehistory | 0.7067901234567902 | |
|
| hendrycksTest-professional_accounting | 0.4645390070921986 | |
|
| hendrycksTest-professional_law | 0.45697522816166886 | |
|
| hendrycksTest-professional_medicine | 0.5514705882352942 | |
|
| hendrycksTest-professional_psychology | 0.6013071895424836 | |
|
| hendrycksTest-public_relations | 0.6636363636363637 | |
|
| hendrycksTest-security_studies | 0.6448979591836734 | |
|
| hendrycksTest-sociology | 0.7611940298507462 | |
|
| hendrycksTest-us_foreign_policy | 0.84 | |
|
| hendrycksTest-virology | 0.4819277108433735 | |
|
| hendrycksTest-world_religions | 0.7894736842105263 | |
|
| truthfulqa:mc | 0.4762440289139372 | |
|
| winogrande | 0.7616416732438832 | |
|
| gsm8k | 0.20621683093252463 | |
|
|
|
|
|
## Citations |
|
|
|
``` |
|
@misc{open-llm-leaderboard, |
|
author = {Edward Beeching and Clémentine Fourrier and Nathan Habib and Sheon Han and Nathan Lambert and Nazneen Rajani and Omar Sanseviero and Lewis Tunstall and Thomas Wolf}, |
|
title = {Open LLM Leaderboard}, |
|
year = {2023}, |
|
publisher = {Hugging Face}, |
|
howpublished = "\url{https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard}" |
|
} |
|
``` |
|
``` |
|
@software{eval-harness, |
|
author = {Gao, Leo and |
|
Tow, Jonathan and |
|
Biderman, Stella and |
|
Black, Sid and |
|
DiPofi, Anthony and |
|
Foster, Charles and |
|
Golding, Laurence and |
|
Hsu, Jeffrey and |
|
McDonell, Kyle and |
|
Muennighoff, Niklas and |
|
Phang, Jason and |
|
Reynolds, Laria and |
|
Tang, Eric and |
|
Thite, Anish and |
|
Wang, Ben and |
|
Wang, Kevin and |
|
Zou, Andy}, |
|
title = {A framework for few-shot language model evaluation}, |
|
month = sep, |
|
year = 2021, |
|
publisher = {Zenodo}, |
|
version = {v0.0.1}, |
|
doi = {10.5281/zenodo.5371628}, |
|
url = {https://doi.org/10.5281/zenodo.5371628} |
|
} |
|
``` |
|
``` |
|
@misc{clark2018think, |
|
title={Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge}, |
|
author={Peter Clark and Isaac Cowhey and Oren Etzioni and Tushar Khot and Ashish Sabharwal and Carissa Schoenick and Oyvind Tafjord}, |
|
year={2018}, |
|
eprint={1803.05457}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.AI} |
|
} |
|
``` |
|
``` |
|
@misc{zellers2019hellaswag, |
|
title={HellaSwag: Can a Machine Really Finish Your Sentence?}, |
|
author={Rowan Zellers and Ari Holtzman and Yonatan Bisk and Ali Farhadi and Yejin Choi}, |
|
year={2019}, |
|
eprint={1905.07830}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CL} |
|
} |
|
``` |
|
``` |
|
@misc{hendrycks2021measuring, |
|
title={Measuring Massive Multitask Language Understanding}, |
|
author={Dan Hendrycks and Collin Burns and Steven Basart and Andy Zou and Mantas Mazeika and Dawn Song and Jacob Steinhardt}, |
|
year={2021}, |
|
eprint={2009.03300}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CY} |
|
} |
|
``` |
|
``` |
|
@misc{lin2022truthfulqa, |
|
title={TruthfulQA: Measuring How Models Mimic Human Falsehoods}, |
|
author={Stephanie Lin and Jacob Hilton and Owain Evans}, |
|
year={2022}, |
|
eprint={2109.07958}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CL} |
|
} |
|
``` |
|
``` |
|
@misc{DBLP:journals/corr/abs-1907-10641, |
|
title={{WINOGRANDE:} An Adversarial Winograd Schema Challenge at Scale}, |
|
author={Keisuke Sakaguchi and Ronan Le Bras and Chandra Bhagavatula and Yejin Choi}, |
|
year={2019}, |
|
eprint={1907.10641}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CL} |
|
} |
|
``` |
|
``` |
|
@misc{DBLP:journals/corr/abs-2110-14168, |
|
title={Training Verifiers to Solve Math Word Problems}, |
|
author={Karl Cobbe and |
|
Vineet Kosaraju and |
|
Mohammad Bavarian and |
|
Mark Chen and |
|
Heewoo Jun and |
|
Lukasz Kaiser and |
|
Matthias Plappert and |
|
Jerry Tworek and |
|
Jacob Hilton and |
|
Reiichiro Nakano and |
|
Christopher Hesse and |
|
John Schulman}, |
|
year={2021}, |
|
eprint={2110.14168}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CL} |
|
} |
|
``` |
|
|
