File size: 2,720 Bytes
fb55dfd 8bda519 b253413 fb55dfd b253413 fb55dfd 6d9e514 fb55dfd b253413 fb55dfd b253413 fb55dfd b253413 fb55dfd b253413 fb55dfd b253413 fb55dfd b253413 fb55dfd 4d1247a b253413 fb55dfd b253413 fb55dfd b253413 fb55dfd b253413 fb55dfd b253413 8bda519 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 |
---
license: apache-2.0
language:
- en
library_name: transformers
tags:
- medical
---
# JSL-MedMX-7X
[<img src="https://repository-images.githubusercontent.com/104670986/2e728700-ace4-11ea-9cfc-f3e060b25ddf">](http://www.johnsnowlabs.com)
This model is developed by [John Snow Labs](https://www.johnsnowlabs.com/).
Performance on biomedical benchmarks: [Open Medical LLM Leaderboard](https://huggingface.co/spaces/openlifescienceai/open_medical_llm_leaderboard).
This model is available under a [CC-BY-NC-ND](https://creativecommons.org/licenses/by-nc-nd/4.0/deed.en) license and must also conform to this [Acceptable Use Policy](https://huggingface.co/johnsnowlabs). If you need to license this model for commercial use, please contact us at [email protected].
## 💻 Usage
```python
!pip install -qU transformers accelerate
from transformers import AutoTokenizer
import transformers
import torch
model = "johnsnowlabs/JSL-MedMX-7X"
messages = [{"role": "user", "content": "What is a large language model?"}]
tokenizer = AutoTokenizer.from_pretrained(model)
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
pipeline = transformers.pipeline(
"text-generation",
model=model,
torch_dtype=torch.float16,
device_map="auto",
)
outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
print(outputs[0]["generated_text"])
```
## 🏆 Evaluation
| Tasks |Version|Filter|n-shot| Metric |Value | |Stderr|
|-------------------------------|-------|------|-----:|--------|-----:|---|-----:|
|stem |N/A |none | 0|acc_norm|0.5783|± |0.0067|
| | |none | 0|acc |0.6177|± |0.0057|
| - medmcqa |Yaml |none | 0|acc |0.5668|± |0.0077|
| | |none | 0|acc_norm|0.5668|± |0.0077|
| - medqa_4options |Yaml |none | 0|acc |0.6159|± |0.0136|
| | |none | 0|acc_norm|0.6159|± |0.0136|
| - anatomy (mmlu) | 0|none | 0|acc |0.7111|± |0.0392|
| - clinical_knowledge (mmlu) | 0|none | 0|acc |0.7396|± |0.0270|
| - college_biology (mmlu) | 0|none | 0|acc |0.7778|± |0.0348|
| - college_medicine (mmlu) | 0|none | 0|acc |0.6647|± |0.0360|
| - medical_genetics (mmlu) | 0|none | 0|acc |0.7200|± |0.0451|
| - professional_medicine (mmlu)| 0|none | 0|acc |0.7868|± |0.0249|
| - pubmedqa | 1|none | 0|acc |0.7840|± |0.0184| |