Update README.md

README.md

@@ -3,10 +3,17 @@ license: cc-by-nc-4.0
 datasets:
 - argilla/dpo-mix-7k
 - nvidia/HelpSteer
+- jondurbin/airoboros-3.2
+- hkust-nlp/deita-10k-v0
+- LDJnr/Capybara
+- HPAI-BSC/CareQA
+- GBaker/MedQA-USMLE-4-options
+- lukaemon/mmlu
+- bigbio/pubmed_qa
+- openlifescienceai/medmcqa
+- bigbio/med_qa
 language:
 - en
-metrics:
-- accuracy
 library_name: transformers
 tags:
 - biology
@@ -26,20 +33,19 @@ Aloe is a new family of healthcare LLMs that is highly competitive with all prev
 - **Developed by:** [HPAI](https://hpai.bsc.es/)
 - **Model type:** Causal decoder-only transformer language model
 - **Language(s) (NLP):** English (mainly)
-- **License:** [Meta Llama 3 License](https://llama.meta.com/llama3/license/)
-- **Finetuned from model :** [meta-llama/Meta-Llama-3-8B · Hugging Face](https://huggingface.co/meta-llama/Meta-Llama-3-8B)
+- **License:** This model is based on Meta Llama 3 8B and is governed by the [Meta Llama 3 License](https://llama.meta.com/llama3/license/). All our modifications are available with a [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) license.
+- **Finetuned from model :** [meta-llama/Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B)
 
 ### [](https://huggingface.co/templates/model-card-example#model-sources-optional)Model Sources [optional]
 
 - **Repository:** Coming Soon
-- **Paper [optional]:** Coming soon
+- **Paper:** Coming soon
 
 ## Uses
 
 ### Direct Use
 
-We encourage the use of Aloe for research purposes, as a  
-stepping stone to build better foundational models for healthcare.
+We encourage the use of Aloe for research purposes, as a stepping stone to build better foundational models for healthcare.
 
 ### Out-of-Scope Use
 
@@ -47,13 +53,13 @@ These models are not to be used for clinical practice, medical diagnosis, or any
 
 ## Bias, Risks, and Limitations
 
-First let us consider Healthcare professional impersonation, a fraudulent behaviour which currently generates billions of dollars in profit https://www.justice.gov/opa/pr/justice-department-charges-dozens-12-billion-health-care-fraud. A model such as Aloe could be used to increase the efficacy of such deceiving activities, making them more widespread. The main preventive actions are public literacy on the unreliability of digitised information and the importance of medical registration, and legislation enforcing AI-generated content disclaimers. The second risk we consider is medical decision-making without professional supervision. While this is already an issue in modern societies (\eg self-medication) a model such as Aloe, capable of producing high-quality conversational data, can facilitate self-delusion, particularly in the presence of sycophancy. By producing tailored responses, it can also be used to generate actionable answers. Public literacy on the dangers of self-diagnosis is one of the main defences, together with the introduction of disclaimers and warnings on the models' outputs. The last risk we consider is the access to information on dangerous substances or procedures. While the literature on sensitive content can already be found on different sources (\eg libraries, internet, dark web), LLMs can centralize such access, making it nearly impossible to control the flow of such information. Model alignment can help in that regard, but so far the effects remain insufficient, as jailbreaking methods still overcome it.
+First let us consider Healthcare professional impersonation, a fraudulent behaviour which currently generates billions of dollars in [profit](https://www.justice.gov/opa/pr/justice-department-charges-dozens-12-billion-health-care-fraud). A model such as Aloe could be used to increase the efficacy of such deceiving activities, making them more widespread. The main preventive actions are public literacy on the unreliability of digitised information and the importance of medical registration, and legislation enforcing AI-generated content disclaimers. The second risk we consider is medical decision-making without professional supervision. While this is already an issue in modern societies (eg self-medication) a model such as Aloe, capable of producing high-quality conversational data, can facilitate self-delusion, particularly in the presence of sycophancy. By producing tailored responses, it can also be used to generate actionable answers. Public literacy on the dangers of self-diagnosis is one of the main defences, together with the introduction of disclaimers and warnings on the models' outputs. The last risk we consider is the access to information on dangerous substances or procedures. While the literature on sensitive content can already be found on different sources (eg libraries, internet, dark web), LLMs can centralize such access, making it nearly impossible to control the flow of such information. Model alignment can help in that regard, but so far the effects remain insufficient, as jailbreaking methods still overcome it.
 
-<img src="https://cdn-uploads.huggingface.co/production/uploads/62972c4979f193515da1d38e/JR7AU-DwJRNAmk8vFPmfT.png" width="90%">
+<img src="https://cdn-uploads.huggingface.co/production/uploads/62972c4979f193515da1d38e/T6Jblpf1kmTkM04K716rM.png" width="90%">
 
 ### Recommendations
 
-We avoid the use of all personal data in our training. Model safety cannot be guaranteed, as shown in the red teaming results. Aloe can produce toxic content under the appropriate prompts. For these reasons, minors should not be left alone to interact with Aloe without supervision.
+We avoid the use of all personal data in our training. Model safety cannot be guaranteed. Aloe can produce toxic content under the appropriate prompts. For these reasons, minors should not be left alone to interact with Aloe without supervision.
 
 ## How to Get Started with the Model
 
@@ -150,25 +156,14 @@ Supervised fine-tuning on top of Llama 3 8B using medical and general domain dat
 
 ### Training Data
 
-- Medical and general domain datasets, including synthetic data generated using Mixtral-8x7B and Genstruct
+- Medical domain datasets, including synthetic data generated using Mixtral-8x7B and Genstruct
+- LDJnr/Capybara
+- hkust-nlp/deita-10k-v0
+- jondurbin/airoboros-3.2
 - argilla/dpo-mix-7k
 - nvidia/HelpSteer
 - Custom preference data with adversarial prompts generated from Anthropic Harmless, Chen et al., and original prompts
 
-### Training Procedure
-
-#### Preprocessing [optional]
-
-[More Information Needed]
-
-#### Training Hyperparameters
-
-- **Training regime:** [More Information Needed]
-
-#### Speeds, Sizes, Times [optional]
-
-[More Information Needed]
-
 ## Evaluation
 
 ### Testing Data, Factors & Metrics
@@ -180,10 +175,7 @@ Supervised fine-tuning on top of Llama 3 8B using medical and general domain dat
 - [PubMedQA](https://huggingface.co/datasets/bigbio/pubmed_qa)
 - [MMLU-Medical](https://huggingface.co/datasets/lukaemon/mmlu)
 - [MedQA-4-Option](https://huggingface.co/datasets/GBaker/MedQA-USMLE-4-options)
-
-#### Factors
-
-[More Information Needed]
+- [CareQA](https://huggingface.co/datasets/HPAI-BSC/CareQA)  
 
 #### Metrics
 
@@ -191,20 +183,15 @@ Supervised fine-tuning on top of Llama 3 8B using medical and general domain dat
 
 ### Results
 
-<img src="https://cdn-uploads.huggingface.co/production/uploads/62972c4979f193515da1d38e/5viYXiXgsTnlMLVRNa1NP.png" width="90%">
-
+<img src="https://cdn-uploads.huggingface.co/production/uploads/62972c4979f193515da1d38e/STlPSggXr9P9JeWAvmAsi.png" width="90%">
 
 #### Summary
 
-To compare Aloe with the most competitive open models (both general purpose and healthcare-specific) we use popular healthcare datasets (\eg PubMedQA, MedMCQA, MedQA and MMLU for six medical tasks only), together with the new and highly reliable CareQA. We produce the standard MultiMedQA score for reference, by computing the weighted average accuracy on all scores except CareQA. Additionally, we calculate the arithmetic mean across all datasets. The Medical MMLU is calculated by averaging the six medical subtasks: Anatomy, Clinical knowledge, College Biology, College medicine, Medical genetics, and Professional medicine.
-
-Benchmark results indicate the training conducted on Aloe  has boosted its performance slightly above Llama3-8B-Instruct. Llama3-Aloe-8B-Alpha outperforms larger models like Meditron 70B, and is close to larger base models, like Yi-34} For the former, this gain is consistent even when using SC-CoT, using their best-reported variant. All these results make Llama3-Aloe-8B-Alpha the best healthcare LLM of its size.
-
-With the help of prompting techniques the performance of Llama3-Aloe-8B-Alpha is significantly improved. Medprompting in particular provides a 7\% increase in reported accuracy, after which \aloealpha only lags behind the ten times bigger Llama-3-70B-Instruct. This improvement is mostly consistent across medical fields. Llama3-Aloe-8B-Alpha with medprompting beats the performance of Meditron 70B with their self reported 20 shot SC-CoT in MMLU med and is slightly worse in the other benchmarks. 
+To compare Aloe with the most competitive open models (both general purpose and healthcare-specific) we use popular healthcare datasets (PubMedQA, MedMCQA, MedQA and MMLU for six medical tasks only), together with the new and highly reliable CareQA. We produce the standard MultiMedQA score for reference, by computing the weighted average accuracy on all scores except CareQA. Additionally, we calculate the arithmetic mean across all datasets. The Medical MMLU is calculated by averaging the six medical subtasks: Anatomy, Clinical knowledge, College Biology, College medicine, Medical genetics, and Professional medicine.
 
-## Model Examination [optional]
+Benchmark results indicate the training conducted on Aloe has boosted its performance above Llama3-8B-Instruct. Llama3-Aloe-8B-Alpha outperforms larger models like Meditron 70B, and is close to larger base models, like Yi-34. For the former, this gain is consistent even when using SC-CoT, using their best-reported variant. All these results make Llama3-Aloe-8B-Alpha the best healthcare LLM of its size.
 
-[More Information Needed]
+With the help of prompting techniques the performance of Llama3-Aloe-8B-Alpha is significantly improved. Medprompting in particular provides a 7% increase in reported accuracy, after which Llama3-Aloe-8B-Alpha only lags behind the ten times bigger Llama-3-70B-Instruct. This improvement is mostly consistent across medical fields. Llama3-Aloe-8B-Alpha with medprompting beats the performance of Meditron 70B with their self reported 20 shot SC-CoT in MMLU med and is slightly worse in the other benchmarks. 
 
 ## Environmental Impact
 
@@ -214,46 +201,6 @@ With the help of prompting techniques the performance of Llama3-Aloe-8B-Alpha is
 - **Compute Region:** Spain
 - **Carbon Emitted:** 439.25kg
 
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
-
-## Citation [optional]
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
-
-[More Information Needed]
-
-## More Information [optional]
-
-[More Information Needed]
-
-## Model Card Authors [optional]
-
-[More Information Needed]
-
 ## Model Card Contact
 
-mailto:[email protected]
+mailto:[email protected]