Update README.md

README.md

@@ -5,3 +5,52 @@ license: mit
 tags:
 - legal
 ---
+
+# Lawma 8B
+
+Lawma 8B is a fine-tune of Llama 3 8B Instruct on 260 legal classification tasks derived from [Supreme Court](http://scdb.wustl.edu/data.php) and [Songer Court of Appeals](www.songerproject.org/us-courts-of-appeals-databases.html) databases. Lawma was fine-tuned on over 500k task examples, totalling 2B tokens. As a result, Lawma 8B outperforms GPT-4 on 95\% of these legal classification tasks, on average by over 17 accuracy points. See our [arXiv preprint](https://arxiv.org/abs/2407.16615) and [GitHub repository](https://github.com/socialfoundations/lawma) for more details.
+
+## Evaluations
+
+We report mean classification accuracy across the 260 legal classification tasks that we consider. We use the standard MMLU multiple-choice prompt, and evaluate models zero-shot. You can find our evaluation code [here](https://github.com/socialfoundations/lawma/tree/main/evaluation).
+
+| Model   | All tasks | Supreme Court tasks | Court of Appeals tasks |
+|---------|:---------:|:-------------:|:----------------:|
+| Lawma 70B | **81.9** | **84.1** | **81.5** |
+| Lawma 8B | 80.3 | 82.4 | 79.9 |
+| GPT4 | 62.9 | 59.8 | 63.4 |
+| Llama 3 70B Inst | 58.4 | 47.1 | 60.3 |
+| Mixtral 8x7B Inst | 43.2 | 24.4 | 46.4 |
+| Llama 3 8B Inst | 42.6 | 32.8 | 44.2 |
+| Majority classifier | 41.7 | 31.5 | 43.5 |
+| Mistral 7B Inst | 39.9 | 19.5 | 43.4 |
+| Saul 7B Inst | 34.4 | 20.2 | 36.8 |
+| LegalBert | 24.6 | 13.6 | 26.4 |
+
+## FAQ
+
+**What are the Lawma models useful for?** We recommend using the Lawma models only for legal classification tasks that are very similar to those in the Supreme Court and Court of Appeals databases. The main take-away of our paper is that fine-tuning on specific tasks of interest leads to large improvements in performance. Therefore, we strongly recommend to further fine-tune on the tasks that you whish to use the model for. Relatively few examples --i.e, dozens or hundreds-- may already lead to large gains in performance. Since Lawma was fine-tuned on a diverse set of legal classification tasks, fine-tuning Lawma may yield better resuts compared to fine-tuning more general instruction-tuned models.
+
+**What legal classification tasks do we consider?** We consider almost all of the variables of the [Supreme Court](http://scdb.wustl.edu/data.php) and [Songer Court of Appeals](www.songerproject.org/us-courts-of-appeals-databases.html) databases. Our reasons to study these legal classification tasks are both technical and substantive. From a technical machine learning perspective, these tasks provide highly non-trivial classification problems where
+even the best models leave much room for improvement. From a substantive legal perspective, efficient
+solutions to such classification problems have rich and important applications in legal research.
+
+## Citation
+
+This model was trained for the project
+
+*Lawma: The Power of Specizalization for Legal Tasks. Ricardo Dominguez-Olmedo and Vedant Nanda and Rediet Abebe and Stefan Bechtold and Christoph Engel and Jens Frankenreiter and Krishna Gummadi and Moritz Hardt and Michael Livermore. 2024*
+
+Please cite as:
+
+```
+@misc{dominguezolmedo2024lawmapowerspecializationlegal,
+      title={Lawma: The Power of Specialization for Legal Tasks}, 
+      author={Ricardo Dominguez-Olmedo and Vedant Nanda and Rediet Abebe and Stefan Bechtold and Christoph Engel and Jens Frankenreiter and Krishna Gummadi and Moritz Hardt and Michael Livermore},
+      year={2024},
+      eprint={2407.16615},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2407.16615}, 
+}
+```