initial commit

README.md

@@ -0,0 +1,63 @@
+<p align="center" width="100%">
+<img src="assets/logo.png" alt="Vigogne" style="width: 30%; min-width: 300px; display: block; margin: auto;">
+</p>
+
+# LLaMandement-13B: A Llama-2-based French Models for Summarization of French Legislative Proposals.
+
+LLaMandement-13B is a French chat LLM, based on [LLaMA-2-13B](https://ai.meta.com/llama), optimized to summarize of French Legislative Proposals.
+
+## Model Details
+
+- **Developed by:** [DGFIP](https://www.impots.gouv.fr/presentation-de-la-dgfip-overview-dgfip) : 
+   - [email protected] 
+   - [email protected] 
+- **Model type:** An auto-regressive language model based on the transformer architecture
+- **License:** Llama 2 Community License Agreement
+- **Finetuned from model:** [Llama 2](https://arxiv.org/abs/2307.09288)
+- **Repository:** https://gitlab.adullact.net/dgfip/projets-ia/llamandement
+- **Paper:** working
+
+## Prompt Template
+
+
+The prompt for LLaMandement-13B is based on alpaca template :
+
+```
+Below is an instruction that describes a task. Write a response that appropriately completes the request.
+
+### Instruction:
+{instruction}
+
+### Response:
+```
+
+## How to Get Started with the Model
+
+- Command line interface: https://github.com/lm-sys/FastChat
+- APIs (OpenAI API, Huggingface API): https://github.com/lm-sys/FastChat/tree/main#api  
+
+## Training Details
+
+Llamandement-13B is fine-tuned from Llama 2 using Low-Rank Adaptation (LORA). This method is efficient and adds minimal computational load. It introduces additional low-rank parameters, enabling the model to better handle complex legislative language without major changes to the original structure.
+
+**LORA Settings Adjustments:**
+
+- **Learning Rate (LR):** Set to a low value of 2e-5 to ensure stable and gradual improvements.
+- **Adaptation Depth (lora_r):** Set at 64, influencing the dimension of the low-rank matrix in LORA. This affected about 0.40% of the model's weights.
+- **Decay Rate:** Employed at 0.01 to prevent overfitting to specific legislative text structures.
+- **LORA Alpha (α):** Set at 16, it fine-tunes the model's response to legislative text.
+- **LORA Dropout:** A rate of 0.1 applied to LORA layers to prevent overfitting and enhance generalization.
+- **Optimizer and Scheduler:** Utilized a cosine learning rate scheduler with a warmup ratio of 0.03 for optimal training.
+
+
+For more information, visit [dgfip.finance.com](http://dgfip.finance.com). Additional details about the training dataset composition can be found [here](http://dgfip.finance.com/training-dataset-info).
+
+
+
+## Citation
+
+Please cite the repo if you use the data, method or code in this repo.
+
+[...]
+
+