Update README.md

README.md

@@ -41,15 +41,12 @@ This is the model card of a 🤗 transformers model that has been pushed on the
 ## Uses
 
 <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
-### Direct Use
-
 <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
 
 You can use this model to build local/cloud RAG applications.
 It can serve as the:
 - Answer synthesizer,
-- Summarizer
+- Summarizer,
 - Or query rewriter model.
 
 ### Limitations 
@@ -160,7 +157,6 @@ Output:
 I used [SlimOrca](https://huggingface.co/datasets/Open-Orca/SlimOrca) dataset, a new curated subset of our OpenOrca data.
 In the course of this study, the [SlimOrca](https://huggingface.co/datasets/Open-Orca/SlimOrca) dataset was used, representing a meticulously curated subset derived from the broader OpenOrca dataset.  This release provides an efficient means of reaching performance on-par with using larger slices of the [OpenOrca](https://huggingface.co/datasets/Open-Orca/OpenOrca), while only including ~500k GPT-4 completions.
 
-
 Subsequently, two distinct subsets were crafted, comprising 102,000 and 1,000 samples, denoted as:
 
 - [prince-canuma/SmallOrca](https://huggingface.co/datasets/prince-canuma/SmallOrca)
@@ -173,24 +169,39 @@ succinct answers to prompts are often favored, especially for straightforward qu
 
 ### Training Procedure
 
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-[TODO]
-
 #### Preprocessing
 
 1. Convert dataset to chatML format
 2. Remove all samples with more than 2048 tokens (Phi-2 context size)
 3. Mask instructions (System and User) at training time.
 
+#### LoRA Config
+  - **lora_alpha:** 128,
+  - **lora_dropout:** 0.05,
+  - **r:** 256,
+  - **bias:** "none",
+  - **target_modules:** "all-linear",
+  - **task_type:** "CAUSAL_LM",
 
 #### Training Hyperparameters
 
-  - **Training regime:** bf16 mixed precision <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision --> 
+  - **Training regime:** bf16 mixed precision,  <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision --> 
+  - **max_steps:** 100,
+  - **per_device_train_batch_size:** 2,
+  - **gradient_accumulation_steps:** 2,
+  - **optim:** "adamw_torch_fused",
+  - **learning_rate:** 2e-4,
+  - **max_grad_norm:** 0.3,
+  - **warmup_ratio:** 0.03,
+  - **lr_scheduler_type:** "constant",
 
-[TODO] 
+#### Trainer 
+  - **max_seq_length:** 1744,
+  - **data_collator:** DataCollatorForCompletionOnlyLM
 
 ## Evaluation
 
+<img src="truthfulQA.png" width="800" alt="Damysus-2.7B-chat truthfulQA benchmark results"/>
 <!-- This section describes the evaluation protocols and provides the results. -->
 
 We evaluate models on 7 key benchmarks using the Eleuther AI Language Model Evaluation Harness , a unified framework to test generative language models on a large number of different evaluation tasks.