Update README.md

README.md

@@ -95,26 +95,50 @@ Use the code below to get started with the model.
 
 <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
 
-[More Information Needed]
+The model has been fine-tuned on the **dmedhi/wiki_medical_terms** dataset. This dataset is designed to improve medical terminology comprehension and consists of:
+
+- Medical definitions and terminologies
+- Disease symptoms and conditions
+- Healthcare and clinical knowledge from Wikipedia's medical section
+
+This dataset ensures that the fine-tuned model performs well in understanding and responding to medical queries with enhanced accuracy.
 
 ### Training Procedure
 
 <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
 
-#### Preprocessing [optional]
+#### Preprocessing
 
-[More Information Needed]
+- The dataset was cleaned and tokenized using the Llama 3.1 tokenizer, ensuring that medical terms were preserved.
+
+- Special medical terminologies were handled properly to maintain context.
+
+- The dataset was formatted into a question-answer style to align with the instruction-based nature of Llama 3.1 3B Instruct.
 
 
 #### Training Hyperparameters
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+- **Training regime:**  <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
 
-#### Speeds, Sizes, Times [optional]
+- **Training regime:** bf16 mixed precision (to balance efficiency and precision)
+- **Batch Size:** 1 per device
+- **Gradient Accumulation Steps:** 4 (to simulate a larger batch size)
+- **Learning Rate:** 2e-4
+- **Warmup Steps:** 100
+- **Epochs:** 3
+- **Optimizer:** paged_adamw_8bit (efficient low-memory optimizer)
+- **LoRA Rank (r):** 16
+- **LoRA Alpha:** 32
+- **LoRA Dropout:** 0.05
+
+#### Speeds, Sizes, Times
 
 <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
 
-[More Information Needed]
+- **Training Hardware:** Single GPU (consumer-grade, VRAM-optimized)
+- **Model Size after Fine-Tuning:** Approx. 3B parameters with LoRA adapters
+- **Training Time:** ~3-4 hours per epoch on A100 40GB GPU
+- **Final Checkpoint Size:** ~2.8GB (with LoRA adapters stored separately)
 
 ## Evaluation
 
@@ -160,10 +184,10 @@ Use the code below to get started with the model.
 
 Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
 
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
+- **Hardware Type:** A100 40 GB GPU
+- **Hours used:** Approximatly 3 to 4 hours
+- **Cloud Provider:** Google Colabs
+- **Compute Region:** US-East
 - **Carbon Emitted:** [More Information Needed]
 
 ## Technical Specifications [optional]