Update README.md

README.md

@@ -25,6 +25,14 @@ base_model:
 ## Training
 
 
+Bigger models, more data, and better hardware have consistently improved deep learning performance. Whether in NLP or computer vision, larger models have led to major breakthroughs. However, most cutting-edge models are still trained from scratch, meaning they start with randomly initialized weights. The problem? Training costs are skyrocketing.
+
+### **More Scientific Version:**
+
+To address the escalating computational costs of training large-scale models, various approaches have been proposed. For instance, **[arXiv.2212.05055](https://doi.org/10.48550/arXiv.2212.05055)** demonstrates a method where pretrained large models are upscaled by selectively retaining dense layers, followed by continued pretraining. This strategy can potentially reduce the training budget by up to **50%** while maintaining performance.
+
+In this work, we take a step toward realizing such an approach. Specifically, we extend an existing **8B**-parameter model to **10B** parameters by initializing the additional layers with pretrained weights, followed by continued pretraining on a smaller dataset across multiple epochs. Due to budget constraints, we were unable to surpass the foundational model on the **EleutherAI** evaluation benchmark. However, our approach yielded improved performance in terms of **perplexity**, demonstrating potential for cost-efficient scaling strategies in large language model development.
+
 
 ## Usage
 
@@ -34,14 +42,12 @@ base_model:
 ```python
 import torch
 from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer
-
-tokenizer = AutoTokenizer.from_pretrained("empirischtech/Llama-3.1-10b-instruct")
+model_id="empirischtech/Llama-3.1-10b-instruct"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
 model = AutoModelForCausalLM.from_pretrained(
-    "upstage/llama-65b-instruct",
+    model_id,
     device_map="auto",
-    torch_dtype=torch.float16,
-    load_in_8bit=True,
-    rope_scaling={"type": "dynamic", "factor": 2} # allows handling of longer inputs
+    torch_dtype=torch.float16
 )
 
 prompt = "### User:\nThomas is healthy, but he has to go to the hospital. What could be the reasons?\n\n### Assistant:\n"