Update README.md

README.md

@@ -9,12 +9,25 @@ metrics:
 pipeline_tag: text-generation
 ---
 
-## FBI-LLM-7B
+# FBI-LLM-7B
+## Model Details
+
 This work presents a Fully BInarized Large Language Model (FBI-LLM), demonstrating for the first time how to train a large-scale binary language model (not the ternary LLM like BitNet b1.58 from scratch to match the performance of its full-precision counterparts (e.g., FP16 or BF16) in transformer-based LLMs. It achieves this by employing an autoregressive distillation (AD) loss with maintaining equivalent model dimensions (130M, 1.3B, 7B) and training data volume as regular LLM pretraining, while delivering competitive results in terms of perplexity and task-specific effectiveness. Intriguingly, by analyzing the training trajectory, we find that the pretrained weight is not necessary for training binarized LLMs from scratch. This research encourages a new computational framework and may facilitate the future design of specialized hardware tailored for fully 1-bit LLMs. We make all models, code, and training dataset fully accessible and transparent to support further research.
 
+**Input**: Models input text only.
+
+**Output**: Models generate text only.
+
 ## Tokenizer
 We use the same tokenizer as [meta-llama/Llama-2-7b-hf](https://huggingface.co/meta-llama/Llama-2-7b-hf)
 
+## Training Data
+We use [AmberDateset](https://huggingface.co/datasets/LLM360/AmberDatasets) to train our models.
+
+## Result
+
+![image](https://huggingface.co/LiqunMa/FBI-LLM_7B/blob/main/main_result.jpg)
+
 ## How to use
 Please download the code from [LiqunMa/FBI-LLM](https://github.com/LiqunMa/FBI-LLM) firstly
 ```python
@@ -49,4 +62,9 @@ def load_model(model_size, model_dir):
         param.data = param.data.to(torch.float16)
 
     return model, tokenizer
+```
+
+## Citation
+### BibTeX:
+```
 ```