Update README.md

README.md

@@ -1,3 +1,63 @@
 ---
-license: mit
+language:
+- en
+library_name: transformers
+license: llama2
 ---
+
+
+
+### Background
+
+Sparse computation is increasingly recognized as an important direction in enhancing the computational efficiency of large language models (LLMs). 
+Previous work has shown that models after relufication are characterised by sparse activation, which naturally introduces a new problem: Which activation function is optimal for sparse LLMs? Although previous works on activation function selection have focused on the performance of LLMs, we argue that the efficiency of sparse computation should also be considered so that the LLMs can proceed with efficient inference while preserving performance.
+
+To answer this question, we pretrain 4 LLMs with different activation functions, including ReLU, SwiGLU, ReGLU, and ReLU$^2$ to do more comprehensive experiments.
+
+### Dataset
+
+We pretrain the model on  100 billion tokens, including:
+
+* Refinedweb
+* SlimPajama
+
+
+### Training Details
+
+We jointly optimize the model on the conventional language modeling objective and the knowledge distillation objective. The knowledge distillation objective is to minimize the KL divergence between the teacher model and the student model. The teacher model is the original LLM, and the student model is the ReLU-activated version. Since the size of the fine-tuning data is relatively small, we introduce the knowledge distillation objective to avoid overfitting and enhance the generalization ability of the model, which can be also seen as a technique of label smoothing.
+
+| Parameter             | Value       |
+|-----------------------|-------------|
+| Batch_Size            | 4M          |
+| GPUs                  | 64xA100(80G)|
+| LR_Scheduler          | cosine      |
+| LR                    | 3e-4        |
+
+
+
+
+### License Disclaimer:
+
+This model is bound by the license & usage restrictions of the original Llama-2 model. And comes with no warranty or gurantees of any kind.
+
+### Limitations & Biases:
+
+Llama 2 and fine-tuned variants are a new technology that carries risks with use. Testing conducted to date has been in English, and has not covered, nor could it cover all scenarios. For these reasons, as with all LLMs, Llama 2 and any fine-tuned varient's potential outputs cannot be predicted in advance, and the model may in some instances produce inaccurate, biased or other objectionable responses to user prompts. Therefore, before deploying any applications of Llama 2 variants, developers should perform safety testing and tuning tailored to their specific applications of the model.
+
+Please see the Responsible Use Guide available at https://ai.meta.com/llama/responsible-use-guide/
+
+### Citiation:
+
+Please kindly cite using the following BibTeX:
+
+```bibtex
+@misc{sparsellm,
+    title={Sparse Large Language Models with ReLU Activation}, 
+    author={SpaseLLM Team},
+    year={2023}
+}
+```
+
+#### Acknowledgments:
+
+The model card is modified from [ORCA_LLaMA_70B_QLoRA](https://huggingface.co/fangloveskari/ORCA_LLaMA_70B_QLoRA).