Create README.md

README.md

@@ -0,0 +1,56 @@
+---
+license: llama2
+language:
+- en
+library_name: transformers
+---
+
+## llama-2-7b-chat-marlin
+
+Example of converting a GPTQ model to Marlin format for fast batched decoding with [Marlin Kernels](https://github.com/IST-DASLab/marlin)
+
+### Install Marlin
+```bash
+pip install torch
+git clone https://github.com/IST-DASLab/marlin.git
+cd marlin
+pip install -e .
+```
+
+### Convert Model
+
+The following converts the model from GPTQ to Marlin format. Note that this requires:
+- `sym=true`
+- `group_size=128`
+- `desc_activations=false`
+
+```bash
+pip install -U transformers accelerate auto-gptq optimum
+```
+
+Convert with the `convert.py` script in this repo:
+
+```bash
+python3 convert.py --model-id "TheBloke/Llama-2-7B-Chat-GPTQ" --save-path "./marlin-model" --do-generation
+```bash
+
+### Run Model
+
+Load with the `load.load_model` utility from this repo and run inference as usual.
+
+```python
+from load import load_model
+from transformers import AutoTokenizer
+
+# Load model from disk.
+model_path = "./marlin-model"
+model = load_model(model_path).to("cuda")
+tokenizer = AutoTokenizer.from_pretrained(model_path)
+
+
+# Run inference to confirm it is working.
+inputs = tokenizer("My favorite song is", return_tensors="pt")
+inputs = {k: v.to("cuda") for k, v in inputs.items()}
+outputs = model.generate(**inputs, max_new_tokens=50, do_sample=False)
+print(tokenizer.batch_decode(outputs)[0])
+```