Update README.md

README.md

@@ -1,48 +1,90 @@
 ---
-base_model: GeneZC/MiniChat-2-3B
-inference: True
-model_type: Llama
+base_model: NousResearch/Nous-Hermes-2-Yi-34B
+inference: true
+model_type: llama
+quantized_by: mgoin
+tags:
+- nm-vllm
+- sparse
 ---
-# Nous-Hermes-2-Yi-34B 
-This repo contains pruned model files for [Nous-Hermes-2-Yi-34B ](https://huggingface.co/NousResearch/Nous-Hermes-2-Yi-34B).
+
+## llama2.c-stories110M-pruned50
+This repo contains model files for [Nous Hermes 2 - Yi-34B](https://huggingface.co/NousResearch/Nous-Hermes-2-Yi-34B) optimized for [NM-vLLM](https://github.com/neuralmagic/nm-vllm), a high-throughput serving engine for compressed LLMs.
 
 This model was pruned with [SparseGPT](https://arxiv.org/abs/2301.00774), using [SparseML](https://github.com/neuralmagic/sparseml).
+
+## Inference
+Install [NM-vLLM](https://github.com/neuralmagic/nm-vllm) for fast inference and low memory-usage: 
+```bash
+pip install nm-vllm[sparse]
+```
+Run in a Python pipeline for local inference:
 ```python
-import torch
-from transformers import AutoTokenizer, AutoModelForCausalLM
+from vllm import LLM, SamplingParams
+
+model = LLM("nm-testing/Nous-Hermes-2-Yi-34B-pruned2.4", sparsity="sparse_w16a16")
 prompt = "How to make banana bread?"
 formatted_prompt =  f"<|im_start|>User:{prompt}\n<|im_start|>assistant:\n"
-model_id = "nm-testing/Nous-Hermes-2-Yi-34B-pruned50-24"
-model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto", torch_dtype=torch.float16)
-tokenizer = AutoTokenizer.from_pretrained(model_id)
-inputs = tokenizer(formatted_prompt, return_tensors="pt")
-outputs = model.generate(**inputs, max_new_tokens=200)
-print(tokenizer.batch_decode(outputs)[0])
-"""
-<|im_start|> User:How to make banana bread?
-<|im_start|> assistant:
-To make banana bread, you can follow these steps:
 
-Ingredients:
+sampling_params = SamplingParams(max_tokens=100, temperature=0)
+outputs = model.generate(formatted_prompt, sampling_params=sampling_params)
+print(outputs[0].outputs[0].text)
+"""
+To make banana bread, follow these steps:
+1. Gather the ingredients:
 - 2 ripe bananas
-- 2 cups flour
-- 1/2 cup sugar
-- 1/2 cup butter
-- 1/2 cup milk
-- 1 teaspoon baking powder
-- 1 teaspoon baking soda
-- 1 teaspoon salt
-
-Instructions:
-1. Preheat the oven to 350°F (175°C).
-2. In a mixing bowl, mash the bananas and mix them with the flour, sugar, butter, milk, baking powder, baking soda, and salt.
-3. Mix the ingredients until they form a dough.
-4. Pour the dough into a baking pan.
-5. Bake the banana bread for 30 minutes.
-6. Remove the banana bread from the oven and let it cool.
-7. Enjoy your banana bread.
-
-Note: You can adjust the ingredients
+- 2 cups of flour
+- 1 teaspoon of baking powder
+- 1 teaspoon of salt
+- 1 teaspoon of sugar
+- 1 teaspoon of vanilla extract
+2. Preheat the oven to 350°F.
+3. In a mixing bowl, combine the flour, baking powder, salt, sugar, and vanilla extract.
+4.
+"""
+```
+
+## Prompt template
+
+```
+<|im_start|>User:{prompt}\n<|im_start|>assistant:\n
+```
+
+## Sparsification
+For details on how this model was sparsified, see the `recipe.yaml` in this repo and follow the instructions below.
+
+Install [SparseML](https://github.com/neuralmagic/sparseml):
+```bash
+git clone https://github.com/neuralmagic/sparseml
+pip install -e "sparseml[transformers]"
+```
+
+Replace the recipe as you like and run this one-shot compression script to apply SparseGPT:
+```python
+import sparseml.transformers
+
+original_model_name = "NousResearch/Nous-Hermes-2-Yi-34B"
+calibration_dataset = "open_platypus"
+output_directory = "output/"
+
+recipe = """
+test_stage:
+  obcq_modifiers:
+    SparseGPTModifier:
+      sparsity: 0.5
+      sequential_update: true
+      targets: ['re:model.layers.\d*$']
 """
 
-```
+# Apply SparseGPT to the model
+sparseml.transformers.oneshot(
+    model=original_model_name,
+    dataset=calibration_dataset,
+    recipe=recipe,
+    output_dir=output_directory,
+)
+```
+
+## Slack
+
+For further support, and discussions on these models and AI in general, join [Neural Magic's Slack Community](https://join.slack.com/t/discuss-neuralmagic/shared_invite/zt-q1a1cnvo-YBoICSIw3L1dmQpjBeDurQ)