metadata
base_model: microsoft/phi-2
inference: true
model_type: phi
quantized_by: mgoin
tags:
- nm-vllm
- sparse
phi-2-pruned50
This repo contains model files for Phi 2 optimized for NM-vLLM, a high-throughput serving engine for compressed LLMs.
This model was pruned with SparseGPT, using SparseML.
Inference
Install NM-vLLM for fast inference and low memory-usage:
pip install nm-vllm[sparse]
Run in a Python pipeline for local inference:
from vllm import LLM, SamplingParams
# Create a sparse LLM
llm = LLM("nm-testing/phi-2-pruned50", sparsity="sparse_w16a16")
prompt = "Once upon a time, there was a little car named Beep."
# Create a sampling params object.
sampling_params = SamplingParams(temperature=0.0, max_tokens=200)
# Generate texts from the prompts. The output is a list of RequestOutput objects
# that contain the prompt, generated text, and other information.
outputs = llm.generate(prompt, sampling_params)
# Print the outputs.
for output in outputs:
prompt = output.prompt
generated_text = output.outputs[0].text
print(f"\nGenerated text: {prompt}{generated_text}\n")
"""
Generated text: Once upon a time, there was a little car named Beep. Beep was a small car, but he was very fast and loved to go on adventures. Beep had a friend named Bop who was a big car. Bop was very slow and loved to stay at home. Beep and Bop were very different, but they were still friends.
One day, Beep and Bop decided to go on an adventure together. Beep was excited to explore new places and Bop was excited to see Beep explore. They started their adventure by driving on a bumpy road. Beep was having a great time, but Bop was having a hard time. Bop was so big that he couldn't fit in the small spaces between the bumps. Beep was having a great time, but Bop was having a hard time.
As they continued their adventure, they came across a big hill. Beep was excited to climb the hill, but Bop was scared. Bop was so big that he couldn't
"""
Prompt template
"Instruct: <prompt>\nOutput:"
Sparsification
For details on how this model was sparsified, see the recipe.yaml in this repo and follow the instructions below.
Install SparseML:
git clone https://github.com/neuralmagic/sparseml
pip install -e "sparseml[transformers]"
Replace the recipe as you like and run this one-shot compression script to apply SparseGPT:
import sparseml.transformers
original_model_name = microsoft/phi-2"
calibration_dataset = "open_platypus"
output_directory = "output/"
recipe = """
test_stage:
obcq_modifiers:
SparseGPTModifier:
sparsity: 0.5
sequential_update: true
targets: ['re:model.layers.\d*$']
"""
# Apply SparseGPT to the model
sparseml.transformers.oneshot(
model=original_model_name,
dataset=calibration_dataset,
recipe=recipe,
output_dir=output_directory,
)
Slack
For further support, and discussions on these models and AI in general, join Neural Magic's Slack Community