RedHatAI/TinyLlama-1.1B-Chat-v0.3-pruned50-quant-ds

Usage

from deepsparse import TextGeneration

prompt = "How to get in a good university?"
formatted_prompt = f"<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant\n"

model = TextGeneration(model="hf:neuralmagic/TinyLlama-1.1B-Chat-v0.3-pruned50-quant-ds")
print(model(formatted_prompt, max_new_tokens=200).generations[0].text)

"""
Getting into a good university is a complex process that involves factors such as academic performance, financial aid, and personal qualifications. Here are some steps you can follow to get in a good university:

1. Academic performance:

- Look for a university that has a strong academic program, including a well-rounded curriculum that covers a wide range of subjects.
- Check if the university offers a clear curriculum that includes a clear sequence of courses.
- Check if the university offers a clear pathway to graduation, including clear dates and deadlines.

2. Financial aid:

- Look for a university that offers financial aid, such as scholarships, grants, or loans.
- Check if the university offers financial aid that fits your budget.
- Consider the university's financial aid package, including the cost of tuition, room and board, and other expenses.
"""

One-shot and Export

git clone https://github.com/neuralmagic/sparseml
pip install -e "sparseml[transformers]" "torch<2"
python sparseml/src/sparseml/transformers/sparsification/obcq/obcq.py PY007/TinyLlama-1.1B-Chat-v0.3 open_platypus --recipe recipe.yaml --save True
python sparseml/src/sparseml/transformers/sparsification/obcq/export.py --task text-generation --model_path obcq_deployment --sequence_length 512
cp deployment/model.onnx deployment/model-orig.onnx
python onnx_kv_inject.py --input-file deployment/model-orig.onnx --output-file deployment/model.onnx

recipe.yaml

test_stage:
  obcq_modifiers:
    SparseGPTModifier:
      sparsity: 0.5
      block_size: 128
      sequential_update: false
      quantize:
        QuantizationModifier:
          ignore:
          - LlamaRotaryEmbedding
          - LlamaRMSNorm
          - SiLUActivation
          - model.layers.21.mlp.down_proj
          - model.layers.7.mlp.down_proj
          - model.layers.2.mlp.down_proj
          - model.layers.20.mlp.down_proj
          - model.layers.19.mlp.down_proj
          post_oneshot_calibration: false
          scheme_overrides:
            Embedding:
              input_activations: null
              weights:
                num_bits: 8
                symmetric: false
      percdamp: 0.01
      prunen: 0
      prunem: 0
      targets:
      - model.layers.0
      - model.layers.1
      - model.layers.2
      - model.layers.3
      - model.layers.4
      - model.layers.5
      - model.layers.6
      - model.layers.7
      - model.layers.8
      - model.layers.9
      - model.layers.10
      - model.layers.11
      - model.layers.12
      - model.layers.13
      - model.layers.14
      - model.layers.15
      - model.layers.16
      - model.layers.17
      - model.layers.18
      - model.layers.19
      - model.layers.20
      - model.layers.21
      target_ids:
      - attention_mask
      - position_ids