Speculative decoding with Qwen 32B + Qwen 1.5B

Example:

llama-server \
    -m qwen2.5-coder-32b-instruct-q4_k_m.gguf \
    -md qwen2.5-coder-1.5b-instruct-q4_k_m.gguf \
    --ctx-size 65536
Downloads last month
38
GGUF
Model size
1.78B params
Architecture
qwen2

4-bit

Inference Examples
Inference API (serverless) does not yet support gguf models for this pipeline type.

Model tree for ggml-org/Qwen2.5-Coder-1.5B-32B-speculative-GGUF

Base model

Qwen/Qwen2.5-32B
Quantized
(78)
this model