fredericowieser/TinyLlama_v1.1_mix_wikitext_alpaca_1bit_BitDistiller_baseline

TinyLlama_v1.1_1bit_BitDistiller

This is a 1-bit quantized version of TinyLlama v1.1, trained using BitDistiller with asymmetric quantization and self-distillation (CAKLD) to optimize accuracy retention under extreme compression. The model is fine-tuned on WikiText-2 and Alpaca-cleaned datasets and evaluated on multiple-choice QA benchmarks.

Key Features:

• 1-bit quantization for ultra-efficient inference.
• Asymmetric weight clipping to reduce precision loss.
• CAKLD knowledge distillation to preserve performance.
• Tested on ARC-Challenge, HellaSwag, PIQA, and Winogrande.