File size: 2,312 Bytes
0ed7d8c 86f3198 0ed7d8c d114a5e 0ed7d8c |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 |
---
license: apache-2.0
datasets:
- pico-lm/pretokenized-dolma
language:
- en
metrics:
- pico-lm/perplexity
pipeline_tag: text-generation
---
# Pico Decoder Tiny
**pico-decoder-tiny** is the smallest (11M) model in the `pico-decoder` suite — a lightweight, LLaMA-style decoder-only transformer trained from scratch using [`pico-train`](https://github.com/pico-lm/pico-train). It is designed for transparent and reproducible research into the learning dynamics of language models, and is fully compatible with the `pico-analyze` toolkit for detailed interpretability analysis.
> NOTE: The `pico-decoder-tiny-1` branch contains the full commit history for the training run.
## 🔧 Model Details
| Field | Value |
|---------------------|------------------------------------|
| **Architecture** | Decoder-only transformer (LLaMA-style) |
| **Parameters** | 11M |
| **Layers** | 12 |
| **Hidden Size** | 96 |
| **Feed Foward Size** | 384 |
| **Attention Heads** | 12 |
| **Key/Value Heads** | 4 |
## 📚 Training
- **Dataset**: [`pretokenized-dolma`](https://huggingface.co/datasets/pico-lm/pretokenized-dolma), English-only
- **Training steps**: 200,000
- **Batch size**: 1024
- **Sequence length**: 2048
- **Optimizer**: AdamW
- **Learning rate schedule**: Linear decay with warmup
- **Compute**: 16 A100-SXM4-80GB GPUs
## 📈 Evaluation and Analysis
This model supports fine-grained analysis using [`pico-analyze`](https://github.com/pico-lm/pico-analyze). This tool enables researchers to understand how learning unfolds over training, even at very small scales.
We also evaluate perplexity of the model on the [`pico-paloma-tinsy`](https://huggingface.co/datasets/pico-lm/pretokenized-paloma-tinsy) dataset.
## 📄 Citation
If you use `pico-tiny` or any other `pico-decoder` model in your research, please cite:
```bibtex
@software{pico2025,
author = {Diehl Martinez, Richard},
title = {Pico: A Lightweight Framework for Studying Language Model Learning Dynamics},
year = {2025,
url = {https://github.com/pico-lm}
}
``` |