File size: 1,393 Bytes
b1e8332 181d0e5 b1e8332 181d0e5 b1e8332 181d0e5 aa0ac5e 181d0e5 aa0ac5e 181d0e5 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 |
---
base_model: EleutherAI/pythia-70m-deduped
language: en
library_name: mlsae
license: mit
tags:
- arxiv:2409.04185
- model_hub_mixin
- pytorch_model_hub_mixin
---
# Model Card for tim-lawson/mlsae-pythia-70m-deduped-x8-k32-tfm
A Multi-Layer Sparse Autoencoder (MLSAE) trained on the residual stream activation
vectors from [EleutherAI/pythia-70m-deduped](https://huggingface.co/EleutherAI/pythia-70m-deduped) with an
expansion factor of R = 8 and sparsity k = 32, over 1 billion
tokens from [monology/pile-uncopyrighted](https://huggingface.co/datasets/monology/pile-uncopyrighted).
This model is a PyTorch Lightning MLSAETransformer module, which includes the underlying
transformer.
### Model Sources
- **Repository:** <https://github.com/tim-lawson/mlsae>
- **Paper:** <https://arxiv.org/abs/2409.04185>
- **Weights & Biases:** <https://wandb.ai/timlawson-/mlsae>
## Citation
**BibTeX:**
```bibtex
@misc{lawson_residual_2024,
title = {Residual {{Stream Analysis}} with {{Multi-Layer SAEs}}},
author = {Lawson, Tim and Farnik, Lucy and Houghton, Conor and Aitchison, Laurence},
year = {2024},
month = oct,
number = {arXiv:2409.04185},
eprint = {2409.04185},
primaryclass = {cs},
publisher = {arXiv},
doi = {10.48550/arXiv.2409.04185},
urldate = {2024-10-08},
archiveprefix = {arXiv}
}
``` |