update README.md
Browse files
README.md
CHANGED
|
@@ -10,6 +10,8 @@ tags:
|
|
| 10 |
---
|
| 11 |
# TinyLlama-1.1B-32k
|
| 12 |
|
|
|
|
|
|
|
| 13 |
32k context finetune of TinyLlama-1.1B using increased rope theta (rope frequency base) meant to serve as a long-context speculative decoding model.
|
| 14 |
|
| 15 |
Created using [TinyLlama-1.1B](https://huggingface.co/TinyLlama/tinyLlama-intermediate-checkpoints-after-1T-token) and further pretraining at 32768 context length on [togethercomputer/RedPajama-Data-1T-Sample](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T-Sample).
|
|
|
|
| 10 |
---
|
| 11 |
# TinyLlama-1.1B-32k
|
| 12 |
|
| 13 |
+
#### NOTE: This is a fork of the original model at https://huggingface.co/Doctor-Shotgun/TinyLlama-1.1B-32k but with fixed safetensors
|
| 14 |
+
|
| 15 |
32k context finetune of TinyLlama-1.1B using increased rope theta (rope frequency base) meant to serve as a long-context speculative decoding model.
|
| 16 |
|
| 17 |
Created using [TinyLlama-1.1B](https://huggingface.co/TinyLlama/tinyLlama-intermediate-checkpoints-after-1T-token) and further pretraining at 32768 context length on [togethercomputer/RedPajama-Data-1T-Sample](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T-Sample).
|