|
--- |
|
datasets: |
|
- cognitivecomputations/WizardLM_evol_instruct_V2_196k_unfiltered_merged_split |
|
- cognitivecomputations/Code-74k-ShareGPT-Vicuna |
|
- jondurbin/airoboros-3.1 |
|
- Norquinal/claude_multiround_chat_30k |
|
- Doctor-Shotgun/no-robots-sharegpt |
|
language: |
|
- en |
|
tags: |
|
- llama |
|
- llama 2 |
|
- smol_llama |
|
--- |
|
# smol_llama-220M-GQA-32k-theta-sft |
|
|
|
Experimental model meant to serve as a long-context speculative decoding model. |
|
|
|
Created using [Doctor-Shotgun/smol_llama-220M-GQA-32k-theta](https://huggingface.co/Doctor-Shotgun/smol_llama-220M-GQA-32k-theta) and finetuning at 32768 context length on several instruction datasets. |
|
|
|
This variant uses the rope theta (rope frequency base) method for context extension. |
|
|
|
The trained instruction format is Alpaca: |
|
``` |
|
### Instruction: |
|
{{instruction}} |
|
|
|
### Input: |
|
{{user input}} |
|
|
|
### Response: |
|
{{model response}} |
|
``` |
