File size: 4,798 Bytes
a1466b1 e89d270 a1466b1 e89d270 4958a9e e8d5d89 e89d270 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 |
---
license: apache-2.0
base_model:
- meta-llama/Llama-3.2-3B
tags:
- llama-3.2
- thought-chain
- instruction-finetuning
- transformers
library_name: transformers
pipeline_tag: text-generation
---
# Thought-Ranked Llama 3.2 3B
## Model Description
This model is a fine-tuned version of Meta's Llama 3.2 3B (Base) that has been specially trained to generate high-quality thought processes before producing answers. The model underwent 4 rounds of specialized fine-tuning using a thought-chain ranking approach.
(Weekend project, just a few hundred steps of training)
### Training Process
1. **Initial Generation**: For each training sample, the model generates multiple thought chains by prefixing different thought tokens: `<thought>{char}</thought>` for each character in `[a-zA-Z0-9]`. Each thought chain is allowed up to 128 tokens.
2. **Answer Generation**: Following each thought chain, the model generates a complete answer with up to 2048 tokens.
3. **Ranking & Selection**: An external LLM ranking system evaluates the quality of answers without seeing the thought processes, creating a ranking of the most effective thought patterns.
4. **Final Training**: The model is then trained on the highest-ranked thought-answer pairs, learning to generate the most effective thought patterns autonomously.
### Key Features
- **Thought Chain Generation**: The model has learned to generate explicit thought processes before providing answers
- **Greedy Sampling**: Uses greedy sampling for both thought generation and final answers
- **Length Parameters**:
- Thought chains: Up to 128 tokens
- Final answers: Up to 2048 tokens
### Model Architecture
- Base model: Llama 3.2 3B (Base)
- Architecture: Transformer-based language model
- Parameters: ~3.2 billion
- Training Strategy: Supervised Fine-Tuning (SFT) with thought-chain ranking
## Intended Use
This model is designed for tasks that benefit from explicit reasoning chains, including but not limited to:
- Problem-solving
- Mathematical reasoning
- Logical deduction
- Step-by-step explanations
- Complex decision making
### Out-of-Scope Uses
- Direct deployment without safety measures
- Applications requiring guaranteed accuracy
- Critical decision-making without human oversight
- Tasks requiring capabilities beyond the base Llama 3.2 3B model
## Training Details
### Training Data
The model was trained using:
- Sample questions paired with multiple thought variations
- Thought chains generated using systematic character prefixes
- Rankings derived from LLM evaluation of answer quality
### Training Procedure
1. **Thought Generation Phase**
- Generated 62 variations of thoughts per sample (a-z, A-Z, 0-9)
- Sampled with temperature=0.0
- Maximum thought length: 128 tokens
2. **Answer Generation Phase**
- Generated completions following each thought chain
- Maximum answer length: 2048 tokens
- Sampled with temperature=0.0
3. **Ranking Phase**
- External LLM evaluated answer quality
- Ranking performed without access to thought chains
- Selected highest-performing thought-answer pairs
4. **Final Training Phase**
- Fine-tuned on best-performing thought-answer combinations
- 4 complete rounds of training
## Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("ericflo/Llama-3.2-3B-COT")
tokenizer = AutoTokenizer.from_pretrained("ericflo/Llama-3.2-3B-COT")
# Example usage
prompt = "Solve this math problem: 2x + 3 = 7"
input_ids = tokenizer.apply_chat_template(
[{"role": "user", "content": prompt}],
return_tensors="pt"
)
# Generate response with thought chain
output = model.generate(
input_ids,
temperature=1.0,
)
response = tokenizer.decode(output[0])
```
## Limitations
- Limited to the capabilities of the base Llama 3.2 3B model
- May generate thought chains that are not always optimal
- Performance depends on the quality of the LLM ranking system used during training
- Training process may not capture all possible effective thought patterns
- Limited by the context window of the base model
## Ethical Considerations
- The model inherits biases from the base Llama 3.2 3B model
- Generated thought chains should be reviewed for accuracy and appropriateness
- The model's reasoning process should not be relied upon for critical decisions without human verification
- Users should implement appropriate content filtering and safety measures
## Citation
If you use this model in your research, please cite:
```bibtex
@misc{thought-ranked-llama,
title={Thought-Ranked Llama 3.2: Fine-tuning Language Models with Ranked Thought Chains},
author={[Eric Florenzano]},
year={2024},
howpublished={\url{https://huggingface.co/ericflo/Llama-3.2-3B-COT}}
}
``` |