File size: 8,680 Bytes
cefd69e 4975b81 cefd69e ad1aebb cefd69e 14cd9f3 cefd69e 3d00a29 14cd9f3 3d00a29 5272bef 3d00a29 14cd9f3 5272bef 3d00a29 5272bef cefd69e |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 |
---
license: apache-2.0
pipeline_tag: text-generation
datasets:
- aiplanet/buddhi-dataset
language:
- en
---
<p align="center" style="font-size:34px;"><b>Buddhi-128K-Chat</b></p>
# Buddhi-128K-Chat (7B) vLLM Inference: [](https://colab.research.google.com/drive/11_8W8FpKK-856QdRVJLyzbu9g-DMxNfg?usp=sharing)
# Read release article: [🔗 Introducing Buddhi: Open-Source Chat Model with a 128K Context Window 🔗 ](https://medium.aiplanet.com/introducing-buddhi-open-source-chat-model-with-a-128k-context-window-06a1848121d0)
## Model Description
Buddhi-128k-Chat is a general-purpose first chat model with 128K context length window. It is meticulously fine-tuned on the Mistral 7B Instruct, and optimised to handle an extended context length of up to 128,000 tokens using the innovative YaRN (Yet another Rope Extension) Technique. This enhancement allows Buddhi to maintain a deeper understanding of context in long documents or conversations, making it particularly adept at tasks requiring extensive context retention, such as comprehensive document summarization, detailed narrative generation, and intricate question-answering.
## Architecture
The Buddhi-128K-Chat model is fine-tuned on the Mistral-7B Instruct base model. We selected the Mistral 7B Instruct v0.2 as the parent model due to its superior reasoning capabilities. The architecture of the Mistral-7B model includes features like Grouped-Query Attention and Byte-fallback BPE tokenizer. Originally, this model has 32,768 maximum position embeddings. To increase the context size to 128K, we needed to modify the positional embeddings, which is where YaRN comes into play.
In our approach, we utilized the NTK-aware technique, which recommends alternative interpolation techniques for positional interpolation. One experimentation involved Dynamic-YARN, suggesting the dynamic value of the 's' scale factor. This is because during inference, the sequence length changes by 1 after every word prediction. By integrating these position embeddings with the Mistral-7B Instruct base model, we achieved the 128K model.
Additionally, we fine-tuned the model on our dataset to contribute one of the very few 128K chat-based models available in the open-source community with greater reasoning capabilities than all of it.
### Hardware requirements:
> For 128k Context Length
> - 80GB VRAM - A100 Preferred
> For 32k Context Length
> - 40GB VRAM - A100 Preferred
### vLLM - For Faster Inference
#### Installation
```
!pip install vllm
!pip install flash_attn # If Flash Attention 2 is supported by your System
```
Please check out [Flash Attention 2](https://github.com/Dao-AILab/flash-attention) Github Repository for more instructions on how to Install it.
**Implementation**:
> Note: The actual hardware requirements to run the model is roughly around 70GB VRAM. For experimentation, we are limiting the context length to 75K instead of 128K. This make it suitable for testing the model in 30-35 GB VRAM
```python
from vllm import LLM, SamplingParams
llm = LLM(
model='aiplanet/buddhi-128k-chat-7b',
trust_remote_code=True,
dtype = 'bfloat16',
gpu_memory_utilization=1,
max_model_len= 75000
)
prompts = [
"""<s> [INST] Please tell me a joke. [/INST] """,
"""<s> [INST] What is Machine Learning? [/INST] """
]
sampling_params = SamplingParams(
temperature=0.8,
top_p=0.95,
max_tokens=1000
)
outputs = llm.generate(prompts, sampling_params)
for output in outputs:
prompt = output.prompt
generated_text = output.outputs[0].text
print(generated_text)
print("\n\n")
# we have also attached a colab notebook, that contains: 2 more experimentations: Long Essay and Entire Book
```
For Output, do check out the colab notebook: [](https://colab.research.google.com/drive/11_8W8FpKK-856QdRVJLyzbu9g-DMxNfg?usp=sharing)
### Transformers - Basic Implementation
```python
import torch
import transformers
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_use_double_quant=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16
)
model_name = "aiplanet/Buddhi-128K-Chat"
model = AutoModelForCausalLM.from_pretrained(
model_name,
quantization_config=bnb_config,
device_map="sequential",
trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(
model,
trust_remote_code=True
)
prompt = "<s> [INST] Please tell me a small joke. [/INST] "
tokens = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(
**tokens,
max_new_tokens=100,
do_sample=True,
top_p=0.95,
temperature=0.8,
)
decoded_output = tokenizer.batch_decode(outputs.detach().cpu().numpy(), skip_special_tokens=True)[0]
print(f"Output:\n{decoded_output[len(prompt):]}")
```
Output
```
Output:
Why don't scientists trust atoms?
Because they make up everything.
```
## Prompt Template for Buddi-128-Chat
In order to leverage instruction fine-tuning, your prompt should be surrounded by [INST] and [/INST] tokens. The very first instruction should begin with a begin of sentence id. The next instructions should not. The assistant generation will be ended by the end-of-sentence token id.
```
"<s>[INST] What is your favourite condiment? [/INST]"
"Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!</s> "
"[INST] Do you have mayonnaise recipes? [/INST]"
```
# Benchmarks
### Long Context Benchmark
<strong>LongICLBench Banking77</strong>
<div>
| Model | 1R/2k | 2R/4K | 3R/7K | 4R/9K | 5R/14K |
|-----------------------------------------|-------|-------|-------|-------|--------|
| aiplanet/buddhi-128k-chat-7b | 47.8 | 60.8 | 57.8 | 62.4 | 57.2 |
| NousResearch/Yarn-Mistral-7b-128k | 31.6 | 68.6 | 68 | 47 | 65.6 |
| CallComply/zephyr-7b-beta-128k | 40.2 | 41.2 | 33.6 | 03 | 0 |
| Eric111/Yarn-Mistral-7b-128k-DPO | 28.6 | 62.8 | 58 | 41.6 | 59.8 |
</div>
<strong>Short Context Benchmark</strong>
<div>
| Model | # Params | Average | ARC (25-shot) | HellaSwag (10-shot) | Winogrande (5-shot) | TruthfulOA (0-shot) | MMLU (5-shot) |
|-----------------------------------|----------|---------|---------------|---------------------|---------------------|---------------------|---------------|
| aiplanet/buddhi-128k-chat-7b | 7B | 64.42 | 60.84 | 84 | 77.27 | 65.72 | 60.42 |
| migtissera/Tess-XS-vl-3-yarn-128K | 7B | 62.66 | 61.09 | 82.95 | 74.43 | 50.13 | 62.15 |
| migtissera/Tess-XS-v1-3-yarn-128K | 7B | 62.49 | 61.6 | 82.96 | 74.74 | 50.2 | 62.1 |
| Eric111/Yarn-Mistral-7b-128k-DPO | 7B | 60.15 | 60.84 | 82.99 | 78.3 | 43.55 | 63.09 |
| NousResearch/Yam-Mistral-7b-128k | 7B | 59.42 | 59.64 | 82.5 | 76.95 | 41.78 | 63.02 |
| CallComply/openchat-3.5-0106-128k | 7B | 59.38 | 64.25 | 77.31 | 77.66 | 46.5 | 57.58 |
| CallComply/zephyr-7b-beta-128k | 7B | 54.45 | 58.28 | 81 | 74.74 | 46.1 | 53.57 |
</div>
## Get in Touch
You can schedule a 1:1 meeting with our DevRel & Community Team to get started with AI Planet Open Source LLMs and GenAI Stack. Schedule the call here: [https://calendly.com/jaintarun](https://calendly.com/jaintarun)
Stay tuned for more updates and be a part of the coding evolution. Join us on this exciting journey as we make AI accessible to all at AI Planet!
### Framework versions
- Transformers 4.39.2
- Pytorch 2.2.1+cu121
- Datasets 2.18.0
- Accelerate 0.27.2
- flash_attn 2.5.6
### Citation
```
@misc {Chaitanya890, lucifertrj ,
author = { Chaitanya Singhal, Tarun Jain },
title = { Buddhi-128k-Chat by AI Planet},
year = 2024,
url = { https://huggingface.co/aiplanet//Buddhi-128K-Chat },
publisher = { Hugging Face }
}
``` |