Update README.md
Browse files
README.md
CHANGED
|
@@ -5,4 +5,14 @@ datasets:
|
|
| 5 |
base_model:
|
| 6 |
- openchat/openchat-3.5-0106
|
| 7 |
library_name: transformers
|
| 8 |
-
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 5 |
base_model:
|
| 6 |
- openchat/openchat-3.5-0106
|
| 7 |
library_name: transformers
|
| 8 |
+
---
|
| 9 |
+
<p align="center">
|
| 10 |
+
<a href="https://ko-fi.com/pretergeek">Buy me a Ko-Fi</a> •
|
| 11 |
+
<a href="https://patreon.com/Pretergeek">Support my work using Patreon</a>
|
| 12 |
+
</p>
|
| 13 |
+
|
| 14 |
+
# OpenChat-3.5-0106_32K-PoSE
|
| 15 |
+
|
| 16 |
+
This model is [Openchat-3.5-0106](https://huggingface.co/openchat/openchat-3.5-0106) with the context length extended from 8192 tokens to 32768 tokens using [PoSE](https://huggingface.co/papers/2309.10400).
|
| 17 |
+
|
| 18 |
+
The model was fine-tuned using [Rank-Stabilized LoRA](https://huggingface.co/blog/damjan-k/rslora) and the [LongAlpaca-12K](Yukang/LongAlpaca-12k) dataset. I hope to continue extending the context in future versions and then apply the same methods to my [upscaled versions of OpenChat-3.5](https://huggingface.co/collections/Pretergeek/openchat-35-0106-with-additional-layers-66a8d3262c7c3ebdd7783a29).
|