Hugging Face's logo

mlinmg
/
SG-Raccoon-Yi-55B-200k

Text Generation
Transformers
Safetensors
llama
conversational
text-generation-inference
Inference Endpoints
Model card Files Community
2
SG-Raccoon-Yi-55B-200k / README.md
mlinmg's picture
mlinmg
Update README.md
18a9272
preview code
|
raw
history blame
1.82 kB
metadata
license: other
license_name: yi-license
license_link: https://huggingface.co/01-ai/Yi-34B/blob/main/LICENSE
language:
  - en,
pipeline_tag: conversational

SG Raccoon 55B

The first 55B auto-regressive causal LM created by combining 2x finetuned Yi 34b with 200K context into one.

Prompting Format 

SYSTEM: <ANY SYSTEM CONTEXT>
USER: 
ASSISTANT:

Merge process

The models used in the merge are Tess-M-v1.3 and Nous-Capybara-34B.

The layer ranges used are as follows:

- model: migtissera/Tess-M-v1.3
  layer_range: [0, 14]
- model: NousResearch/Nous-Capybara-34B
  layer_range: [7, 21]  
- model: migtissera/Tess-M-v1.3
  layer_range: [15, 29] 
- model: NousResearch/Nous-Capybara-34B
  layer_range: [22, 36] 
- model: migtissera/Tess-M-v1.3
  layer_range: [30, 44] 
- model: NousResearch/Nous-Capybara-34B
  layer_range: [37, 51]  
- model: migtissera/Tess-M-v1.3
  layer_range: [45, 59]

Benchmarks

Coming soon.

Acknowledgements

  • Special thanks to MSS for sponsoring this project

  • @chargoddard for developing the framework used to merge the model - mergekit.

  • Great thanks to @Undi95 for helping figuring out model merge options

  • Also credits to the 01-ai team for their amazing models

  • This merged model is inspired by Goliath 120B