Quantization made by Richard Erkhov.

Faro-Yi-34B - GGUF

Model creator: https://huggingface.co/wenbopan/
Original model: https://huggingface.co/wenbopan/Faro-Yi-34B/

Name	Quant method	Size
Faro-Yi-34B.Q2_K.gguf	Q2_K	11.94GB
Faro-Yi-34B.IQ3_XS.gguf	IQ3_XS	13.26GB
Faro-Yi-34B.IQ3_S.gguf	IQ3_S	13.99GB
Faro-Yi-34B.Q3_K_S.gguf	Q3_K_S	13.93GB
Faro-Yi-34B.IQ3_M.gguf	IQ3_M	14.5GB
Faro-Yi-34B.Q3_K.gguf	Q3_K	15.51GB
Faro-Yi-34B.Q3_K_M.gguf	Q3_K_M	15.51GB
Faro-Yi-34B.Q3_K_L.gguf	Q3_K_L	16.89GB
Faro-Yi-34B.IQ4_XS.gguf	IQ4_XS	17.36GB
Faro-Yi-34B.Q4_0.gguf	Q4_0	18.13GB
Faro-Yi-34B.IQ4_NL.gguf	IQ4_NL	18.3GB
Faro-Yi-34B.Q4_K_S.gguf	Q4_K_S	18.25GB
Faro-Yi-34B.Q4_K.gguf	Q4_K	19.24GB
Faro-Yi-34B.Q4_K_M.gguf	Q4_K_M	19.24GB
Faro-Yi-34B.Q4_1.gguf	Q4_1	20.1GB
Faro-Yi-34B.Q5_0.gguf	Q5_0	22.08GB
Faro-Yi-34B.Q5_K_S.gguf	Q5_K_S	22.08GB
Faro-Yi-34B.Q5_K.gguf	Q5_K	22.65GB
Faro-Yi-34B.Q5_K_M.gguf	Q5_K_M	22.65GB
Faro-Yi-34B.Q5_1.gguf	Q5_1	24.05GB
Faro-Yi-34B.Q6_K.gguf	Q6_K	26.28GB
Faro-Yi-34B.Q8_0.gguf	Q8_0	34.03GB

Original model description:

license: mit datasets: - wenbopan/Fusang-v1 - wenbopan/OpenOrca-zh-20k language: - zh - en

The Faro chat model focuses on practicality and long-context modeling. It handles various downstream tasks with higher quality, delivering stable and reliable results even when inputs contain lengthy documents or complex instructions. Faro seamlessly works in both English and Chinese.

Faro-Yi-34B

Faro-Yi-34B is an improved Yi-34B-200K with extensive instruction tuning on Fusang-V1. Compared to Yi-34B-200K, Faro-Yi-34B has gained greater capability in various downstream tasks and long-context modeling thanks to the large-scale synthetic data in Fusang-V1.

Just like Yi-34B-200K, Faro-Yi-34B supports up to 200K context length.

How to Use

Faro-Yi-9B-200K uses chatml template. I recommend using vLLM for long inputs.

import io
import requests
from PyPDF2 import PdfReader
from vllm import LLM, SamplingParams

llm = LLM(model="wenbopan/Faro-Yi-34B")

pdf_data = io.BytesIO(requests.get("https://arxiv.org/pdf/2303.08774.pdf").content)
document = "".join(page.extract_text() for page in PdfReader(pdf_data).pages) # 100 pages

question = f"{document}\n\nAccording to the paper, what is the parameter count of GPT-4?"
messages = [ {"role": "user", "content": question} ] # 83K tokens
prompt = llm.get_tokenizer().apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
output = llm.generate(prompt, SamplingParams(temperature=0.8, max_tokens=500))
print(output[0].outputs[0].text)
# Yi-9B-200K:      175B. GPT-4 has 175B \nparameters. How many models were combined to create GPT-4? Answer: 6. ...
# Faro-Yi-9B-200K: GPT-4 does not have a publicly disclosed parameter count due to the competitive landscape and safety implications of large-scale models like GPT-4. ...

Or With Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('wenbopan/Faro-Yi-34B', device_map="cuda")
tokenizer = AutoTokenizer.from_pretrained('wenbopan/Faro-Yi-34B')
messages = [
    {"role": "system", "content": "You are a helpful assistant. Always answer with a short response."},
    {"role": "user", "content": "Tell me what is Pythagorean theorem like you are a pirate."}
]

input_ids = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt").to(model.device)
generated_ids = model.generate(input_ids, max_new_tokens=512, temperature=0.5)
response = tokenizer.decode(generated_ids[0], skip_special_tokens=True) # Aye, matey! The Pythagorean theorem is a nautical rule that helps us find the length of the third side of a triangle. ...

For more info please refer to wenbopan/Faro-Yi-9B