---
license: mit
language:
- zh
---


---
 基于baichuan-inc/Baichuan-13B-Chat 做的GPTQ的量化，可直接加载，占用GPU约12G左右，用起来效果不错
---


调用代码：

----------------
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.generation.utils import GenerationConfig

model_dir = 'yinfupai/Baichuan-13B-Chat-GPTQ'

tokenizer = AutoTokenizer.from_pretrained(model_dir,trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_dir,device_map="auto",torch_dtype=torch.float16,trust_remote_code=True)

model.generation_config = GenerationConfig.from_pretrained(model_dir)
model.eval()

messages = []

#按baichuan要求的格式
messages.append({"role": "user", "content": "列举一下先天八卦的卦象"})

response = model.chat(tokenizer, messages)

print(response)

-----------------------

请注意模型的商用授权，请遵照baichuan-inc/Baichuan-13B-Chat的页面中的声明