File size: 2,409 Bytes
ff4222c f7b0efa e7a87a4 f7b0efa ff4222c f7b0efa ff4222c f7b0efa ff4222c 164d3c9 f7b0efa ff4222c f7b0efa ff4222c 774424e |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 |
---
language:
- zh
- en
---
<!-- 标题 -->
<h1 align="center">ChatYuan-7B-merge</h1>
<!-- 图标 -->
<p align="center">
<a href="https://space.bilibili.com/28606893?spm_id_from=333.1007.0.0">
bilibili
</a>
<a href="https://github.com/tiansztiansz">
github
</a>
<a href="https://www.kaggle.com/tiansztianszs">
kaggle
</a>
<a href="https://huggingface.co/tiansz">
huggingface
</a>
</p>
<!-- 项目介绍 -->
<p align="center">Based on LLAMA's latest Chinese-English dialogue language large model</p>
<br>
You can see more detail in this [repo](https://github.com/clue-ai/ChatYuan-7B)
<br>
## How to use
```python
from transformers import LlamaForCausalLM, AutoTokenizer
import torch
ckpt = "tiansz/ChatYuan-7B-merge"
device = torch.device('cuda')
model = LlamaForCausalLM.from_pretrained(ckpt)
tokenizer = AutoTokenizer.from_pretrained(ckpt)
def answer(prompt):
prompt = f"用户：{prompt}\n小元："
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)
generate_ids = model.generate(input_ids, max_new_tokens=1024, do_sample = True, temperature = 0.7)
output = tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
response = output[len(prompt):]
return response
result = answer("你好")
print(result)
```
<br>
int8:
```python
from transformers import LlamaForCausalLM, AutoTokenizer
import torch
ckpt = "tiansz/ChatYuan-7B-merge"
device = torch.device('cuda')
max_memory = f'{int(torch.cuda.mem_get_info()[0]/1024**3)-1}GB'
n_gpus = torch.cuda.device_count()
max_memory = {i: max_memory for i in range(n_gpus)}
model = LlamaForCausalLM.from_pretrained(ckpt, device_map='auto', load_in_8bit=True, max_memory=max_memory)
tokenizer = AutoTokenizer.from_pretrained(ckpt)
def answer(prompt):
prompt = f"用户：{prompt}\n小元："
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)
generate_ids = model.generate(input_ids, max_new_tokens=1024, do_sample = True, temperature = 0.7)
output = tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
response = output[len(prompt):]
return response
result = answer("你好")
print(result)
```
<br>
## License
- [ChatYuan-7B](https://github.com/clue-ai/ChatYuan-7B)
- [llama](https://github.com/facebookresearch/llama) |