---
license: other
language:
- zh
- en
metrics:
- accuracy
library_name: transformers
pipeline_tag: text-generation
---

## Inference

```
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
  'Nanbeige/Nanbeige2-8B-Chat',
  use_fast=False,
  trust_remote_code=True
)
model = AutoModelForCausalLM.from_pretrained(
  'Nanbeige/Nanbeige2-8B-Chat',
  torch_dtype='auto',
  device_map='auto',
  trust_remote_code=True
)

messages = [
  {'role': 'user', 'content': 'Hello'}
]
prompt = tokenizer.apply_chat_template(
  messages,
  add_generation_prompt=True,
  tokenize=False
)
input_ids = tokenizer(prompt, add_special_tokens=False, return_tensors='pt').input_ids
output_ids = model.generate(input_ids.to('cuda'))
resp = tokenizer.decode(output_ids[0][len(input_ids[0]):], skip_special_tokens=True)

print(resp)
```