The model outputs `!!!!!!!!!!!!!!!!!!!!` only

by seungduk - opened May 26

May 26

Hi there,

Thanks for sharing a good model. I wanted to test it out but I failed to get the expected output from the model. Can you point me out what I did wrong?

Code:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "nayohan/llama3-instrucTrans-enko-8b"

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_id)

system_prompt = "당신은 번역기 입니다. 영어를 한국어로 번역하세요."
sentence = "The aerospace industry is a flower in the field of technology and science."
conversation = [{'role': 'system', 'content': system_prompt},
                {'role': 'user', 'content': sentence}]

encoded_inputs = tokenizer.apply_chat_template(
  conversation,
  tokenize=True,
  add_generation_prompt=True,
  return_tensors='pt'
)

input_ids = encoded_inputs.to('cuda')
attention_mask = torch.ones_like(input_ids)

with torch.no_grad():
    outputs = model.generate(input_ids=input_ids, attention_mask=attention_mask
                             , max_new_tokens=46) # Finetuned with length 4096
    print(tokenizer.decode(outputs[0]))

Output:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

당신은 번역기 입니다. 영어를 한국어로 번역하세요.<|eot_id|><|start_header_id|>user<|end_header_id|>

The aerospace industry is a flower in the field of technology and science.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Thanks,
Seungduk

ehottl

May 30

I can not re-generate your problems.
In my case, it works beautifully.

Loading checkpoint shards: 100%|██████████| 4/4 [00:01<00:00,  2.39it/s]
WARNING:root:Some parameters are on the meta device device because they were offloaded to the cpu.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.
<|begin_of_text|><|start_header_id|>system<|end_header_id|>

당신은 번역기 입니다. 영어를 한국어로 번역하세요.<|eot_id|><|start_header_id|>user<|end_header_id|>

The aerospace industry is a flower in the field of technology and science.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

항공우주 산업은 기술과 과학 분야의 꽃입니다.<|eot_id|>

nayohan

Owner May 30

•

edited May 30

I also got the same result as above. even though I used your fixed code.

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

당신은 번역기 입니다. 영어를 한국어로 번역하세요.<|eot_id|><|start_header_id|>user<|end_header_id|>

The aerospace industry is a flower in the field of technology and science.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

항공우주 산업은 기술과 과학 분야의 꽃입니다.<|eot_id|>

nayohan

Owner May 30

I'll share my environment settings, would you like to try this?

--find-links https://download.pytorch.org/whl/torch_stable.html
torch==2.2.1+cu121
transformers==4.40.1
datasets==2.18.0
accelerate==0.26.1
deepspeed==0.14.0
peft==0.10.0
trl==0.8.6
pyarrow==15.0.0
# flash-attn==2.5.6
bitsandbytes==0.43.1

ehottl

Jun 3

My guess is that you installed the package using pip. Problems caused by pip are ambiguous and hard to solve. If you can, try installing using the conda package manager and the conda-forge channel.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment