Model Card

Description

Uses

Use the code below to get started with the model.

from peft import PeftModel
from unsloth import FastLanguageModel

BASE_MODEL_ID = "llm-jp/llm-jp-3-13b"
ADAPTER_ID = "nkmry/llmjp-13b-comp"

# Get the base model

base_model, tokenizer = FastLanguageModel.from_pretrained(
    model_name=BASE_MODEL_ID,
    dtype=None,
    load_in_4bit=True,
    trust_remote_code=True,
)

# Apply the LoRA adapter

model = PeftModel.from_pretrained(
    base_model,
    model_id=ADAPTER_ID,
    is_trainable=False
)

# Generate outputs

datasets = []
with open("./elyza-tasks-100-TV_0.jsonl", "r") as f:
    item = ""
    for line in f:
      line = line.strip()
      item += line
      if item.endswith("}"):
        datasets.append(json.loads(item))
        item = ""

PROMPT = """### 指示
{input}
### 回答
"""

results = []
FastLanguageModel.for_inference(model)
for data in tqdm(datasets):
    input = data["input"]
    tokenized_input = tokenizer.encode(
        PROMPT.format(input=input), add_special_tokens=False, return_tensors="pt").to(model.device)
    attention_mask = torch.ones_like(tokenized_input)
    with torch.no_grad():
        outputs = model.generate(
            tokenized_input,
            attention_mask=attention_mask,
            max_new_tokens=512,
            do_sample=False,
            repetition_penalty=1.2,
            pad_token_id=tokenizer.eos_token_id
        )[0]
    output = tokenizer.decode(outputs[tokenized_input.size(1):], skip_special_tokens=True)
    results.append({"task_id": data["task_id"], "input": input, "output": output})

with open(f"./outputs.jsonl", 'w', encoding='utf-8') as f:
    for result in results:
        json.dump(result, f, ensure_ascii=False)
        f.write('\n')

Details

Training Data

Framework versions

  • PyTorch 2.5.1
  • Transformers 4.46.3
  • Unsloth 2024.12.4
  • PEFT 0.14.0
  • xformers 0.0.2
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model is not currently available via any of the supported Inference Providers.
The model cannot be deployed to the HF Inference API: The model has no pipeline_tag.

Model tree for nkmry/llmjp-13b-comp

Finetuned
(1120)
this model