YuJaq/llm-jp-3-13b-it_YuT-LoRA2

モデル概要

llm-jp/llm-jp-3-13bをベースとし、ichikara-instruction-003-001-1を用いてファインチューニングし、4bit量子化を行ったモデルです。

推論方法

必要なライブラリのインストール

%%capture
!pip install unsloth
!pip uninstall unsloth -y && pip install --upgrade --no-cache-dir "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
!pip install -U torch
!pip install -U peft

必要なライブラリの読込

from unsloth import FastLanguageModel
from peft import PeftModel
import torch
import json
from tqdm import tqdm
import re

model_id = "llm-jp/llm-jp-3-13b"
adapter_id = "YuJaq/llm-jp-3-13b-it_YuT-LoRA2"

HF_TOKEN = "HF_TOKEN"

dtype = None
load_in_4bit = True

# unslothのFastLanguageModelで元のモデルをロード
model, tokenizer = FastLanguageModel.from_pretrained(
  model_name=model_id,
  dtype=dtype,
  load_in_4bit = load_in_4bit,
  trust_remote_code = True,
)

# 元のモデルにLoRAのアダプタを統合
model = PeftModel.from_pretrained(model, adapter_id, token = HF_TOKEN)

# タスクとなるデータの読み込み。
# 事前にデータをアップロードしてください。
datasets = []
with open("./elyza-tasks-100-TV_0.jsonl", "r") as f:
  item = ""
  for line in f:
    line = line.strip()
    item += line
    if item.endswith("}"):
      datasets.append(json.loads(item))
      item = ""

# タスクの推論
# モデルのモードを変更
FastLanguageModel.for_inference(model)

results = []
for dt in tqdm(datasets):
  input = dt["input"]

  prompt = f"""### 指示\n{input}\n### 回答\n"""

  inputs = tokenizer([prompt], return_tensors = "pt").to(model.device)

  outputs = model.generate(**inputs, max_new_tokens = 512, use_cache = True, do_sample = False, repetition_penalty = 1.2)
  prediction = tokenizer.decode(outputs[0], skip_special_tokens = True).split('\n### 回答')[-1]

  results.append({"task_id": dt["task_id"], "input": input, "output": prediction})


# 結果をjsonlで保存
json_file_id = re.sub(".*/", "", adapter_id)
with open(f"/content/{json_file_id}_output.jsonl", 'w', encoding = 'utf-8') as f:
  for result in results:
    json.dump(result, f, ensure_ascii = False)
    f.write('\n')

---

base_model: llm-jp/llm-jp-3-13b tags: - text-generation-inference - transformers - unsloth - llama - trl license: apache-2.0 language: - en - ja

Uploaded model

• Developed by: YuJaq
• License: apache-2.0
• Finetuned from model : llm-jp/llm-jp-3-13b

This llama model was trained 2x faster with Unsloth and Huggingface's TRL library.