|
--- |
|
base_model: |
|
- unsloth/Llama-3.2-1B-Instruct |
|
library_name: transformers |
|
language: |
|
- en |
|
license: cc0-1.0 |
|
tags: |
|
- unsloth |
|
--- |
|
# A !!!!!disclaimer uh. for now, the experimentation does not lead me anywhere due to limit resources that I have and do not recommend to download this model. Working on working on it. |
|
|
|
PEFT Finnegan-tuned LLaMA 3.2-1B-instruct on part of Finnegans Wake dataset for text generation in the style of James Joyce. |
|
|
|
Space: https://huggingface.co/spaces/genaforvena/huivam_finnegans_spaceship |
|
|
|
## Iteration 3: |
|
Realized that was doing it all wrong and this tie used https://huggingface.co/unsloth/Llama-3.2-1B-Instruct and collab available from there. Only changed dataset. |
|
|
|
My collab is here: https://colab.research.google.com/drive/1JrqcU9idXXR3Wru5mw2e6Uh2TKJWwu7U?usp=sharing |
|
|
|
The only difference: Created dataset like below |
|
``` |
|
from unsloth.chat_templates import get_chat_template |
|
import json |
|
import random |
|
from transformers import AutoTokenizer |
|
from unsloth.chat_templates import get_chat_template # For chat template formatting |
|
from datasets import Dataset, load_dataset |
|
|
|
# Configuration |
|
INPUT_FILE = "finnegans_30.txt" # Path to your Finnegans Wake text file |
|
OUTPUT_FILE = "finnegans_wake_dataset.jsonl" # Local file to save the dataset |
|
CHUNK_SIZE = 24 |
|
|
|
# Apply the chat template |
|
tokenizer = get_chat_template( |
|
tokenizer, |
|
chat_template="llama-3.1", # Use the LLaMA-3.1 chat template |
|
) |
|
|
|
# Load the text |
|
with open(INPUT_FILE, "r", encoding="utf-8") as file: |
|
text = file.read() |
|
|
|
# Tokenize the text |
|
tokens = tokenizer.encode(text, truncation=False, add_special_tokens=False) |
|
|
|
# Split tokens into chunks |
|
chunks = [tokens[i:i + CHUNK_SIZE] for i in range(0, len(tokens), CHUNK_SIZE)] |
|
|
|
# Prepare dataset in conversational format |
|
dataset = [] |
|
for chunk in chunks: |
|
chunk_text = tokenizer.decode(chunk, skip_special_tokens=True) |
|
|
|
# Split the chunk into three parts randomly |
|
split_points = sorted(random.sample(range(len(chunk_text)), 2)) # Two random split points |
|
context = chunk_text[:split_points[0]] |
|
instruction = chunk_text[split_points[0]:split_points[1]] |
|
response = chunk_text[split_points[1]:] |
|
|
|
# Format as a conversation |
|
conversation = [ |
|
{"role": "user", "content": f"### GIVEN THE CONTEXT: {context} ### INSTRUCTION: {instruction}"}, |
|
{"role": "assistant", "content": response}, |
|
] |
|
|
|
# Add to dataset |
|
dataset.append({"conversations": conversation}) |
|
|
|
# Save dataset locally as a .jsonl file |
|
with open(OUTPUT_FILE, "w", encoding="utf-8") as file: |
|
for item in dataset: |
|
json.dump(item, file) |
|
file.write("\n") |
|
|
|
print(f"Dataset saved locally to {OUTPUT_FILE}") |
|
|
|
# Apply the formatting function |
|
def formatting_prompts_func(examples): |
|
convos = examples["conversations"] |
|
texts = [tokenizer.apply_chat_template(convo, tokenize=False, add_generation_prompt=False) for convo in convos] |
|
return {"text": texts} |
|
|
|
# Apply the formatting function using Dataset.from_dict |
|
dataset = Dataset.from_dict({"conversations": [d['conversations'] for d in dataset]}) |
|
|
|
formatted_dataset = dataset.map(formatting_prompts_func, batched=True, remove_columns=['conversations']) |
|
|
|
# Save the formatted dataset |
|
formatted_dataset.to_json("formatted_finnegans_wake_dataset.jsonl") |
|
print("Formatted dataset saved to formatted_finnegans_wake_dataset.jsonl") |
|
|
|
# Load the formatted dataset using load_dataset |
|
dataset = load_dataset("json", data_files="formatted_finnegans_wake_dataset.jsonl", split="train") |
|
dataset = dataset |
|
``` |
|
|
|
## Iteration 2 (Fail): |
|
|
|
Dataset: same (forgot to save config with new dataset). |
|
|
|
finnetune.yaml: |
|
``` |
|
# The ID of the dataset you created |
|
dataset: huivam-finnegans-2 |
|
|
|
# Configuration for text completion fine-tuning |
|
text_completion: |
|
# How the fields of the JSON dataset should be formatted into the input text |
|
input_template: "### GIVEN THE CONTEXT: {context} ### INSTRUCTION: {instruction} ### RESPONSE IS: " |
|
|
|
# How the fields of the JSON dataset should be formatted into the output text |
|
output_template: "ANSWER: {response}" |
|
|
|
# The Fireworks model name of the base model |
|
base_model: accounts/fireworks/models/llama-v3p2-1b-instruct |
|
``` |
|
|
|
Finne-tuning commands used: |
|
``` |
|
./firectl create dataset huivam-finnegans-2 .\finnegans_wake_dataset_2.jsonl |
|
./firectl create fine-tuning-job --settings-file finnetune.yaml --epochs=3 --learning-rate=2e-5 --batch-size=8 |
|
``` |
|
|
|
New params used to finne-tune: |
|
``` |
|
Text Completion: |
|
Input Template: ### GIVEN THE CONTEXT: {context} ### INSTRUCTION: {instruction} ### RESPONSE IS: |
|
Output Template: ANSWER: {response} |
|
Base Model: accounts/fireworks/models/llama-v3p2-1b-instruct |
|
Epochs: 3 |
|
Learning Rate: 2e-05 |
|
Lora Rank: 8 |
|
Batch Size: 8 |
|
Evaluation Split: 0 |
|
``` |
|
|
|
Spent: $0.08 |
|
Time: 5 mins |
|
|
|
## Iteration 1: |
|
|
|
Dataset I prepared like that: |
|
``` |
|
# Load the tokenizer |
|
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME) |
|
|
|
# Load the text |
|
with open(INPUT_FILE, "r", encoding="utf-8") as file: |
|
text = file.read() |
|
|
|
# Tokenize the text |
|
tokens = tokenizer.encode(text, truncation=False, add_special_tokens=False) |
|
|
|
# Split tokens into chunks |
|
chunks = [tokens[i:i + CHUNK_SIZE] for i in range(0, len(tokens), CHUNK_SIZE)] |
|
|
|
# Prepare dataset |
|
dataset = [] |
|
for chunk in chunks: |
|
chunk_text = tokenizer.decode(chunk, skip_special_tokens=True) |
|
|
|
# Split the chunk into three parts randomly |
|
split_points = sorted(random.sample(range(len(chunk_text)), 2)) # Two random split points |
|
context = chunk_text[:split_points[0]] |
|
instruction = chunk_text[split_points[0]:split_points[1]] |
|
response = chunk_text[split_points[1]:] |
|
|
|
# Add to dataset |
|
dataset.append({ |
|
"context": context, |
|
"instruction": instruction, |
|
"response": response, |
|
}) |
|
|
|
# Save dataset locally as a .jsonl file |
|
with open(OUTPUT_FILE, "w", encoding="utf-8") as file: |
|
for item in dataset: |
|
json.dump(item, file) |
|
file.write("\n") |
|
|
|
print(f"Dataset saved locally to {OUTPUT_FILE}") |
|
``` |
|
|
|
Example of dataset entry: |
|
``` |
|
{"context": "riverrun, past Eve and Adam's, from swerve of shore to bend of bay...", "instruction": "Sir Tristram, violer d'amores, fr'over the short sea...", "response": "O here here how hoth sprowled met the duskt the father of fornicationists..."} |
|
``` |
|
|
|
fine-tuned on 1/10th of text on fireworks.ai with params: |
|
``` |
|
dataset: finnegans_wake_dataset |
|
|
|
text_completion: |
|
# How the fields of the JSON dataset should be formatted into the input text |
|
input_template: "### GIVEN THE CONTEXT: {context} ### INSTRUCTION: {instruction} ### RESPONSE IS: " |
|
|
|
# How the fields of the JSON dataset should be formatted into the output text |
|
output_template: "ANSWER: {response}" |
|
|
|
# The Fireworks model name of the base model |
|
base_model: accounts/fireworks/models/llama-v3p2-1b |
|
|
|
# Hyperparameters for fine-tuning (should be passed as args and removed from here) |
|
hyperparameters: |
|
learning_rate: 1e-5 # Learning rate for the optimizer |
|
epochs: 1 # Number of epochs to train |
|
batch_size: 4 # Batch size for training |
|
``` |
|
|
|
Spent: $0.01 |
|
Time: 2 mins |
|
|
|
Result: Seemingly not enough data to affect model output. |
