![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "# 🦙🎛️ LLaMA-LoRA Tuner\n",
+ "\n",
+ "TL;DR: **Runtime > Run All** (`⌘/Ctrl+F9`). Takes about 5 minutes to start. You will be promped to authorize Google Drive access."
+ ],
+ "metadata": {
+ "id": "bb4nzBvLfZUj"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "#@markdown To prevent Colab from disconnecting you, here is a music player that will loop infinitely (it's silent):\n",
+ "%%html\n",
+ ""
+ ],
+ "metadata": {
+ "id": "DwarOgXbG77C",
+ "cellView": "form"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "# @title A small workaround { display-mode: \"form\" }\n",
+ "# @markdown Don't panic if you see an error here. Just click the `RESTART RUNTIME` button in the output below, then Run All again.\n",
+ "# @markdown The error will disappear on the next run.\n",
+ "%pip install Pillow==9.3.0 numpy==1.23.5\n",
+ "\n",
+ "import pkg_resources as r\n",
+ "import PIL\n",
+ "import numpy\n",
+ "for module, min_version in [(PIL, \"9.3\"), (numpy, \"1.23\")]:\n",
+ " lib_version = r.parse_version(module.__version__)\n",
+ " print(module.__name__, lib_version)\n",
+ " if lib_version < r.parse_version(min_version):\n",
+ " raise Exception(\"Restart the runtime by clicking the 'RESTART RUNTIME' button above (or Runtime > Restart Runtime).\")"
+ ],
+ "metadata": {
+ "id": "XcJ4WO3KhOX1"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "# Config\n",
+ "\n",
+ "Some configurations to run this notebook. "
+ ],
+ "metadata": {
+ "id": "5uS5jJ8063f_"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "# @title Git/Project { display-mode: \"form\", run: \"auto\" }\n",
+ "# @markdown Project settings.\n",
+ "\n",
+ "# @markdown The URL of the LLaMA-LoRA-Tuner project
+
+Making evaluating and fine-tuning LLaMA models with low-rank adaptation (LoRA) easy.
+
+
+## Features
+
+**[See a demo on Hugging Face](https://huggingface.co/spaces/zetavg/LLaMA-LoRA-UI-Demo)** **Only serves UI demonstration. To try training or text generation, [run on Colab](#run-on-google-colab).*
+
+* **[1-click up and running in Google Colab](#run-on-google-colab)** with a standard GPU runtime.
+ * Loads and stores data in Google Drive.
+* Evaluate various LLaMA LoRA models stored in your folder or from Hugging Face.
+* Switch between base models such as `decapoda-research/llama-7b-hf`, `nomic-ai/gpt4all-j`, `databricks/dolly-v2-7b`, `EleutherAI/gpt-j-6b`, or `EleutherAI/pythia-6.9b`.
+* Fine-tune LLaMA models with different prompt templates and training dataset format.
+ * Load JSON and JSONL datasets from your folder, or even paste plain text directly into the UI.
+ * Supports Stanford Alpaca [seed_tasks](https://github.com/tatsu-lab/stanford_alpaca/blob/main/seed_tasks.jsonl), [alpaca_data](https://github.com/tatsu-lab/stanford_alpaca/blob/main/alpaca_data.json) and [OpenAI "prompt"-"completion"](https://platform.openai.com/docs/guides/fine-tuning/data-formatting) format.
+ * Use prompt templates to keep your dataset DRY.
+
+
+## How to Start
+
+There are various ways to run this app:
+
+* **[Run on Google Colab](#run-on-google-colab)**: The simplest way to get started, all you need is a Google account. Standard (free) GPU runtime is sufficient to run generation and training with micro batch size of 8. However, the text generation and training is much slower than on other cloud services, and Colab might terminate the execution in inactivity while running long tasks.
+* **[Run on a cloud service via SkyPilot](#run-on-a-cloud-service-via-skypilot)**: If you have a cloud service (Lambda Labs, GCP, AWS, or Azure) account, you can use SkyPilot to run the app on a cloud service. A cloud bucket can be mounted to preserve your data.
+* **[Run locally](#run-locally)**: Depends on the hardware you have.
+
+### Run On Google Colab
+
+*See [video](https://youtu.be/lByYOMdy9h4) for step-by-step instructions.*
+
+Open [this Colab Notebook](https://colab.research.google.com/github/zetavg/LLaMA-LoRA-Tuner/blob/main/LLaMA_LoRA.ipynb) and select **Runtime > Run All** (`⌘/Ctrl+F9`).
+
+You will be prompted to authorize Google Drive access, as Google Drive will be used to store your data. See the "Config"/"Google Drive" section for settings and more info.
+
+After approximately 5 minutes of running, you will see the public URL in the output of the "Launch"/"Start Gradio UI 🚀" section (like `Running on public URL: https://xxxx.gradio.live`). Open the URL in your browser to use the app.
+
+### Run on a cloud service via SkyPilot
+
+After following the [installation guide of SkyPilot](https://skypilot.readthedocs.io/en/latest/getting-started/installation.html), create a `.yaml` to define a task for running the app:
+
+```yaml
+# llm-tuner.yaml
+
+resources:
+ accelerators: A10:1 # 1x NVIDIA A10 GPU, about US$ 0.6 / hr on Lambda Cloud. Run `sky show-gpus` for supported GPU types, and `sky show-gpus [GPU_NAME]` for the detailed information of a GPU type.
+ cloud: lambda # Optional; if left out, SkyPilot will automatically pick the cheapest cloud.
+
+file_mounts:
+ # Mount a presisted cloud storage that will be used as the data directory.
+ # (to store train datasets trained models)
+ # See https://skypilot.readthedocs.io/en/latest/reference/storage.html for details.
+ /data:
+ name: llm-tuner-data # Make sure this name is unique or you own this bucket. If it does not exists, SkyPilot will try to create a bucket with this name.
+ store: s3 # Could be either of [s3, gcs]
+ mode: MOUNT
+
+# Clone the LLaMA-LoRA Tuner repo and install its dependencies.
+setup: |
+ conda create -q python=3.8 -n llm-tuner -y
+ conda activate llm-tuner
+
+ # Clone the LLaMA-LoRA Tuner repo and install its dependencies
+ [ ! -d llm_tuner ] && git clone https://github.com/zetavg/LLaMA-LoRA-Tuner.git llm_tuner
+ echo 'Installing dependencies...'
+ pip install -r llm_tuner/requirements.lock.txt
+
+ # Optional: install wandb to enable logging to Weights & Biases
+ pip install wandb
+
+ # Optional: patch bitsandbytes to workaround error "libbitsandbytes_cpu.so: undefined symbol: cget_col_row_stats"
+ BITSANDBYTES_LOCATION="$(pip show bitsandbytes | grep 'Location' | awk '{print $2}')/bitsandbytes"
+ [ -f "$BITSANDBYTES_LOCATION/libbitsandbytes_cpu.so" ] && [ ! -f "$BITSANDBYTES_LOCATION/libbitsandbytes_cpu.so.bak" ] && [ -f "$BITSANDBYTES_LOCATION/libbitsandbytes_cuda121.so" ] && echo 'Patching bitsandbytes for GPU support...' && mv "$BITSANDBYTES_LOCATION/libbitsandbytes_cpu.so" "$BITSANDBYTES_LOCATION/libbitsandbytes_cpu.so.bak" && cp "$BITSANDBYTES_LOCATION/libbitsandbytes_cuda121.so" "$BITSANDBYTES_LOCATION/libbitsandbytes_cpu.so"
+ conda install -q cudatoolkit -y
+
+ echo 'Dependencies installed.'
+
+ # Optional: Install and setup Cloudflare Tunnel to expose the app to the internet with a custom domain name
+ [ -f /data/secrets/cloudflared_tunnel_token.txt ] && echo "Installing Cloudflare" && curl -L --output cloudflared.deb https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64.deb && sudo dpkg -i cloudflared.deb && sudo cloudflared service uninstall || : && sudo cloudflared service install "$(cat /data/secrets/cloudflared_tunnel_token.txt | tr -d '\n')"
+
+ # Optional: pre-download models
+ echo "Pre-downloading base models so that you won't have to wait for long once the app is ready..."
+ python llm_tuner/download_base_model.py --base_model_names='decapoda-research/llama-7b-hf,nomic-ai/gpt4all-j'
+
+# Start the app. `hf_access_token`, `wandb_api_key` and `wandb_project` are optional.
+run: |
+ conda activate llm-tuner
+ python llm_tuner/app.py \
+ --data_dir='/data' \
+ --hf_access_token="$([ -f /data/secrets/hf_access_token.txt ] && cat /data/secrets/hf_access_token.txt | tr -d '\n')" \
+ --wandb_api_key="$([ -f /data/secrets/wandb_api_key.txt ] && cat /data/secrets/wandb_api_key.txt | tr -d '\n')" \
+ --wandb_project='llm-tuner' \
+ --timezone='Atlantic/Reykjavik' \
+ --base_model='decapoda-research/llama-7b-hf' \
+ --base_model_choices='decapoda-research/llama-7b-hf,nomic-ai/gpt4all-j,databricks/dolly-v2-7b' \
+ --share
+```
+
+Then launch a cluster to run the task:
+
+```
+sky launch -c llm-tuner llm-tuner.yaml
+```
+
+`-c ...` is an optional flag to specify a cluster name. If not specified, SkyPilot will automatically generate one.
+
+You will see the public URL of the app in the terminal. Open the URL in your browser to use the app.
+
+Note that exiting `sky launch` will only exit log streaming and will not stop the task. You can use `sky queue --skip-finished` to see the status of running or pending tasks, `sky logs "
+ui_dev_mode_title_prefix: ""
+ui_show_sys_info: false
+
+# WandB
+# enable_wandb: false
+# wandb_api_key: ""
+# default_wandb_project: LLM-Tuner
+
+# Special Modes
+ui_dev_mode: true
diff --git a/config.yaml.sample b/config.yaml.sample
new file mode 100644
index 0000000000000000000000000000000000000000..e063d5f884c7461849976f694ef657d57ad72e64
--- /dev/null
+++ b/config.yaml.sample
@@ -0,0 +1,29 @@
+server_name: 0.0.0.0
+
+# Basic Configurations
+data_dir: ./data
+default_base_model_name: decapoda-research/llama-7b-hf
+base_model_choices:
+ - decapoda-research/llama-7b-hf
+ - nomic-ai/gpt4all-j
+load_8bit: false
+trust_remote_code: false
+
+# timezone: Atlantic/Reykjavik
+
+# auth_username: username
+# auth_password: password
+
+# UI Customization
+# ui_title: LLM Tuner
+# ui_emoji: 🦙🎛️
+# ui_subtitle: Have fun!
+# ui_show_sys_info: true
+
+# WandB
+# enable_wandb: false
+# wandb_api_key: ""
+# default_wandb_project: LLM-Tuner
+
+# Special Modes
+ui_dev_mode: false
diff --git a/download_base_model.py b/download_base_model.py
new file mode 100644
index 0000000000000000000000000000000000000000..fa9576e1d3c20acc9b7df72a78d3e8569bf7cb98
--- /dev/null
+++ b/download_base_model.py
@@ -0,0 +1,33 @@
+import fire
+
+from huggingface_hub import snapshot_download
+
+
+def main(
+ base_model_names: str = "",
+):
+ '''
+ Download and cache base models form Hugging Face.
+
+ :param base_model_names: Names of the base model you want to download, seperated by ",". For example: 'decapoda-research/llama-7b-hf,nomic-ai/gpt4all-j'.
+ '''
+
+ assert (
+ base_model_names
+ ), "Please specify --base_model_names, e.g. --base_model_names='decapoda-research/llama-7b-hf,nomic-ai/gpt4all-j'"
+
+ base_model_names_list = base_model_names.split(',')
+ base_model_names_list = [name.strip() for name in base_model_names_list]
+
+ print(f"Base models: {', '.join(base_model_names_list)}.")
+
+ for name in base_model_names_list:
+ print(f"Preparing {name}...")
+ snapshot_download(name)
+
+ print("")
+ print("Done.")
+
+
+if __name__ == "__main__":
+ fire.Fire(main)
diff --git a/llama_lora/__init__.py b/llama_lora/__init__.py
new file mode 100644
index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
diff --git a/llama_lora/config.py b/llama_lora/config.py
new file mode 100644
index 0000000000000000000000000000000000000000..6885d7094f12a6de0da66731b713b9cf192861ed
--- /dev/null
+++ b/llama_lora/config.py
@@ -0,0 +1,64 @@
+import os
+import pytz
+from typing import List, Union, Any
+
+
+class Config:
+ """
+ Stores the application configuration. This is a singleton class.
+ """
+
+ # Where data is stored
+ data_dir: str = ""
+
+ # Model Related
+ default_base_model_name: str = "freQuensy23/ru-openllama-3b"
+ base_model_choices: Union[List[str], str] = []
+ load_8bit: bool = False
+ trust_remote_code: bool = False
+
+ # Application Settings
+ timezone: Any = pytz.UTC
+
+ # Authentication
+ auth_username: Union[str, None] = None
+ auth_password: Union[str, None] = None
+
+ # Hugging Face
+ hf_access_token: Union[str, None] = None
+
+ # WandB
+ enable_wandb: Union[bool, None] = None
+ wandb_api_key: Union[str, None] = None
+ default_wandb_project: str = "llama-lora-tuner"
+
+ # UI related
+ ui_title: str = "ru-llama"
+ ui_emoji: str = "🦙🎛️"
+ ui_subtitle: str = "Лама, которая понимает на русском. UI forked from zetavg/LLaMA-LoRA-Tuner"
+ ui_show_sys_info: bool = True
+ ui_dev_mode: bool = False
+ ui_dev_mode_title_prefix: str = "[UI DEV MODE] "
+
+
+def process_config():
+ Config.data_dir = os.path.abspath(Config.data_dir)
+
+ if isinstance(Config.base_model_choices, str):
+ base_model_choices = Config.base_model_choices.split(',')
+ base_model_choices = [name.strip() for name in base_model_choices]
+ Config.base_model_choices = base_model_choices
+
+ if isinstance(Config.timezone, str):
+ Config.timezone = pytz.timezone(Config.timezone)
+
+ if Config.default_base_model_name not in Config.base_model_choices:
+ Config.base_model_choices = [
+ Config.default_base_model_name] + Config.base_model_choices
+
+ if Config.enable_wandb is None:
+ if (
+ Config.wandb_api_key and len(Config.wandb_api_key) > 0
+ and Config.default_wandb_project and len(Config.default_wandb_project) > 0
+ ):
+ Config.enable_wandb = True
diff --git a/llama_lora/dynamic_import.py b/llama_lora/dynamic_import.py
new file mode 100644
index 0000000000000000000000000000000000000000..9bb1cb5da2cb19e0e85091dd978787486c296e97
--- /dev/null
+++ b/llama_lora/dynamic_import.py
@@ -0,0 +1,5 @@
+import importlib
+
+
+def dynamic_import(module):
+ return importlib.import_module(module, package=__package__)
diff --git a/llama_lora/globals.py b/llama_lora/globals.py
new file mode 100644
index 0000000000000000000000000000000000000000..4e65a28ef972ffdc7a8c51180523bc80c3275326
--- /dev/null
+++ b/llama_lora/globals.py
@@ -0,0 +1,177 @@
+import importlib
+import os
+import subprocess
+import psutil
+import math
+
+from typing import Any, Dict, List, Optional, Tuple, Union
+from transformers import TrainingArguments
+from numba import cuda
+import nvidia_smi
+
+from .dynamic_import import dynamic_import
+from .config import Config
+from .utils.lru_cache import LRUCache
+from .utils.eta_predictor import ETAPredictor
+
+
+class Global:
+ """
+ A singleton class holding global states.
+ """
+
+ version: Union[str, None] = None
+
+ base_model_name: str = ""
+ tokenizer_name: Union[str, None] = None
+
+ # Functions
+ inference_generate_fn: Any
+ finetune_train_fn: Any
+
+ # Training Control
+ should_stop_training: bool = False
+
+ # Training Status
+ is_train_starting: bool = False
+ is_training: bool = False
+ train_started_at: float = 0.0
+ training_error_message: Union[str, None] = None
+ training_error_detail: Union[str, None] = None
+ training_total_epochs: int = 0
+ training_current_epoch: float = 0.0
+ training_total_steps: int = 0
+ training_current_step: int = 0
+ training_progress: float = 0.0
+ training_log_history: List[Any] = []
+ training_status_text: str = ""
+ training_eta_predictor = ETAPredictor()
+ training_eta: Union[int, None] = None
+ training_args: Union[TrainingArguments, None] = None
+ train_output: Union[None, Any] = None
+ train_output_str: Union[None, str] = None
+ training_params_info_text: str = ""
+
+ # Generation Control
+ should_stop_generating: bool = False
+ generation_force_stopped_at: Union[float, None] = None
+
+ # Model related
+ loaded_models = LRUCache(1)
+ loaded_tokenizers = LRUCache(1)
+ new_base_model_that_is_ready_to_be_used = None
+ name_of_new_base_model_that_is_ready_to_be_used = None
+
+ # GPU Info
+ gpu_cc = None # GPU compute capability
+ gpu_sms = None # GPU total number of SMs
+ gpu_total_cores = None # GPU total cores
+ gpu_total_memory = None
+
+
+def initialize_global():
+ Global.base_model_name = Config.default_base_model_name
+ commit_hash = get_git_commit_hash()
+
+ if commit_hash:
+ Global.version = commit_hash[:8]
+
+ if not Config.ui_dev_mode:
+ ModelLRUCache = dynamic_import('.utils.model_lru_cache').ModelLRUCache
+ Global.loaded_models = ModelLRUCache(1)
+ Global.inference_generate_fn = dynamic_import('.lib.inference').generate
+ Global.finetune_train_fn = dynamic_import('.lib.finetune').train
+ load_gpu_info()
+
+
+def get_package_dir():
+ current_file_path = os.path.abspath(__file__)
+ parent_directory_path = os.path.dirname(current_file_path)
+ return os.path.abspath(parent_directory_path)
+
+
+def get_git_commit_hash():
+ try:
+ original_cwd = os.getcwd()
+ project_dir = get_package_dir()
+ try:
+ os.chdir(project_dir)
+ commit_hash = subprocess.check_output(
+ ['git', 'rev-parse', 'HEAD']).strip().decode('utf-8')
+ return commit_hash
+ except Exception as e:
+ print(f"Cannot get git commit hash: {e}")
+ finally:
+ os.chdir(original_cwd)
+ except Exception as e:
+ print(f"Cannot get git commit hash: {e}")
+
+
+def load_gpu_info():
+ # cuda = importlib.import_module('numba').cuda
+ # nvidia_smi = importlib.import_module('nvidia_smi')
+ print("")
+ try:
+ cc_cores_per_SM_dict = {
+ (2, 0): 32,
+ (2, 1): 48,
+ (3, 0): 192,
+ (3, 5): 192,
+ (3, 7): 192,
+ (5, 0): 128,
+ (5, 2): 128,
+ (6, 0): 64,
+ (6, 1): 128,
+ (7, 0): 64,
+ (7, 5): 64,
+ (8, 0): 64,
+ (8, 6): 128,
+ (8, 9): 128,
+ (9, 0): 128
+ }
+ # the above dictionary should result in a value of "None" if a cc match
+ # is not found. The dictionary needs to be extended as new devices become
+ # available, and currently does not account for all Jetson devices
+ device = cuda.get_current_device()
+ device_sms = getattr(device, 'MULTIPROCESSOR_COUNT')
+ device_cc = device.compute_capability
+ cores_per_sm = cc_cores_per_SM_dict.get(device_cc)
+ total_cores = cores_per_sm*device_sms
+ print("GPU compute capability: ", device_cc)
+ print("GPU total number of SMs: ", device_sms)
+ print("GPU total cores: ", total_cores)
+ Global.gpu_cc = device_cc
+ Global.gpu_sms = device_sms
+ Global.gpu_total_cores = total_cores
+
+ nvidia_smi.nvmlInit()
+ handle = nvidia_smi.nvmlDeviceGetHandleByIndex(0)
+ info = nvidia_smi.nvmlDeviceGetMemoryInfo(handle)
+ total_memory = info.total
+
+ total_memory_mb = total_memory / (1024 ** 2)
+ total_memory_gb = total_memory / (1024 ** 3)
+
+ # Print the memory size
+ print(
+ f"GPU total memory: {total_memory} bytes ({total_memory_mb:.2f} MB) ({total_memory_gb:.2f} GB)")
+ Global.gpu_total_memory = total_memory
+
+ available_cpu_ram = psutil.virtual_memory().available
+ available_cpu_ram_mb = available_cpu_ram / (1024 ** 2)
+ available_cpu_ram_gb = available_cpu_ram / (1024 ** 3)
+ print(
+ f"CPU available memory: {available_cpu_ram} bytes ({available_cpu_ram_mb:.2f} MB) ({available_cpu_ram_gb:.2f} GB)")
+ preserve_loaded_models_count = math.floor(
+ (available_cpu_ram * 0.8) / total_memory) - 1
+ if preserve_loaded_models_count > 1:
+ ModelLRUCache = dynamic_import('.utils.model_lru_cache').ModelLRUCache
+ print(
+ f"Will keep {preserve_loaded_models_count} offloaded models in CPU RAM.")
+ Global.loaded_models = ModelLRUCache(preserve_loaded_models_count)
+ Global.loaded_tokenizers = LRUCache(preserve_loaded_models_count)
+
+ except Exception as e:
+ print(f"Notice: cannot get GPU info: {e}")
+
+ print("")
diff --git a/llama_lora/lib/csv_logger.py b/llama_lora/lib/csv_logger.py
new file mode 100644
index 0000000000000000000000000000000000000000..028cd29eb72fd0b62cb489dabd4a9071bf466d1f
--- /dev/null
+++ b/llama_lora/lib/csv_logger.py
@@ -0,0 +1,96 @@
+from gradio import FlaggingCallback, utils
+import csv
+import datetime
+import os
+import re
+import secrets
+from pathlib import Path
+from typing import Any, List, Union
+
+class CSVLogger(FlaggingCallback):
+ """
+ The default implementation of the FlaggingCallback abstract class. Each flagged
+ sample (both the input and output data) is logged to a CSV file with headers on the machine running the gradio app.
+ Example:
+ import gradio as gr
+ def image_classifier(inp):
+ return {'cat': 0.3, 'dog': 0.7}
+ demo = gr.Interface(fn=image_classifier, inputs="image", outputs="label",
+ flagging_callback=CSVLogger())
+ Guides: using_flagging
+ """
+
+ def __init__(self):
+ pass
+
+ def setup(
+ self,
+ components: List[Any],
+ flagging_dir: Union[str, Path],
+ ):
+ self.components = components
+ self.flagging_dir = flagging_dir
+ os.makedirs(flagging_dir, exist_ok=True)
+
+ def flag(
+ self,
+ flag_data: List[Any],
+ flag_option: str = "",
+ username: Union[str, None] = None,
+ filename="log.csv",
+ ) -> int:
+ flagging_dir = self.flagging_dir
+ filename = re.sub(r"[/\\?%*:|\"<>\x7F\x00-\x1F]", "-", filename)
+ log_filepath = Path(flagging_dir) / filename
+ is_new = not Path(log_filepath).exists()
+ headers = [
+ getattr(component, "label", None) or f"component {idx}"
+ for idx, component in enumerate(self.components)
+ ] + [
+ "flag",
+ "username",
+ "timestamp",
+ ]
+
+ csv_data = []
+ for idx, (component, sample) in enumerate(zip(self.components, flag_data)):
+ save_dir = Path(
+ flagging_dir
+ ) / (
+ getattr(component, "label", None) or f"component {idx}"
+ )
+ if utils.is_update(sample):
+ csv_data.append(str(sample))
+ else:
+ csv_data.append(
+ component.deserialize(sample, save_dir=save_dir)
+ if sample is not None
+ else ""
+ )
+ csv_data.append(flag_option)
+ csv_data.append(username if username is not None else "")
+ csv_data.append(str(datetime.datetime.now()))
+
+ try:
+ with open(log_filepath, "a", newline="", encoding="utf-8") as csvfile:
+ writer = csv.writer(csvfile)
+ if is_new:
+ writer.writerow(utils.sanitize_list_for_csv(headers))
+ writer.writerow(utils.sanitize_list_for_csv(csv_data))
+ except Exception as e:
+ # workaround "OSError: [Errno 95] Operation not supported" with open(log_filepath, "a") on some cloud mounted directory
+ random_hex = secrets.token_hex(16)
+ tmp_log_filepath = str(log_filepath) + f".tmp_{random_hex}"
+ with open(tmp_log_filepath, "a", newline="", encoding="utf-8") as csvfile:
+ writer = csv.writer(csvfile)
+ if is_new:
+ writer.writerow(utils.sanitize_list_for_csv(headers))
+ writer.writerow(utils.sanitize_list_for_csv(csv_data))
+ os.system(f"mv '{log_filepath}' '{log_filepath}.old_{random_hex}'")
+ os.system(f"cat '{log_filepath}.old_{random_hex}' '{tmp_log_filepath}' > '{log_filepath}'")
+ os.system(f"rm '{tmp_log_filepath}'")
+ os.system(f"rm '{log_filepath}.old_{random_hex}'")
+
+ with open(log_filepath, "r", encoding="utf-8") as csvfile:
+ line_count = len([None for row in csv.reader(csvfile)]) - 1
+ return line_count
diff --git a/llama_lora/lib/finetune.py b/llama_lora/lib/finetune.py
new file mode 100644
index 0000000000000000000000000000000000000000..58047b62ce008895f75e319a07fe51cac0f83764
--- /dev/null
+++ b/llama_lora/lib/finetune.py
@@ -0,0 +1,474 @@
+import os
+import sys
+import re
+import importlib
+from typing import Any, List, Union
+
+import json
+
+import fire
+import torch
+import transformers
+from datasets import Dataset, load_dataset
+
+
+from peft import (
+ LoraConfig,
+ get_peft_model,
+ get_peft_model_state_dict,
+ prepare_model_for_int8_training,
+ set_peft_model_state_dict,
+)
+from transformers import AutoModelForCausalLM, AutoTokenizer, LlamaTokenizer
+
+
+def train(
+ # model/data params
+ base_model: Any,
+ tokenizer: Any,
+ output_dir: str,
+ train_data: List[Any],
+ #
+ load_in_8bit=True,
+ fp16=True,
+ bf16=False,
+ gradient_checkpointing=False,
+ # training hyperparams
+ micro_batch_size: int = 4,
+ gradient_accumulation_steps: int = 32,
+ num_train_epochs: int = 3,
+ learning_rate: float = 3e-4,
+ cutoff_len: int = 256,
+ val_set_size: int = 2000,
+ # lora hyperparams
+ lora_r: int = 8,
+ lora_alpha: int = 16,
+ lora_dropout: float = 0.05,
+ lora_target_modules: List[str] = [
+ "q_proj",
+ "v_proj",
+ ],
+ lora_modules_to_save: Union[List[str], None] = [],
+ # llm hyperparams
+ train_on_inputs: bool = True, # if False, masks out inputs in loss
+ group_by_length: bool = False, # faster, but produces an odd training loss curve
+ # either training checkpoint or final adapter
+ resume_from_checkpoint=None,
+ save_steps: int = 200,
+ save_total_limit: int = 3,
+ logging_steps: int = 10,
+ #
+ additional_training_arguments: Union[dict, str, None] = None,
+ additional_lora_config: Union[dict, str, None] = None,
+ # logging
+ callbacks: List[Any] = [],
+ # wandb params
+ wandb_api_key=None,
+ wandb_project: str = "",
+ wandb_group=None,
+ wandb_run_name: str = "",
+ wandb_tags: List[str] = [],
+ wandb_watch: str = "false", # options: false | gradients | all
+ wandb_log_model: str = "true", # options: false | true
+ additional_wandb_config: Union[dict, None] = None,
+ hf_access_token: Union[str, None] = None,
+ status_message_callback: Any = None,
+ params_info_callback: Any = None,
+):
+ if status_message_callback:
+ cb_result = status_message_callback("Preparing...")
+ if cb_result:
+ return
+
+ if lora_modules_to_save is not None and len(lora_modules_to_save) <= 0:
+ lora_modules_to_save = None
+
+ if isinstance(additional_training_arguments, str):
+ additional_training_arguments = additional_training_arguments.strip()
+ if not additional_training_arguments:
+ additional_training_arguments = None
+ if isinstance(additional_training_arguments, str):
+ try:
+ additional_training_arguments = json.loads(
+ additional_training_arguments)
+ except Exception as e:
+ raise ValueError(
+ f"Could not parse additional_training_arguments: {e}")
+
+ if isinstance(additional_lora_config, str):
+ additional_lora_config = additional_lora_config.strip()
+ if not additional_lora_config:
+ additional_lora_config = None
+ if isinstance(additional_lora_config, str):
+ try:
+ additional_lora_config = json.loads(additional_lora_config)
+ except Exception as e:
+ raise ValueError(f"Could not parse additional_lora_config: {e}")
+
+ # for logging
+ finetune_args = {
+ 'micro_batch_size': micro_batch_size,
+ 'gradient_accumulation_steps': gradient_accumulation_steps,
+ 'num_train_epochs': num_train_epochs,
+ 'learning_rate': learning_rate,
+ 'cutoff_len': cutoff_len,
+ 'val_set_size': val_set_size,
+ 'lora_r': lora_r,
+ 'lora_alpha': lora_alpha,
+ 'lora_dropout': lora_dropout,
+ 'lora_target_modules': lora_target_modules,
+ 'lora_modules_to_save': lora_modules_to_save or [],
+ 'train_on_inputs': train_on_inputs,
+ 'group_by_length': group_by_length,
+ 'load_in_8bit': load_in_8bit,
+ 'fp16': fp16,
+ 'bf16': bf16,
+ 'gradient_checkpointing': gradient_checkpointing,
+ 'save_steps': save_steps,
+ 'save_total_limit': save_total_limit,
+ 'logging_steps': logging_steps,
+ 'additional_training_arguments': additional_training_arguments,
+ 'additional_lora_config': additional_lora_config,
+ }
+ if val_set_size and val_set_size > 0:
+ finetune_args['val_set_size'] = val_set_size
+ # if lora_modules_to_save:
+ # finetune_args['lora_modules_to_save'] = lora_modules_to_save
+ if resume_from_checkpoint:
+ finetune_args['resume_from_checkpoint'] = resume_from_checkpoint
+
+ wandb = None
+ if wandb_api_key:
+ os.environ["WANDB_API_KEY"] = wandb_api_key
+
+ # wandb: WARNING Changes to your `wandb` environment variables will be ignored because your `wandb` session has already started. For more information on how to modify your settings with `wandb.init()` arguments, please refer to https://wandb.me/wandb-init.
+ # if wandb_project:
+ # os.environ["WANDB_PROJECT"] = wandb_project
+ # if wandb_run_name:
+ # os.environ["WANDB_RUN_NAME"] = wandb_run_name
+
+ if wandb_watch:
+ os.environ["WANDB_WATCH"] = wandb_watch
+ if wandb_log_model:
+ os.environ["WANDB_LOG_MODEL"] = wandb_log_model
+ use_wandb = (wandb_project and len(wandb_project) > 0) or (
+ "WANDB_PROJECT" in os.environ and len(os.environ["WANDB_PROJECT"]) > 0
+ )
+ if use_wandb:
+ os.environ['WANDB_MODE'] = "online"
+ wandb = importlib.import_module("wandb")
+ wandb.init(
+ project=wandb_project,
+ resume="auto",
+ group=wandb_group,
+ name=wandb_run_name,
+ tags=wandb_tags,
+ reinit=True,
+ magic=True,
+ config={'finetune_args': finetune_args},
+ # id=None # used for resuming
+ )
+ if additional_wandb_config:
+ wandb.config.update(additional_wandb_config)
+ else:
+ os.environ['WANDB_MODE'] = "disabled"
+
+ if os.path.exists(output_dir):
+ if (not os.path.isdir(output_dir)) or os.path.exists(os.path.join(output_dir, 'adapter_config.json')):
+ raise ValueError(
+ f"The output directory already exists and is not empty. ({output_dir})")
+
+ device_map = "auto"
+ world_size = int(os.environ.get("WORLD_SIZE", 1))
+ ddp = world_size != 1
+ if ddp:
+ device_map = {"": int(os.environ.get("LOCAL_RANK") or 0)}
+
+ if status_message_callback:
+ if isinstance(base_model, str):
+ cb_result = status_message_callback(
+ f"Preparing model '{base_model}' for training...")
+ if cb_result:
+ return
+ else:
+ cb_result = status_message_callback(
+ "Preparing model for training...")
+ if cb_result:
+ return
+
+ model = base_model
+ if isinstance(model, str):
+ model_name = model
+ print(f"Loading base model {model_name}...")
+ model = AutoModelForCausalLM.from_pretrained(
+ base_model,
+ load_in_8bit=load_in_8bit,
+ torch_dtype=torch.float16,
+ llm_int8_skip_modules=lora_modules_to_save,
+ device_map=device_map,
+ use_auth_token=hf_access_token
+ )
+ if re.match("[^/]+/llama", model_name):
+ print(f"Setting special tokens for LLaMA model {model_name}...")
+ model.config.pad_token_id = 0
+ model.config.bos_token_id = 1
+ model.config.eos_token_id = 2
+
+ print(f"Loaded model {model_name}")
+
+ if isinstance(tokenizer, str):
+ tokenizer_name = tokenizer
+ try:
+ tokenizer = AutoTokenizer.from_pretrained(
+ tokenizer, use_auth_token=hf_access_token
+ )
+ except Exception as e:
+ if 'LLaMATokenizer' in str(e):
+ tokenizer = LlamaTokenizer.from_pretrained(
+ tokenizer_name,
+ use_auth_token=hf_access_token
+ )
+ else:
+ raise e
+
+ if re.match("[^/]+/llama", tokenizer_name):
+ print(
+ f"Setting special tokens for LLaMA tokenizer {tokenizer_name}...")
+ tokenizer.pad_token_id = 0
+ tokenizer.bos_token_id = 1
+ tokenizer.eos_token_id = 2
+
+ print(f"Loaded tokenizer {tokenizer_name}")
+
+ # tokenizer.pad_token_id = (
+ # 0 # unk. we want this to be different from the eos token
+ # )
+ tokenizer.padding_side = "left" # Allow batched inference
+
+ try:
+ model = prepare_model_for_int8_training(model)
+ except Exception as e:
+ print(
+ f"Got error while running prepare_model_for_int8_training(model), maybe the model has already be prepared. Original error: {e}.")
+
+ if status_message_callback:
+ cb_result = status_message_callback(
+ "Preparing PEFT model for training...")
+ if cb_result:
+ return
+
+ lora_config_args = {
+ 'r': lora_r,
+ 'lora_alpha': lora_alpha,
+ 'target_modules': lora_target_modules,
+ 'modules_to_save': lora_modules_to_save,
+ 'lora_dropout': lora_dropout,
+ 'bias': "none",
+ 'task_type': "CAUSAL_LM",
+ }
+ config = LoraConfig(**{
+ **lora_config_args,
+ **(additional_lora_config or {}),
+ })
+ model = get_peft_model(model, config)
+ if bf16:
+ model = model.to(torch.bfloat16)
+
+ if resume_from_checkpoint:
+ # Check the available weights and load them
+ checkpoint_name = os.path.join(
+ resume_from_checkpoint, "pytorch_model.bin"
+ ) # Full checkpoint
+ if not os.path.exists(checkpoint_name):
+ checkpoint_name = os.path.join(
+ resume_from_checkpoint, "adapter_model.bin"
+ ) # only LoRA model - LoRA config above has to fit
+ resume_from_checkpoint = (
+ False # So the trainer won't try loading its state
+ )
+ # The two files above have a different name depending on how they were saved, but are actually the same.
+ if os.path.exists(checkpoint_name):
+ print(f"Restarting from {checkpoint_name}")
+ adapters_weights = torch.load(checkpoint_name)
+ model = set_peft_model_state_dict(model, adapters_weights)
+ else:
+ raise ValueError(f"Checkpoint {checkpoint_name} not found")
+
+ # Be more transparent about the % of trainable params.
+ trainable_params = 0
+ all_params = 0
+ for _, param in model.named_parameters():
+ all_params += param.numel()
+ if param.requires_grad:
+ trainable_params += param.numel()
+ print(
+ f"trainable params: {trainable_params} || all params: {all_params} || trainable%: {100 * trainable_params / all_params} (calculated)"
+ )
+ model.print_trainable_parameters()
+ if use_wandb and wandb:
+ wandb.config.update({"model": {"all_params": all_params, "trainable_params": trainable_params,
+ "trainable%": 100 * trainable_params / all_params}})
+ if params_info_callback:
+ cb_result = params_info_callback(
+ all_params=all_params, trainable_params=trainable_params)
+ if cb_result:
+ return
+
+ if status_message_callback:
+ cb_result = status_message_callback("Preparing train data...")
+ if cb_result:
+ return
+
+ def tokenize(prompt, add_eos_token=True):
+ # there's probably a way to do this with the tokenizer settings
+ # but again, gotta move fast
+ result = tokenizer(
+ prompt,
+ truncation=True,
+ max_length=cutoff_len,
+ padding=False,
+ return_tensors=None,
+ )
+ if (
+ result["input_ids"][-1] != tokenizer.eos_token_id
+ and len(result["input_ids"]) < cutoff_len
+ and add_eos_token
+ ):
+ result["input_ids"].append(tokenizer.eos_token_id)
+ result["attention_mask"].append(1)
+
+ result["labels"] = result["input_ids"].copy()
+
+ return result
+
+ def generate_and_tokenize_prompt(data_point):
+ full_prompt = data_point["prompt"] + data_point["completion"]
+ tokenized_full_prompt = tokenize(full_prompt)
+ if not train_on_inputs:
+ user_prompt = data_point["prompt"]
+ tokenized_user_prompt = tokenize(user_prompt, add_eos_token=False)
+ user_prompt_len = len(tokenized_user_prompt["input_ids"])
+
+ tokenized_full_prompt["labels"] = [
+ -100
+ ] * user_prompt_len + tokenized_full_prompt["labels"][
+ user_prompt_len:
+ ] # could be sped up, probably
+ return tokenized_full_prompt
+
+ # If train_data is a list, convert it to datasets.Dataset
+ if isinstance(train_data, list):
+ with open(os.path.join(output_dir, "train_data_samples.json"), 'w') as file:
+ json.dump(list(train_data[:100]), file, indent=2)
+ train_data = Dataset.from_list(train_data)
+
+ if val_set_size > 0:
+ train_val = train_data.train_test_split(
+ test_size=val_set_size, shuffle=True, seed=42
+ )
+ train_data = (
+ train_val["train"].shuffle().map(generate_and_tokenize_prompt)
+ )
+ val_data = (
+ train_val["test"].shuffle().map(generate_and_tokenize_prompt)
+ )
+ else:
+ train_data = train_data.shuffle().map(generate_and_tokenize_prompt)
+ val_data = None
+
+ if not ddp and torch.cuda.device_count() > 1:
+ # keeps Trainer from trying its own DataParallelism when more than 1 gpu is available
+ model.is_parallelizable = True
+ model.model_parallel = True
+
+ if status_message_callback:
+ cb_result = status_message_callback("Train starting...")
+ if cb_result:
+ return
+
+ # https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.TrainingArguments
+ training_args = {
+ 'output_dir': output_dir,
+ 'per_device_train_batch_size': micro_batch_size,
+ 'gradient_checkpointing': gradient_checkpointing,
+ 'gradient_accumulation_steps': gradient_accumulation_steps,
+ 'warmup_steps': 100,
+ 'num_train_epochs': num_train_epochs,
+ 'learning_rate': learning_rate,
+ 'fp16': fp16,
+ 'bf16': bf16,
+ 'logging_steps': logging_steps,
+ 'optim': "adamw_torch",
+ 'evaluation_strategy': "steps" if val_set_size > 0 else "no",
+ 'save_strategy': "steps",
+ 'eval_steps': save_steps if val_set_size > 0 else None,
+ 'save_steps': save_steps,
+ 'output_dir': output_dir,
+ 'save_total_limit': save_total_limit,
+ 'load_best_model_at_end': True if val_set_size > 0 else False,
+ 'ddp_find_unused_parameters': False if ddp else None,
+ 'group_by_length': group_by_length,
+ 'report_to': "wandb" if use_wandb else None,
+ 'run_name': wandb_run_name if use_wandb else None,
+ }
+
+ # https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.Trainer
+ trainer = transformers.Trainer(
+ model=model,
+ train_dataset=train_data,
+ eval_dataset=val_data,
+ tokenizer=tokenizer,
+ args=transformers.TrainingArguments(**{
+ **training_args,
+ **(additional_training_arguments or {})
+ }),
+ data_collator=transformers.DataCollatorForSeq2Seq(
+ tokenizer, pad_to_multiple_of=8, return_tensors="pt", padding=True
+ ),
+ callbacks=callbacks,
+ )
+
+ if not os.path.exists(output_dir):
+ os.makedirs(output_dir)
+ with open(os.path.join(output_dir, "trainer_args.json"), 'w') as trainer_args_json_file:
+ json.dump(trainer.args.to_dict(), trainer_args_json_file, indent=2)
+ with open(os.path.join(output_dir, "finetune_args.json"), 'w') as finetune_args_json_file:
+ json.dump(finetune_args, finetune_args_json_file, indent=2)
+
+ # Not working, will only give us ["prompt", "completion", "input_ids", "attention_mask", "labels"]
+ # if train_data:
+ # with open(os.path.join(output_dir, "train_dataset_samples.json"), 'w') as file:
+ # json.dump(list(train_data[:100]), file, indent=2)
+ # if val_data:
+ # with open(os.path.join(output_dir, "eval_dataset_samples.json"), 'w') as file:
+ # json.dump(list(val_data[:100]), file, indent=2)
+
+ model.config.use_cache = False
+
+ old_state_dict = model.state_dict
+ model.state_dict = (
+ lambda self, *_, **__: get_peft_model_state_dict(
+ self, old_state_dict()
+ )
+ ).__get__(model, type(model))
+
+ if torch.__version__ >= "2" and sys.platform != "win32":
+ model = torch.compile(model)
+
+ train_output = trainer.train(resume_from_checkpoint=resume_from_checkpoint)
+
+ model.save_pretrained(output_dir)
+ print(f"Model saved to {output_dir}.")
+
+ with open(os.path.join(output_dir, "trainer_log_history.jsonl"), 'w') as trainer_log_history_jsonl_file:
+ trainer_log_history = "\n".join(
+ [json.dumps(line) for line in trainer.state.log_history])
+ trainer_log_history_jsonl_file.write(trainer_log_history)
+
+ with open(os.path.join(output_dir, "train_output.json"), 'w') as train_output_json_file:
+ json.dump(train_output, train_output_json_file, indent=2)
+
+ if use_wandb and wandb:
+ wandb.finish()
+
+ return train_output
diff --git a/llama_lora/lib/get_device.py b/llama_lora/lib/get_device.py
new file mode 100644
index 0000000000000000000000000000000000000000..559a6dcbdcb02150d713a0dfb0bd35af1702faea
--- /dev/null
+++ b/llama_lora/lib/get_device.py
@@ -0,0 +1,16 @@
+import importlib
+
+
+def get_device():
+ torch = importlib.import_module('torch')
+ device ="cpu"
+ if torch.cuda.is_available():
+ device = "cuda"
+
+ try:
+ if torch.backends.mps.is_available():
+ device = "mps"
+ except: # noqa: E722
+ pass
+
+ return device
diff --git a/llama_lora/lib/inference.py b/llama_lora/lib/inference.py
new file mode 100644
index 0000000000000000000000000000000000000000..bce8e1f2a9310fe25dbb45e8ab47b945813bb44a
--- /dev/null
+++ b/llama_lora/lib/inference.py
@@ -0,0 +1,85 @@
+import torch
+import transformers
+
+from .get_device import get_device
+from .streaming_generation_utils import Iteratorize, Stream
+
+
+def generate(
+ # model
+ model,
+ tokenizer,
+ # input
+ prompt,
+ generation_config,
+ max_new_tokens,
+ stopping_criteria=[],
+ # output options
+ stream_output=False
+):
+ device = get_device()
+
+ inputs = tokenizer(prompt, return_tensors="pt")
+ input_ids = inputs["input_ids"].to(device)
+ generate_params = {
+ "input_ids": input_ids,
+ "generation_config": generation_config,
+ "return_dict_in_generate": True,
+ "output_scores": True,
+ "max_new_tokens": max_new_tokens,
+ "stopping_criteria": transformers.StoppingCriteriaList() + stopping_criteria
+ }
+
+ skip_special_tokens = True
+
+ if '/dolly' in tokenizer.name_or_path:
+ # dolly has additional_special_tokens as ['### End', '### Instruction:', '### Response:'], skipping them will break the prompter's reply extraction.
+ skip_special_tokens = False
+ # Ensure generation stops once it generates "### End"
+ end_key_token_id = tokenizer.encode("### End")
+ end_key_token_id = end_key_token_id[0] # 50277
+ if isinstance(generate_params['generation_config'].eos_token_id, str):
+ generate_params['generation_config'].eos_token_id = [generate_params['generation_config'].eos_token_id]
+ elif not generate_params['generation_config'].eos_token_id:
+ generate_params['generation_config'].eos_token_id = []
+ generate_params['generation_config'].eos_token_id.append(end_key_token_id)
+
+ if stream_output:
+ # Stream the reply 1 token at a time.
+ # This is based on the trick of using 'stopping_criteria' to create an iterator,
+ # from https://github.com/oobabooga/text-generation-webui/blob/ad37f396fc8bcbab90e11ecf17c56c97bfbd4a9c/modules/text_generation.py#L216-L243.
+ generation_output = None
+
+ def generate_with_callback(callback=None, **kwargs):
+ nonlocal generation_output
+ kwargs["stopping_criteria"].insert(
+ 0,
+ Stream(callback_func=callback)
+ )
+ with torch.no_grad():
+ generation_output = model.generate(**kwargs)
+
+ def generate_with_streaming(**kwargs):
+ return Iteratorize(
+ generate_with_callback, kwargs, callback=None
+ )
+
+ with generate_with_streaming(**generate_params) as generator:
+ for output in generator:
+ decoded_output = tokenizer.decode(output, skip_special_tokens=skip_special_tokens)
+ yield decoded_output, output, False
+
+ if generation_output:
+ output = generation_output.sequences[0]
+ decoded_output = tokenizer.decode(output, skip_special_tokens=skip_special_tokens)
+ yield decoded_output, output, True
+
+ return # early return for stream_output
+
+ # Without streaming
+ with torch.no_grad():
+ generation_output = model.generate(**generate_params)
+ output = generation_output.sequences[0]
+ decoded_output = tokenizer.decode(output, skip_special_tokens=skip_special_tokens)
+ yield decoded_output, output, True
+ return
diff --git a/llama_lora/lib/streaming_generation_utils.py b/llama_lora/lib/streaming_generation_utils.py
new file mode 100644
index 0000000000000000000000000000000000000000..7dedcade55aabb3c01bda254eb9a64615e4fa800
--- /dev/null
+++ b/llama_lora/lib/streaming_generation_utils.py
@@ -0,0 +1,75 @@
+"""
+Helpers to support streaming generate output.
+Borrowed from https://github.com/oobabooga/text-generation-webui/blob/ad37f396fc8bcbab90e11ecf17c56c97bfbd4a9c/modules/callbacks.py
+"""
+
+import gc
+import traceback
+from queue import Queue
+from threading import Thread
+
+import torch
+import transformers
+
+
+class Stream(transformers.StoppingCriteria):
+ def __init__(self, callback_func=None):
+ self.callback_func = callback_func
+
+ def __call__(self, input_ids, scores) -> bool:
+ if self.callback_func is not None:
+ self.callback_func(input_ids[0])
+ return False
+
+
+class Iteratorize:
+
+ """
+ Transforms a function that takes a callback
+ into a lazy iterator (generator).
+ """
+
+ def __init__(self, func, kwargs={}, callback=None):
+ self.mfunc = func
+ self.c_callback = callback
+ self.q = Queue()
+ self.sentinel = object()
+ self.kwargs = kwargs
+ self.stop_now = False
+
+ def _callback(val):
+ if self.stop_now:
+ raise ValueError
+ self.q.put(val)
+
+ def gentask():
+ try:
+ ret = self.mfunc(callback=_callback, **self.kwargs)
+ except ValueError:
+ pass
+ except:
+ traceback.print_exc()
+ pass
+
+ self.q.put(self.sentinel)
+ if self.c_callback:
+ self.c_callback(ret)
+
+ self.thread = Thread(target=gentask)
+ self.thread.start()
+
+ def __iter__(self):
+ return self
+
+ def __next__(self):
+ obj = self.q.get(True, None)
+ if obj is self.sentinel:
+ raise StopIteration
+ else:
+ return obj
+
+ def __enter__(self):
+ return self
+
+ def __exit__(self, exc_type, exc_val, exc_tb):
+ self.stop_now = True
diff --git a/llama_lora/models.py b/llama_lora/models.py
new file mode 100644
index 0000000000000000000000000000000000000000..42050422ac3055f5326bb8d95278b7b5f0c83a9c
--- /dev/null
+++ b/llama_lora/models.py
@@ -0,0 +1,279 @@
+import importlib
+import os
+import sys
+import gc
+import json
+import re
+
+from transformers import (
+ AutoModelForCausalLM, AutoModel,
+ AutoTokenizer, LlamaTokenizer
+)
+
+from .config import Config
+from .globals import Global
+from .lib.get_device import get_device
+
+
+def get_torch():
+ return importlib.import_module('torch')
+
+
+def get_peft_model_class():
+ return importlib.import_module('peft').PeftModel
+
+
+def get_new_base_model(base_model_name):
+ if Config.ui_dev_mode:
+ return
+ if Global.is_train_starting or Global.is_training:
+ raise Exception("Cannot load new base model while training.")
+
+ if Global.new_base_model_that_is_ready_to_be_used:
+ if Global.name_of_new_base_model_that_is_ready_to_be_used == base_model_name:
+ model = Global.new_base_model_that_is_ready_to_be_used
+ Global.new_base_model_that_is_ready_to_be_used = None
+ Global.name_of_new_base_model_that_is_ready_to_be_used = None
+ return model
+ else:
+ Global.new_base_model_that_is_ready_to_be_used = None
+ Global.name_of_new_base_model_that_is_ready_to_be_used = None
+ clear_cache()
+
+ model_class = AutoModelForCausalLM
+ from_tf = False
+ force_download = False
+ has_tried_force_download = False
+ while True:
+ try:
+ model = _get_model_from_pretrained(
+ model_class,
+ base_model_name,
+ from_tf=from_tf,
+ force_download=force_download
+ )
+ break
+ except Exception as e:
+ if 'from_tf' in str(e):
+ print(
+ f"Got error while loading model {base_model_name} with AutoModelForCausalLM: {e}.")
+ print("Retrying with from_tf=True...")
+ from_tf = True
+ force_download = False
+ elif model_class == AutoModelForCausalLM:
+ print(
+ f"Got error while loading model {base_model_name} with AutoModelForCausalLM: {e}.")
+ print("Retrying with AutoModel...")
+ model_class = AutoModel
+ force_download = False
+ else:
+ if has_tried_force_download:
+ raise e
+ print(
+ f"Got error while loading model {base_model_name}: {e}.")
+ print("Retrying with force_download=True...")
+ model_class = AutoModelForCausalLM
+ from_tf = False
+ force_download = True
+ has_tried_force_download = True
+
+ tokenizer = get_tokenizer(base_model_name)
+
+ if re.match("[^/]+/llama", base_model_name):
+ model.config.pad_token_id = tokenizer.pad_token_id = 0
+ model.config.bos_token_id = tokenizer.bos_token_id = 1
+ model.config.eos_token_id = tokenizer.eos_token_id = 2
+
+ return model
+
+
+def _get_model_from_pretrained(
+ model_class, model_name,
+ from_tf=False, force_download=False):
+ torch = get_torch()
+ device = get_device()
+
+ if device == "cuda":
+ return model_class.from_pretrained(
+ model_name,
+ load_in_8bit=Config.load_8bit,
+ torch_dtype=torch.float16,
+ # device_map="auto",
+ # ? https://github.com/tloen/alpaca-lora/issues/21
+ device_map={'': 0},
+ from_tf=from_tf,
+ force_download=force_download,
+ trust_remote_code=Config.trust_remote_code,
+ use_auth_token=Config.hf_access_token
+ )
+ elif device == "mps":
+ return model_class.from_pretrained(
+ model_name,
+ device_map={"": device},
+ torch_dtype=torch.float16,
+ from_tf=from_tf,
+ force_download=force_download,
+ trust_remote_code=Config.trust_remote_code,
+ use_auth_token=Config.hf_access_token
+ )
+ else:
+ return model_class.from_pretrained(
+ model_name,
+ device_map={"": device},
+ low_cpu_mem_usage=True,
+ from_tf=from_tf,
+ force_download=force_download,
+ trust_remote_code=Config.trust_remote_code,
+ use_auth_token=Config.hf_access_token
+ )
+
+
+def get_tokenizer(base_model_name):
+ if Config.ui_dev_mode:
+ return
+
+ if Global.is_train_starting or Global.is_training:
+ raise Exception("Cannot load new base model while training.")
+
+ loaded_tokenizer = Global.loaded_tokenizers.get(base_model_name)
+ if loaded_tokenizer:
+ return loaded_tokenizer
+
+ try:
+ tokenizer = AutoTokenizer.from_pretrained(
+ base_model_name,
+ trust_remote_code=Config.trust_remote_code,
+ use_auth_token=Config.hf_access_token
+ )
+ except Exception as e:
+ if 'LLaMATokenizer' in str(e):
+ tokenizer = LlamaTokenizer.from_pretrained(
+ base_model_name,
+ trust_remote_code=Config.trust_remote_code,
+ use_auth_token=Config.hf_access_token
+ )
+ else:
+ raise e
+
+ Global.loaded_tokenizers.set(base_model_name, tokenizer)
+
+ return tokenizer
+
+
+def get_model(
+ base_model_name,
+ peft_model_name=None):
+ if Config.ui_dev_mode:
+ return
+
+ if Global.is_train_starting or Global.is_training:
+ raise Exception("Cannot load new base model while training.")
+
+ torch = get_torch()
+
+ if peft_model_name == "None":
+ peft_model_name = None
+
+ model_key = base_model_name
+ if peft_model_name:
+ model_key = f"{base_model_name}//{peft_model_name}"
+
+ loaded_model = Global.loaded_models.get(model_key)
+ if loaded_model:
+ return loaded_model
+
+ peft_model_name_or_path = peft_model_name
+
+ if peft_model_name:
+ lora_models_directory_path = os.path.join(
+ Config.data_dir, "lora_models")
+ possible_lora_model_path = os.path.join(
+ lora_models_directory_path, peft_model_name)
+ if os.path.isdir(possible_lora_model_path):
+ peft_model_name_or_path = possible_lora_model_path
+
+ possible_model_info_json_path = os.path.join(
+ possible_lora_model_path, "info.json")
+ if os.path.isfile(possible_model_info_json_path):
+ try:
+ with open(possible_model_info_json_path, "r") as file:
+ json_data = json.load(file)
+ possible_hf_model_name = json_data.get("hf_model_name")
+ if possible_hf_model_name and json_data.get("load_from_hf"):
+ peft_model_name_or_path = possible_hf_model_name
+ except Exception as e:
+ raise ValueError(
+ "Error reading model info from {possible_model_info_json_path}: {e}")
+
+ Global.loaded_models.prepare_to_set()
+ clear_cache()
+
+ model = get_new_base_model(base_model_name)
+
+ if peft_model_name:
+ device = get_device()
+ PeftModel = get_peft_model_class()
+
+ if device == "cuda":
+ model = PeftModel.from_pretrained(
+ model,
+ peft_model_name_or_path,
+ torch_dtype=torch.float16,
+ # ? https://github.com/tloen/alpaca-lora/issues/21
+ device_map={'': 0},
+ use_auth_token=Config.hf_access_token
+ )
+ elif device == "mps":
+ model = PeftModel.from_pretrained(
+ model,
+ peft_model_name_or_path,
+ device_map={"": device},
+ torch_dtype=torch.float16,
+ use_auth_token=Config.hf_access_token
+ )
+ else:
+ model = PeftModel.from_pretrained(
+ model,
+ peft_model_name_or_path,
+ device_map={"": device},
+ use_auth_token=Config.hf_access_token
+ )
+
+ if re.match("[^/]+/llama", base_model_name):
+ model.config.pad_token_id = get_tokenizer(
+ base_model_name).pad_token_id = 0
+ model.config.bos_token_id = 1
+ model.config.eos_token_id = 2
+
+ if not Config.load_8bit:
+ model.half() # seems to fix bugs for some users.
+
+ model.eval()
+ if torch.__version__ >= "2" and sys.platform != "win32":
+ model = torch.compile(model)
+
+ Global.loaded_models.set(model_key, model)
+ clear_cache()
+
+ return model
+
+
+def prepare_base_model(base_model_name=Config.default_base_model_name):
+ Global.new_base_model_that_is_ready_to_be_used = get_new_base_model(
+ base_model_name)
+ Global.name_of_new_base_model_that_is_ready_to_be_used = base_model_name
+
+
+def clear_cache():
+ gc.collect()
+
+ torch = get_torch()
+ # if not shared.args.cpu: # will not be running on CPUs anyway
+ with torch.no_grad():
+ torch.cuda.empty_cache()
+
+
+def unload_models():
+ Global.loaded_models.clear()
+ Global.loaded_tokenizers.clear()
+ clear_cache()
diff --git a/llama_lora/ui/__init__.py b/llama_lora/ui/__init__.py
new file mode 100644
index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
diff --git a/llama_lora/ui/css_styles.py b/llama_lora/ui/css_styles.py
new file mode 100644
index 0000000000000000000000000000000000000000..d4ba6a3fe3ba1c73e41b3981355f092ee92dbe55
--- /dev/null
+++ b/llama_lora/ui/css_styles.py
@@ -0,0 +1,13 @@
+from typing import List
+
+css_styles: List[str] = []
+
+
+def get_css_styles():
+ global css_styles
+ return "\n".join(css_styles)
+
+
+def register_css_style(name, style):
+ global css_styles
+ css_styles.append(style)
diff --git a/llama_lora/ui/finetune/__init__.py b/llama_lora/ui/finetune/__init__.py
new file mode 100644
index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
diff --git a/llama_lora/ui/finetune/data_processing.py b/llama_lora/ui/finetune/data_processing.py
new file mode 100644
index 0000000000000000000000000000000000000000..7dcffb8c02b226e98bb3cedd095b24d9b009d849
--- /dev/null
+++ b/llama_lora/ui/finetune/data_processing.py
@@ -0,0 +1,74 @@
+import json
+from ...utils.data import get_dataset_content
+
+from .values import (
+ default_dataset_plain_text_input_variables_separator,
+ default_dataset_plain_text_input_and_output_separator,
+ default_dataset_plain_text_data_separator,
+)
+
+
+def get_data_from_input(load_dataset_from, dataset_text, dataset_text_format,
+ dataset_plain_text_input_variables_separator,
+ dataset_plain_text_input_and_output_separator,
+ dataset_plain_text_data_separator,
+ dataset_from_data_dir, prompter):
+ if load_dataset_from == "Text Input":
+ if dataset_text_format == "JSON":
+ data = json.loads(dataset_text)
+
+ elif dataset_text_format == "JSON Lines":
+ lines = dataset_text.split('\n')
+ data = []
+ for i, line in enumerate(lines):
+ line_number = i + 1
+ try:
+ data.append(json.loads(line))
+ except Exception as e:
+ raise ValueError(
+ f"Error parsing JSON on line {line_number}: {e}")
+
+ else: # Plain Text
+ data = parse_plain_text_input(
+ dataset_text,
+ (
+ dataset_plain_text_input_variables_separator or
+ default_dataset_plain_text_input_variables_separator
+ ).replace("\\n", "\n"),
+ (
+ dataset_plain_text_input_and_output_separator or
+ default_dataset_plain_text_input_and_output_separator
+ ).replace("\\n", "\n"),
+ (
+ dataset_plain_text_data_separator or
+ default_dataset_plain_text_data_separator
+ ).replace("\\n", "\n"),
+ prompter.get_variable_names()
+ )
+
+ else: # Load dataset from data directory
+ data = get_dataset_content(dataset_from_data_dir)
+
+ return data
+
+
+def parse_plain_text_input(
+ value,
+ variables_separator, input_output_separator, data_separator,
+ variable_names
+):
+ items = value.split(data_separator)
+ result = []
+ for item in items:
+ parts = item.split(input_output_separator)
+ variables = get_val_from_arr(parts, 0, "").split(variables_separator)
+ variables = [it.strip() for it in variables]
+ variables_dict = {name: var for name,
+ var in zip(variable_names, variables)}
+ output = get_val_from_arr(parts, 1, "").strip()
+ result.append({'variables': variables_dict, 'output': output})
+ return result
+
+
+def get_val_from_arr(arr, index, default=None):
+ return arr[index] if -len(arr) <= index < len(arr) else default
diff --git a/llama_lora/ui/finetune/finetune_ui.py b/llama_lora/ui/finetune/finetune_ui.py
new file mode 100644
index 0000000000000000000000000000000000000000..b6f92ddbf23306a2b070f31f5c807c637c302708
--- /dev/null
+++ b/llama_lora/ui/finetune/finetune_ui.py
@@ -0,0 +1,827 @@
+import os
+import json
+from datetime import datetime
+import gradio as gr
+from random_word import RandomWords
+
+from ...config import Config
+from ...globals import Global
+from ...utils.data import (
+ get_available_template_names,
+ get_available_dataset_names,
+ get_available_lora_model_names
+)
+from ...utils.relative_read_file import relative_read_file
+from ..css_styles import register_css_style
+
+from .values import (
+ default_dataset_plain_text_input_variables_separator,
+ default_dataset_plain_text_input_and_output_separator,
+ default_dataset_plain_text_data_separator,
+ sample_plain_text_value,
+ sample_jsonl_text_value,
+ sample_json_text_value,
+)
+from .previewing import (
+ refresh_preview,
+ refresh_dataset_items_count,
+)
+from .training import (
+ do_train,
+ render_training_status,
+ render_loss_plot
+)
+
+register_css_style('finetune', relative_read_file(__file__, "style.css"))
+
+
+def random_hyphenated_word():
+ r = RandomWords()
+ word1 = r.get_random_word()
+ word2 = r.get_random_word()
+ return word1 + '-' + word2
+
+
+def random_name():
+ current_datetime = datetime.now()
+ formatted_datetime = current_datetime.strftime("%Y-%m-%d-%H-%M-%S")
+ return f"{random_hyphenated_word()}-{formatted_datetime}"
+
+
+def reload_selections(current_template, current_dataset):
+ available_template_names = get_available_template_names()
+ available_template_names_with_none = available_template_names + ["None"]
+ if current_template not in available_template_names_with_none:
+ current_template = None
+ current_template = current_template or next(
+ iter(available_template_names_with_none), None)
+
+ available_dataset_names = get_available_dataset_names()
+ if current_dataset not in available_dataset_names:
+ current_dataset = None
+ current_dataset = current_dataset or next(
+ iter(available_dataset_names), None)
+
+ available_lora_models = ["-"] + get_available_lora_model_names()
+
+ return (
+ gr.Dropdown.update(
+ choices=available_template_names_with_none,
+ value=current_template),
+ gr.Dropdown.update(
+ choices=available_dataset_names,
+ value=current_dataset),
+ gr.Dropdown.update(choices=available_lora_models)
+ )
+
+
+def handle_switch_dataset_source(source):
+ if source == "Text Input":
+ return gr.Column.update(visible=True), gr.Column.update(visible=False)
+ else:
+ return gr.Column.update(visible=False), gr.Column.update(visible=True)
+
+
+def handle_switch_dataset_text_format(format):
+ if format == "Plain Text":
+ return gr.Column.update(visible=True)
+ return gr.Column.update(visible=False)
+
+
+def load_sample_dataset_to_text_input(format):
+ if format == "JSON":
+ return gr.Code.update(value=sample_json_text_value)
+ if format == "JSON Lines":
+ return gr.Code.update(value=sample_jsonl_text_value)
+ else: # Plain Text
+ return gr.Code.update(value=sample_plain_text_value)
+
+
+def handle_continue_from_model_change(model_name):
+ try:
+ lora_models_directory_path = os.path.join(
+ Config.data_dir, "lora_models")
+ lora_model_directory_path = os.path.join(
+ lora_models_directory_path, model_name)
+ all_files = os.listdir(lora_model_directory_path)
+ checkpoints = [
+ file for file in all_files if file.startswith("checkpoint-")]
+ checkpoints = ["-"] + checkpoints
+ can_load_params = "finetune_params.json" in all_files or "finetune_args.json" in all_files
+ return (gr.Dropdown.update(choices=checkpoints, value="-"),
+ gr.Button.update(visible=can_load_params),
+ gr.Markdown.update(value="", visible=False))
+ except Exception:
+ pass
+ return (gr.Dropdown.update(choices=["-"], value="-"),
+ gr.Button.update(visible=False),
+ gr.Markdown.update(value="", visible=False))
+
+
+def handle_load_params_from_model(
+ model_name,
+ template, load_dataset_from, dataset_from_data_dir,
+ max_seq_length,
+ evaluate_data_count,
+ micro_batch_size,
+ gradient_accumulation_steps,
+ epochs,
+ learning_rate,
+ train_on_inputs,
+ lora_r,
+ lora_alpha,
+ lora_dropout,
+ lora_target_modules,
+ lora_modules_to_save,
+ load_in_8bit,
+ fp16,
+ bf16,
+ gradient_checkpointing,
+ save_steps,
+ save_total_limit,
+ logging_steps,
+ additional_training_arguments,
+ additional_lora_config,
+ lora_target_module_choices,
+ lora_modules_to_save_choices,
+):
+ error_message = ""
+ notice_message = ""
+ unknown_keys = []
+ try:
+ lora_models_directory_path = os.path.join(
+ Config.data_dir, "lora_models")
+ lora_model_directory_path = os.path.join(
+ lora_models_directory_path, model_name)
+
+ try:
+ with open(os.path.join(lora_model_directory_path, "info.json"), "r") as f:
+ info = json.load(f)
+ if isinstance(info, dict):
+ model_prompt_template = info.get("prompt_template")
+ if model_prompt_template:
+ template = model_prompt_template
+ model_dataset_name = info.get("dataset_name")
+ if model_dataset_name and isinstance(model_dataset_name, str) and not model_dataset_name.startswith("N/A"):
+ load_dataset_from = "Data Dir"
+ dataset_from_data_dir = model_dataset_name
+ except FileNotFoundError:
+ pass
+
+ data = {}
+ possible_files = ["finetune_params.json", "finetune_args.json"]
+ for file in possible_files:
+ try:
+ with open(os.path.join(lora_model_directory_path, file), "r") as f:
+ data = json.load(f)
+ except FileNotFoundError:
+ pass
+
+ for key, value in data.items():
+ if key == "max_seq_length":
+ max_seq_length = value
+ if key == "cutoff_len":
+ max_seq_length = value
+ elif key == "evaluate_data_count":
+ evaluate_data_count = value
+ elif key == "val_set_size":
+ evaluate_data_count = value
+ elif key == "micro_batch_size":
+ micro_batch_size = value
+ elif key == "gradient_accumulation_steps":
+ gradient_accumulation_steps = value
+ elif key == "epochs":
+ epochs = value
+ elif key == "num_train_epochs":
+ epochs = value
+ elif key == "learning_rate":
+ learning_rate = value
+ elif key == "train_on_inputs":
+ train_on_inputs = value
+ elif key == "lora_r":
+ lora_r = value
+ elif key == "lora_alpha":
+ lora_alpha = value
+ elif key == "lora_dropout":
+ lora_dropout = value
+ elif key == "lora_target_modules":
+ lora_target_modules = value
+ if value:
+ for element in value:
+ if element not in lora_target_module_choices:
+ lora_target_module_choices.append(element)
+ elif key == "lora_modules_to_save":
+ lora_modules_to_save = value
+ if value:
+ for element in value:
+ if element not in lora_modules_to_save_choices:
+ lora_modules_to_save_choices.append(element)
+ elif key == "load_in_8bit":
+ load_in_8bit = value
+ elif key == "fp16":
+ fp16 = value
+ elif key == "bf16":
+ bf16 = value
+ elif key == "gradient_checkpointing":
+ gradient_checkpointing = value
+ elif key == "save_steps":
+ save_steps = value
+ elif key == "save_total_limit":
+ save_total_limit = value
+ elif key == "logging_steps":
+ logging_steps = value
+ elif key == "additional_training_arguments":
+ if value:
+ additional_training_arguments = json.dumps(value, indent=2)
+ else:
+ additional_training_arguments = ""
+ elif key == "additional_lora_config":
+ if value:
+ additional_lora_config = json.dumps(value, indent=2)
+ else:
+ additional_lora_config = ""
+ elif key == "group_by_length":
+ pass
+ elif key == "resume_from_checkpoint":
+ pass
+ else:
+ unknown_keys.append(key)
+ except Exception as e:
+ error_message = str(e)
+
+ if len(unknown_keys) > 0:
+ notice_message = f"Note: cannot restore unknown arg: {', '.join([f'`{x}`' for x in unknown_keys])}"
+
+ message = ". ".join([x for x in [error_message, notice_message] if x])
+
+ has_message = False
+ if message:
+ message += "."
+ has_message = True
+
+ return (
+ gr.Markdown.update(value=message, visible=has_message),
+ template, load_dataset_from, dataset_from_data_dir,
+ max_seq_length,
+ evaluate_data_count,
+ micro_batch_size,
+ gradient_accumulation_steps,
+ epochs,
+ learning_rate,
+ train_on_inputs,
+ lora_r,
+ lora_alpha,
+ lora_dropout,
+ gr.CheckboxGroup.update(value=lora_target_modules,
+ choices=lora_target_module_choices),
+ gr.CheckboxGroup.update(
+ value=lora_modules_to_save, choices=lora_modules_to_save_choices),
+ load_in_8bit,
+ fp16,
+ bf16,
+ gradient_checkpointing,
+ save_steps,
+ save_total_limit,
+ logging_steps,
+ additional_training_arguments,
+ additional_lora_config,
+ lora_target_module_choices,
+ lora_modules_to_save_choices
+ )
+
+
+default_lora_target_module_choices = ["q_proj", "k_proj", "v_proj", "o_proj"]
+default_lora_modules_to_save_choices = ["model.embed_tokens", "lm_head"]
+
+
+def handle_lora_target_modules_add(choices, new_module, selected_modules):
+ choices.append(new_module)
+ selected_modules.append(new_module)
+
+ return (choices, "", gr.CheckboxGroup.update(value=selected_modules, choices=choices))
+
+
+def handle_lora_modules_to_save_add(choices, new_module, selected_modules):
+ choices.append(new_module)
+ selected_modules.append(new_module)
+
+ return (choices, "", gr.CheckboxGroup.update(value=selected_modules, choices=choices))
+
+
+def do_abort_training():
+ Global.should_stop_training = True
+ Global.training_status_text = "Aborting..."
+
+
+def finetune_ui():
+ things_that_might_timeout = []
+
+ with gr.Blocks() as finetune_ui_blocks:
+ with gr.Column(elem_id="finetune_ui_content"):
+ with gr.Tab("Prepare"):
+ with gr.Box(elem_id="finetune_ui_select_dataset_source"):
+ with gr.Row():
+ template = gr.Dropdown(
+ label="Template",
+ elem_id="finetune_template",
+ )
+ load_dataset_from = gr.Radio(
+ ["Text Input", "Data Dir"],
+ label="Load Dataset From",
+ value="Text Input",
+ elem_id="finetune_load_dataset_from")
+ reload_selections_button = gr.Button(
+ "↻",
+ elem_id="finetune_reload_selections_button"
+ )
+ reload_selections_button.style(
+ full_width=False,
+ size="sm")
+ with gr.Column(
+ elem_id="finetune_dataset_from_data_dir_group",
+ visible=False
+ ) as dataset_from_data_dir_group:
+ dataset_from_data_dir = gr.Dropdown(
+ label="Dataset",
+ elem_id="finetune_dataset_from_data_dir",
+ )
+ dataset_from_data_dir_message = gr.Markdown(
+ "",
+ visible=False,
+ elem_id="finetune_dataset_from_data_dir_message")
+ with gr.Box(elem_id="finetune_dataset_text_input_group") as dataset_text_input_group:
+ gr.Textbox(
+ label="Training Data", elem_classes="textbox_that_is_only_used_to_display_a_label")
+ dataset_text = gr.Code(
+ show_label=False,
+ language="json",
+ value=sample_plain_text_value,
+ # max_lines=40,
+ elem_id="finetune_dataset_text_input_textbox")
+ dataset_from_text_message = gr.Markdown(
+ "",
+ visible=False,
+ elem_id="finetune_dataset_from_text_message")
+ gr.Markdown(
+ "The data you entered here will not be saved. Do not make edits here directly. Instead, edit the data elsewhere then paste it here.")
+ with gr.Row():
+ with gr.Column():
+ dataset_text_format = gr.Radio(
+ ["Plain Text", "JSON Lines", "JSON"],
+ label="Format", value="Plain Text", elem_id="finetune_dataset_text_format")
+ dataset_text_load_sample_button = gr.Button(
+ "Load Sample", elem_id="finetune_dataset_text_load_sample_button")
+ dataset_text_load_sample_button.style(
+ full_width=False,
+ size="sm")
+ with gr.Column(elem_id="finetune_dataset_plain_text_separators_group") as dataset_plain_text_separators_group:
+ dataset_plain_text_input_variables_separator = gr.Textbox(
+ label="Input Variables Separator",
+ elem_id="dataset_plain_text_input_variables_separator",
+ placeholder=default_dataset_plain_text_input_variables_separator,
+ value=default_dataset_plain_text_input_variables_separator)
+ dataset_plain_text_input_and_output_separator = gr.Textbox(
+ label="Input and Output Separator",
+ elem_id="dataset_plain_text_input_and_output_separator",
+ placeholder=default_dataset_plain_text_input_and_output_separator,
+ value=default_dataset_plain_text_input_and_output_separator)
+ dataset_plain_text_data_separator = gr.Textbox(
+ label="Data Separator",
+ elem_id="dataset_plain_text_data_separator",
+ placeholder=default_dataset_plain_text_data_separator,
+ value=default_dataset_plain_text_data_separator)
+ things_that_might_timeout.append(
+ dataset_text_format.change(
+ fn=handle_switch_dataset_text_format,
+ inputs=[dataset_text_format],
+ outputs=[
+ dataset_plain_text_separators_group # type: ignore
+ ]
+ ))
+
+ things_that_might_timeout.append(
+ dataset_text_load_sample_button.click(fn=load_sample_dataset_to_text_input, inputs=[
+ dataset_text_format], outputs=[dataset_text]))
+ gr.Markdown(
+ "💡 Switch to the \"Preview\" tab to verify that your inputs are correct.")
+ with gr.Tab("Preview"):
+ with gr.Row():
+ finetune_dataset_preview_info_message = gr.Markdown(
+ "Set the dataset in the \"Prepare\" tab, then preview it here.",
+ elem_id="finetune_dataset_preview_info_message"
+ )
+ finetune_dataset_preview_count = gr.Number(
+ label="Preview items count",
+ value=10,
+ # minimum=1,
+ # maximum=100,
+ precision=0,
+ elem_id="finetune_dataset_preview_count"
+ )
+ finetune_dataset_preview = gr.Dataframe(
+ wrap=True, elem_id="finetune_dataset_preview")
+ things_that_might_timeout.append(
+ load_dataset_from.change(
+ fn=handle_switch_dataset_source,
+ inputs=[load_dataset_from],
+ outputs=[
+ dataset_text_input_group,
+ dataset_from_data_dir_group
+ ] # type: ignore
+ ))
+
+ dataset_inputs = [
+ template,
+ load_dataset_from,
+ dataset_from_data_dir,
+ dataset_text,
+ dataset_text_format,
+ dataset_plain_text_input_variables_separator,
+ dataset_plain_text_input_and_output_separator,
+ dataset_plain_text_data_separator,
+ ]
+ dataset_preview_inputs = dataset_inputs + \
+ [finetune_dataset_preview_count]
+
+ with gr.Row():
+ max_seq_length = gr.Slider(
+ minimum=1, maximum=4096, value=512,
+ label="Max Sequence Length",
+ info="The maximum length of each sample text sequence. Sequences longer than this will be truncated.",
+ elem_id="finetune_max_seq_length"
+ )
+
+ train_on_inputs = gr.Checkbox(
+ label="Train on Inputs",
+ value=True,
+ info="If not enabled, inputs will be masked out in loss.",
+ elem_id="finetune_train_on_inputs"
+ )
+
+ with gr.Row():
+ # https://huggingface.co/docs/transformers/main/main_classes/trainer
+
+ micro_batch_size_default_value = 1
+
+ if Global.gpu_total_cores is not None and Global.gpu_total_memory is not None:
+ memory_per_core = Global.gpu_total_memory / Global.gpu_total_cores
+ if memory_per_core >= 6291456:
+ micro_batch_size_default_value = 8
+ elif memory_per_core >= 4000000: # ?
+ micro_batch_size_default_value = 4
+
+ with gr.Column():
+ micro_batch_size = gr.Slider(
+ minimum=1, maximum=100, step=1, value=micro_batch_size_default_value,
+ label="Micro Batch Size",
+ info="The number of examples in each mini-batch for gradient computation. A smaller micro_batch_size reduces memory usage but may increase training time."
+ )
+
+ gradient_accumulation_steps = gr.Slider(
+ minimum=1, maximum=10, step=1, value=1,
+ label="Gradient Accumulation Steps",
+ info="The number of steps to accumulate gradients before updating model parameters. This can be used to simulate a larger effective batch size without increasing memory usage."
+ )
+
+ epochs = gr.Slider(
+ minimum=1, maximum=100, step=1, value=10,
+ label="Epochs",
+ info="The number of times to iterate over the entire training dataset. A larger number of epochs may improve model performance but also increase the risk of overfitting.")
+
+ learning_rate = gr.Slider(
+ minimum=0.00001, maximum=0.01, value=3e-4,
+ label="Learning Rate",
+ info="The initial learning rate for the optimizer. A higher learning rate may speed up convergence but also cause instability or divergence. A lower learning rate may require more steps to reach optimal performance but also avoid overshooting or oscillating around local minima."
+ )
+
+ with gr.Column(elem_id="finetune_eval_data_group"):
+ evaluate_data_count = gr.Slider(
+ minimum=0, maximum=1, step=1, value=0,
+ label="Evaluation Data Count",
+ info="The number of data to be used for evaluation. This specific amount of data will be randomly chosen from the training dataset for evaluating the model's performance during the process, without contributing to the actual training.",
+ elem_id="finetune_evaluate_data_count"
+ )
+ gr.HTML(elem_classes="flex_vertical_grow_area")
+
+ with gr.Accordion("Advanced Options", open=False, elem_id="finetune_advance_options_accordion"):
+ with gr.Row(elem_id="finetune_advanced_options_checkboxes"):
+ load_in_8bit = gr.Checkbox(
+ label="8bit", value=Config.load_8bit)
+ fp16 = gr.Checkbox(label="FP16", value=True)
+ bf16 = gr.Checkbox(label="BF16", value=False)
+ gradient_checkpointing = gr.Checkbox(
+ label="gradient_checkpointing", value=False)
+ with gr.Column(variant="panel", elem_id="finetune_additional_training_arguments_box"):
+ gr.Textbox(
+ label="Additional Training Arguments",
+ info="Additional training arguments to be passed to the Trainer. Note that this can override ALL other arguments set elsewhere. See https://bit.ly/hf20-transformers-training-arguments for more details.",
+ elem_id="finetune_additional_training_arguments_textbox_for_label_display"
+ )
+ additional_training_arguments = gr.Code(
+ label="JSON",
+ language="json",
+ value="",
+ lines=2,
+ elem_id="finetune_additional_training_arguments")
+
+ with gr.Box(elem_id="finetune_continue_from_model_box"):
+ with gr.Row():
+ continue_from_model = gr.Dropdown(
+ value="-",
+ label="Continue from Model",
+ choices=["-"],
+ allow_custom_value=True,
+ elem_id="finetune_continue_from_model"
+ )
+ continue_from_checkpoint = gr.Dropdown(
+ value="-",
+ label="Resume from Checkpoint",
+ choices=["-"],
+ elem_id="finetune_continue_from_checkpoint")
+ with gr.Column():
+ load_params_from_model_btn = gr.Button(
+ "Load training parameters from selected model", visible=False)
+ load_params_from_model_btn.style(
+ full_width=False,
+ size="sm")
+ load_params_from_model_message = gr.Markdown(
+ "", visible=False)
+
+ things_that_might_timeout.append(
+ continue_from_model.change(
+ fn=handle_continue_from_model_change,
+ inputs=[continue_from_model],
+ outputs=[
+ continue_from_checkpoint,
+ load_params_from_model_btn,
+ load_params_from_model_message
+ ]
+ )
+ )
+
+ with gr.Column():
+ lora_r = gr.Slider(
+ minimum=1, maximum=16, step=1, value=8,
+ label="LoRA R",
+ info="The rank parameter for LoRA, which controls the dimensionality of the rank decomposition matrices. A larger lora_r increases the expressiveness and flexibility of LoRA but also increases the number of trainable parameters and memory usage."
+ )
+
+ lora_alpha = gr.Slider(
+ minimum=1, maximum=128, step=1, value=16,
+ label="LoRA Alpha",
+ info="The scaling parameter for LoRA, which controls how much LoRA affects the original pre-trained model weights. A larger lora_alpha amplifies the impact of LoRA but may also distort or override the pre-trained knowledge."
+ )
+
+ lora_dropout = gr.Slider(
+ minimum=0, maximum=1, value=0.05,
+ label="LoRA Dropout",
+ info="The dropout probability for LoRA, which controls the fraction of LoRA parameters that are set to zero during training. A larger lora_dropout increases the regularization effect of LoRA but also increases the risk of underfitting."
+ )
+
+ with gr.Column(elem_id="finetune_lora_target_modules_box"):
+ lora_target_modules = gr.CheckboxGroup(
+ label="LoRA Target Modules",
+ choices=default_lora_target_module_choices,
+ value=["q_proj", "v_proj"],
+ info="Modules to replace with LoRA.",
+ elem_id="finetune_lora_target_modules"
+ )
+ lora_target_module_choices = gr.State(
+ value=default_lora_target_module_choices) # type: ignore
+ with gr.Box(elem_id="finetune_lora_target_modules_add_box"):
+ with gr.Row():
+ lora_target_modules_add = gr.Textbox(
+ lines=1, max_lines=1, show_label=False,
+ elem_id="finetune_lora_target_modules_add"
+ )
+ lora_target_modules_add_btn = gr.Button(
+ "Add",
+ elem_id="finetune_lora_target_modules_add_btn"
+ )
+ lora_target_modules_add_btn.style(
+ full_width=False, size="sm")
+ things_that_might_timeout.append(lora_target_modules_add_btn.click(
+ handle_lora_target_modules_add,
+ inputs=[lora_target_module_choices,
+ lora_target_modules_add, lora_target_modules],
+ outputs=[lora_target_module_choices,
+ lora_target_modules_add, lora_target_modules],
+ ))
+
+ with gr.Accordion("Advanced LoRA Options", open=False, elem_id="finetune_advance_lora_options_accordion"):
+ with gr.Column(elem_id="finetune_lora_modules_to_save_box"):
+ lora_modules_to_save = gr.CheckboxGroup(
+ label="LoRA Modules To Save",
+ choices=default_lora_modules_to_save_choices,
+ value=[],
+ # info="",
+ elem_id="finetune_lora_modules_to_save"
+ )
+ lora_modules_to_save_choices = gr.State(
+ value=default_lora_modules_to_save_choices) # type: ignore
+ with gr.Box(elem_id="finetune_lora_modules_to_save_add_box"):
+ with gr.Row():
+ lora_modules_to_save_add = gr.Textbox(
+ lines=1, max_lines=1, show_label=False,
+ elem_id="finetune_lora_modules_to_save_add"
+ )
+ lora_modules_to_save_add_btn = gr.Button(
+ "Add",
+ elem_id="finetune_lora_modules_to_save_add_btn"
+ )
+ lora_modules_to_save_add_btn.style(
+ full_width=False, size="sm")
+ things_that_might_timeout.append(lora_modules_to_save_add_btn.click(
+ handle_lora_modules_to_save_add,
+ inputs=[lora_modules_to_save_choices,
+ lora_modules_to_save_add, lora_modules_to_save],
+ outputs=[lora_modules_to_save_choices,
+ lora_modules_to_save_add, lora_modules_to_save],
+ ))
+
+ with gr.Column(variant="panel", elem_id="finetune_additional_lora_config_box"):
+ gr.Textbox(
+ label="Additional LoRA Config",
+ info="Additional LoraConfig. Note that this can override ALL other arguments set elsewhere.",
+ elem_id="finetune_additional_lora_config_textbox_for_label_display"
+ )
+ additional_lora_config = gr.Code(
+ label="JSON",
+ language="json",
+ value="",
+ lines=2,
+ elem_id="finetune_additional_lora_config")
+
+ gr.HTML(elem_classes="flex_vertical_grow_area no_limit")
+
+ with gr.Column(elem_id="finetune_log_and_save_options_group_container"):
+ with gr.Row(elem_id="finetune_log_and_save_options_group"):
+ logging_steps = gr.Number(
+ label="Logging Steps",
+ precision=0,
+ value=10,
+ elem_id="finetune_logging_steps"
+ )
+ save_steps = gr.Number(
+ label="Steps Per Save",
+ precision=0,
+ value=500,
+ elem_id="finetune_save_steps"
+ )
+ save_total_limit = gr.Number(
+ label="Saved Checkpoints Limit",
+ precision=0,
+ value=5,
+ elem_id="finetune_save_total_limit"
+ )
+
+ with gr.Column(elem_id="finetune_model_name_group"):
+ model_name = gr.Textbox(
+ lines=1, label="LoRA Model Name", value=random_name,
+ max_lines=1,
+ info="The name of the new LoRA model.",
+ elem_id="finetune_model_name",
+ )
+
+ with gr.Row():
+ with gr.Column():
+ pass
+ with gr.Column():
+
+ with gr.Row():
+ train_btn = gr.Button(
+ "Train", variant="primary", label="Train",
+ elem_id="finetune_start_btn"
+ )
+
+ abort_button = gr.Button(
+ "Abort", label="Abort",
+ elem_id="finetune_stop_btn"
+ )
+ confirm_abort_button = gr.Button(
+ "Confirm Abort", label="Confirm Abort", variant="stop",
+ elem_id="finetune_confirm_stop_btn"
+ )
+
+ things_that_might_timeout.append(reload_selections_button.click(
+ reload_selections,
+ inputs=[template, dataset_from_data_dir],
+ outputs=[template, dataset_from_data_dir, continue_from_model],
+ ))
+
+ for i in dataset_preview_inputs:
+ things_that_might_timeout.append(
+ i.change(
+ fn=refresh_preview,
+ inputs=dataset_preview_inputs,
+ outputs=[
+ finetune_dataset_preview,
+ finetune_dataset_preview_info_message,
+ dataset_from_text_message,
+ dataset_from_data_dir_message
+ ]
+ ).then(
+ fn=refresh_dataset_items_count,
+ inputs=dataset_preview_inputs,
+ outputs=[
+ finetune_dataset_preview_info_message,
+ dataset_from_text_message,
+ dataset_from_data_dir_message,
+ evaluate_data_count,
+ ]
+ ))
+
+ finetune_args = [
+ max_seq_length,
+ evaluate_data_count,
+ micro_batch_size,
+ gradient_accumulation_steps,
+ epochs,
+ learning_rate,
+ train_on_inputs,
+ lora_r,
+ lora_alpha,
+ lora_dropout,
+ lora_target_modules,
+ lora_modules_to_save,
+ load_in_8bit,
+ fp16,
+ bf16,
+ gradient_checkpointing,
+ save_steps,
+ save_total_limit,
+ logging_steps,
+ additional_training_arguments,
+ additional_lora_config,
+ ]
+
+ things_that_might_timeout.append(
+ load_params_from_model_btn.click(
+ fn=handle_load_params_from_model,
+ inputs=(
+ [continue_from_model] +
+ [template, load_dataset_from, dataset_from_data_dir] +
+ finetune_args +
+ [lora_target_module_choices, lora_modules_to_save_choices]
+ ), # type: ignore
+ outputs=(
+ [load_params_from_model_message] +
+ [template, load_dataset_from, dataset_from_data_dir] +
+ finetune_args +
+ [lora_target_module_choices, lora_modules_to_save_choices]
+ ) # type: ignore
+ )
+ )
+
+ train_status = gr.HTML(
+ "",
+ label="Train Output",
+ elem_id="finetune_training_status")
+
+ with gr.Column(visible=False, elem_id="finetune_loss_plot_container") as loss_plot_container:
+ loss_plot = gr.Plot(
+ visible=False, show_label=False,
+ elem_id="finetune_loss_plot")
+
+ training_indicator = gr.HTML(
+ "training_indicator", visible=False, elem_id="finetune_training_indicator")
+
+ train_start = train_btn.click(
+ fn=do_train,
+ inputs=(dataset_inputs + finetune_args + [
+ model_name,
+ continue_from_model,
+ continue_from_checkpoint,
+ ]),
+ outputs=[train_status, training_indicator,
+ loss_plot_container, loss_plot]
+ )
+
+ # controlled by JS, shows the confirm_abort_button
+ abort_button.click(None, None, None, None)
+ confirm_abort_button.click(
+ fn=do_abort_training,
+ inputs=None, outputs=None,
+ cancels=[train_start])
+
+ training_status_updates = finetune_ui_blocks.load(
+ fn=render_training_status,
+ inputs=None,
+ outputs=[train_status, training_indicator],
+ every=0.2
+ )
+ loss_plot_updates = finetune_ui_blocks.load(
+ fn=render_loss_plot,
+ inputs=None,
+ outputs=[loss_plot_container, loss_plot],
+ every=10
+ )
+ finetune_ui_blocks.load(_js=relative_read_file(__file__, "script.js"))
+
+ # things_that_might_timeout.append(training_status_updates)
+ stop_timeoutable_btn = gr.Button(
+ "stop not-responding elements",
+ elem_id="inference_stop_timeoutable_btn",
+ elem_classes="foot_stop_timeoutable_btn")
+ stop_timeoutable_btn.click(
+ fn=None, inputs=None, outputs=None, cancels=things_that_might_timeout)
diff --git a/llama_lora/ui/finetune/previewing.py b/llama_lora/ui/finetune/previewing.py
new file mode 100644
index 0000000000000000000000000000000000000000..59f01d61db5c885e8e0bb695c4591fdacc046ab8
--- /dev/null
+++ b/llama_lora/ui/finetune/previewing.py
@@ -0,0 +1,155 @@
+import os
+import traceback
+import re
+import gradio as gr
+import math
+
+from ...config import Config
+from ...utils.prompter import Prompter
+
+from .data_processing import get_data_from_input
+
+
+def refresh_preview(
+ template,
+ load_dataset_from,
+ dataset_from_data_dir,
+ dataset_text,
+ dataset_text_format,
+ dataset_plain_text_input_variables_separator,
+ dataset_plain_text_input_and_output_separator,
+ dataset_plain_text_data_separator,
+ max_preview_count,
+):
+ try:
+ prompter = Prompter(template)
+ variable_names = prompter.get_variable_names()
+
+ data = get_data_from_input(
+ load_dataset_from=load_dataset_from,
+ dataset_text=dataset_text,
+ dataset_text_format=dataset_text_format,
+ dataset_plain_text_input_variables_separator=dataset_plain_text_input_variables_separator,
+ dataset_plain_text_input_and_output_separator=dataset_plain_text_input_and_output_separator,
+ dataset_plain_text_data_separator=dataset_plain_text_data_separator,
+ dataset_from_data_dir=dataset_from_data_dir,
+ prompter=prompter
+ )
+
+ train_data = prompter.get_train_data_from_dataset(
+ data, max_preview_count)
+
+ train_data = train_data[:max_preview_count]
+
+ data_count = len(data)
+
+ headers = ['Prompt', 'Completion']
+ preview_data = [
+ [item.get("prompt", ""), item.get("completion", "")]
+ for item in train_data
+ ]
+
+ if not prompter.template_module:
+ variable_names = prompter.get_variable_names()
+ headers += [f"Variable: {variable_name}" for variable_name in variable_names]
+ variables = [
+ [item.get(f"_var_{name}", "") for name in variable_names]
+ for item in train_data
+ ]
+ preview_data = [d + v for d, v in zip(preview_data, variables)]
+
+ preview_info_message = f"The dataset has about {data_count} item(s)."
+ if data_count > max_preview_count:
+ preview_info_message += f" Previewing the first {max_preview_count}."
+
+ info_message = f"about {data_count} item(s)."
+ if load_dataset_from == "Data Dir":
+ info_message = "This dataset contains about " + info_message
+ update_message = gr.Markdown.update(info_message, visible=True)
+
+ return (
+ gr.Dataframe.update(
+ value={'data': preview_data, 'headers': headers}),
+ gr.Markdown.update(preview_info_message),
+ update_message,
+ update_message
+ )
+ except Exception as e:
+ update_message = gr.Markdown.update(
+ f"",
+ visible=True)
+ return (
+ gr.Dataframe.update(value={'data': [], 'headers': []}),
+ gr.Markdown.update(
+ "Set the dataset in the \"Prepare\" tab, then preview it here."),
+ update_message,
+ update_message
+ )
+
+
+def refresh_dataset_items_count(
+ template,
+ load_dataset_from,
+ dataset_from_data_dir,
+ dataset_text,
+ dataset_text_format,
+ dataset_plain_text_input_variables_separator,
+ dataset_plain_text_input_and_output_separator,
+ dataset_plain_text_data_separator,
+ max_preview_count,
+):
+ try:
+ prompter = Prompter(template)
+
+ data = get_data_from_input(
+ load_dataset_from=load_dataset_from,
+ dataset_text=dataset_text,
+ dataset_text_format=dataset_text_format,
+ dataset_plain_text_input_variables_separator=dataset_plain_text_input_variables_separator,
+ dataset_plain_text_input_and_output_separator=dataset_plain_text_input_and_output_separator,
+ dataset_plain_text_data_separator=dataset_plain_text_data_separator,
+ dataset_from_data_dir=dataset_from_data_dir,
+ prompter=prompter
+ )
+
+ train_data = prompter.get_train_data_from_dataset(
+ data)
+ data_count = len(train_data)
+
+ preview_info_message = f"The dataset contains {data_count} item(s)."
+ if data_count > max_preview_count:
+ preview_info_message += f" Previewing the first {max_preview_count}."
+
+ info_message = f"{data_count} item(s)."
+ if load_dataset_from == "Data Dir":
+ info_message = "This dataset contains " + info_message
+ update_message = gr.Markdown.update(info_message, visible=True)
+
+ return (
+ gr.Markdown.update(preview_info_message),
+ update_message,
+ update_message,
+ gr.Slider.update(maximum=math.floor(data_count / 2))
+ )
+ except Exception as e:
+ update_message = gr.Markdown.update(
+ f"",
+ visible=True)
+
+ trace = traceback.format_exc()
+ traces = [s.strip() for s in re.split("\n * File ", trace)]
+ traces_to_show = [s for s in traces if os.path.join(
+ Config.data_dir, "templates") in s]
+ traces_to_show = [re.sub(" *\n *", ": ", s) for s in traces_to_show]
+ if len(traces_to_show) > 0:
+ update_message = gr.Markdown.update(
+ f"",
+ visible=True)
+
+ return (
+ gr.Markdown.update(
+ "Set the dataset in the \"Prepare\" tab, then preview it here."),
+ update_message,
+ update_message,
+ gr.Slider.update(maximum=1)
+ )
diff --git a/llama_lora/ui/finetune/script.js b/llama_lora/ui/finetune/script.js
new file mode 100644
index 0000000000000000000000000000000000000000..f127752fdc788edda96d74fc6cdcd9f28ee89ca9
--- /dev/null
+++ b/llama_lora/ui/finetune/script.js
@@ -0,0 +1,202 @@
+function finetune_ui_blocks_js() {
+ // Auto load options
+ setTimeout(function () {
+ document.getElementById('finetune_reload_selections_button').click();
+ }, 100);
+
+ // Add tooltips
+ setTimeout(function () {
+ tippy('#finetune_reload_selections_button', {
+ placement: 'bottom-end',
+ delay: [500, 0],
+ animation: 'scale-subtle',
+ content: 'Press to reload options.',
+ });
+
+ tippy('#finetune_template', {
+ placement: 'right',
+ delay: [500, 0],
+ animation: 'scale-subtle',
+ content:
+ 'Select a template for your prompt. tloen/alpaca-lora-7b.Load training parameters from selected model button, then un-select it.',
+ allowHTML: true,
+ });
+
+ tippy('#finetune_continue_from_checkpoint', {
+ placement: 'right',
+ delay: [500, 0],
+ animation: 'scale-subtle',
+ content:
+ 'If a checkpoint is selected, training will resume from that specific checkpoint, bypassing any previously completed steps up to the checkpoint\'s moment. Load training parameters from selected model button and select the same dataset for training.',
+ allowHTML: true,
+ });
+ }, 100);
+
+ // Show/hide start and stop button base on the state.
+ setTimeout(function () {
+ // Make the '#finetune_training_indicator > .wrap' element appear
+ // if (!document.querySelector('#finetune_training_indicator > .wrap')) {
+ // document.getElementById('finetune_confirm_stop_btn').click();
+ // }
+
+ setTimeout(function () {
+ let resetStopButtonTimer;
+ document
+ .getElementById('finetune_stop_btn')
+ .addEventListener('click', function () {
+ if (resetStopButtonTimer) clearTimeout(resetStopButtonTimer);
+ resetStopButtonTimer = setTimeout(function () {
+ document.getElementById('finetune_stop_btn').style.display = 'block';
+ document.getElementById('finetune_confirm_stop_btn').style.display =
+ 'none';
+ }, 5000);
+ document.getElementById('finetune_confirm_stop_btn').style['pointer-events'] =
+ 'none';
+ setTimeout(function () {
+ document.getElementById('finetune_confirm_stop_btn').style['pointer-events'] =
+ 'inherit';
+ }, 300);
+ document.getElementById('finetune_stop_btn').style.display = 'none';
+ document.getElementById('finetune_confirm_stop_btn').style.display =
+ 'block';
+ });
+ // const training_indicator_wrap_element = document.querySelector(
+ // '#finetune_training_indicator > .wrap'
+ // );
+ const training_indicator_element = document.querySelector(
+ '#finetune_training_indicator'
+ );
+ let isTraining = undefined;
+ function handle_training_indicator_change() {
+ // const wrapperHidden = Array.from(training_indicator_wrap_element.classList).includes('hide');
+ const hidden = Array.from(training_indicator_element.classList).includes('hidden');
+ const newIsTraining = !(/* wrapperHidden && */ hidden);
+ if (newIsTraining === isTraining) return;
+ isTraining = newIsTraining;
+ if (!isTraining) {
+ if (resetStopButtonTimer) clearTimeout(resetStopButtonTimer);
+ document.getElementById('finetune_start_btn').style.display = 'block';
+ document.getElementById('finetune_stop_btn').style.display = 'none';
+ document.getElementById('finetune_confirm_stop_btn').style.display =
+ 'none';
+ } else {
+ document.getElementById('finetune_start_btn').style.display = 'none';
+ document.getElementById('finetune_stop_btn').style.display = 'block';
+ document.getElementById('finetune_confirm_stop_btn').style.display =
+ 'none';
+ }
+ }
+ // new MutationObserver(function (mutationsList, observer) {
+ // handle_training_indicator_change();
+ // }).observe(training_indicator_wrap_element, {
+ // attributes: true,
+ // attributeFilter: ['class'],
+ // });
+ new MutationObserver(function (mutationsList, observer) {
+ handle_training_indicator_change();
+ }).observe(training_indicator_element, {
+ attributes: true,
+ attributeFilter: ['class'],
+ });
+ handle_training_indicator_change();
+ }, 500);
+ }, 0);
+
+ return [];
+}
diff --git a/llama_lora/ui/finetune/style.css b/llama_lora/ui/finetune/style.css
new file mode 100644
index 0000000000000000000000000000000000000000..b5d280bb2c0429fd5040984f36e25dd3a4c37582
--- /dev/null
+++ b/llama_lora/ui/finetune/style.css
@@ -0,0 +1,421 @@
+#finetune_dataset_text_load_sample_button {
+ margin: -4px 12px 8px;
+}
+
+#finetune_reload_selections_button {
+ position: absolute;
+ top: 0;
+ right: 0;
+ margin: 16px;
+ margin-bottom: auto;
+ height: 42px !important;
+ min-width: 42px !important;
+ width: 42px !important;
+ z-index: 1;
+}
+
+#finetune_dataset_from_data_dir {
+ border: 0;
+ box-shadow: none;
+}
+
+#finetune_ui_content > .tabs > .tab-nav::before {
+ content: "Training Dataset:";
+ display: flex;
+ justify-content: center;
+ align-items: center;
+ padding-right: 12px;
+ padding-left: 8px;
+}
+
+#finetune_template,
+#finetune_template + * {
+ border: 0;
+ box-shadow: none;
+}
+
+#finetune_dataset_text_input_group .form {
+ border: 0;
+ box-shadow: none;
+ padding: 0;
+}
+
+#finetune_dataset_text_input_textbox > .wrap:last-of-type {
+ margin-top: -20px;
+}
+
+#finetune_dataset_plain_text_separators_group * {
+ font-size: 0.8rem;
+}
+#finetune_dataset_plain_text_separators_group textarea {
+ height: auto !important;
+}
+#finetune_dataset_plain_text_separators_group > .form {
+ gap: 0 !important;
+}
+
+#finetune_dataset_from_text_message p,
+#finetune_dataset_from_text_message + * p {
+ font-size: 80%;
+}
+#finetune_dataset_from_text_message,
+#finetune_dataset_from_text_message *,
+#finetune_dataset_from_text_message + *,
+#finetune_dataset_from_text_message + * * {
+ display: inline;
+}
+
+
+#finetune_dataset_from_data_dir_message,
+#finetune_dataset_from_data_dir_message * {
+ min-height: 0 !important;
+}
+#finetune_dataset_from_data_dir_message {
+ margin: -20px 24px 0;
+ font-size: 0.8rem;
+}
+
+#finetune_dataset_from_text_message > .wrap > *:first-child,
+#finetune_dataset_from_data_dir_message > .wrap > *:first-child {
+ display: none;
+}
+#finetune_dataset_from_data_dir_message > .wrap {
+ top: -18px;
+}
+#finetune_dataset_from_text_message > .wrap svg,
+#finetune_dataset_from_data_dir_message > .wrap svg {
+ margin: -32px -16px;
+}
+
+#finetune_continue_from_model_box {
+ /* padding: 0; */
+}
+#finetune_continue_from_model_box .block {
+ border: 0;
+ box-shadow: none;
+ padding: 0;
+}
+#finetune_continue_from_model_box > * {
+ /* gap: 0; */
+}
+#finetune_continue_from_model_box button {
+ margin-top: 16px;
+}
+#finetune_continue_from_model {
+ flex-grow: 2;
+}
+
+.finetune_dataset_error_message {
+ color: var(--error-text-color) !important;
+}
+
+#finetune_dataset_preview_info_message {
+ align-items: flex-end;
+ flex-direction: row;
+ display: flex;
+ margin-bottom: -4px;
+}
+
+#finetune_dataset_preview td {
+ white-space: pre-wrap;
+}
+
+/*
+#finetune_dataset_preview {
+ max-height: 100vh;
+ overflow: auto;
+ border: var(--block-border-width) solid var(--border-color-primary);
+ border-radius: var(--radius-lg);
+}
+#finetune_dataset_preview .table-wrap {
+ border: 0 !important;
+}
+*/
+
+#finetune_max_seq_length {
+ flex: 2;
+}
+
+#finetune_lora_target_modules_box,
+#finetune_lora_target_modules_box + #finetune_lora_modules_to_save_box {
+ margin-top: calc((var(--layout-gap) + 8px) * -1);
+ flex-grow: 0 !important;
+}
+#finetune_lora_target_modules_box > .form,
+#finetune_lora_target_modules_box + #finetune_lora_modules_to_save_box > .form {
+ padding-top: calc((var(--layout-gap) + 8px) / 3);
+ border-top: 0;
+ border-top-left-radius: 0;
+ border-top-right-radius: 0;
+ background: var(--block-background-fill);
+ position: relative;
+}
+#finetune_lora_target_modules_box > .form::before,
+#finetune_lora_target_modules_box + #finetune_lora_modules_to_save_box > .form::before {
+ content: "";
+ display: block;
+ position: absolute;
+ top: calc((var(--layout-gap) + 8px) / 3);
+ left: 0;
+ right: 0;
+ height: 1px;
+ z-index: 1;
+ background: var(--block-border-color);
+}
+#finetune_lora_target_modules_add_box,
+#finetune_lora_modules_to_save_add_box {
+ margin-top: -24px;
+ padding-top: 8px;
+ border-top-left-radius: 0;
+ border-top-right-radius: 0;
+ border-top: 0;
+}
+#finetune_lora_target_modules_add_box > * > .form,
+#finetune_lora_modules_to_save_add_box > * > .form {
+ border: 0;
+ box-shadow: none;
+}
+#finetune_lora_target_modules_add,
+#finetune_lora_modules_to_save_add {
+ padding: 0;
+}
+#finetune_lora_target_modules_add input,
+#finetune_lora_modules_to_save_add input {
+ padding: 4px 8px;
+}
+#finetune_lora_target_modules_add_btn,
+#finetune_lora_modules_to_save_add_btn {
+ min-width: 60px;
+}
+
+#finetune_advance_lora_options_accordion > *:last-child:not(.label-wrap) > *:first-child {
+ margin-top: 8px;
+}
+#finetune_advance_lora_options_accordion #finetune_lora_modules_to_save,
+#finetune_advance_lora_options_accordion #finetune_lora_modules_to_save_add_box {
+ padding: var(--spacing-lg);
+ background: var(--panel-background-fill);
+ border: 0;
+}
+#finetune_advance_lora_options_accordion #finetune_lora_modules_to_save_box > .form,
+#finetune_advance_lora_options_accordion #finetune_lora_modules_to_save,
+#finetune_advance_lora_options_accordion #finetune_lora_modules_to_save_add_box {
+ border: 0;
+}
+
+#finetune_save_total_limit,
+#finetune_save_steps,
+#finetune_logging_steps {
+ min-width: min(120px,100%) !important;
+ padding-top: 4px;
+}
+#finetune_save_total_limit span,
+#finetune_save_steps span,
+#finetune_logging_steps span {
+ font-size: 12px;
+ margin-bottom: 5px;
+}
+#finetune_save_total_limit input,
+#finetune_save_steps input,
+#finetune_logging_steps input {
+ padding: 4px 8px;
+}
+
+#finetune_advance_options_accordion > *:last-child:not(.label-wrap) > *:first-child {
+ margin-top: 8px;
+}
+#finetune_advanced_options_checkboxes > * > * {
+ min-width: auto;
+}
+
+#finetune_log_and_save_options_group_container {
+ flex-grow: 0 !important;
+}
+#finetune_model_name_group {
+ flex-grow: 0 !important;
+}
+
+#finetune_eval_data_group {
+ flex-grow: 0 !important;
+}
+
+#finetune_additional_training_arguments_box > .form,
+#finetune_additional_lora_config_box > .form {
+ border: 0;
+ background: transparent;
+}
+.form:has(> #finetune_additional_training_arguments_textbox_for_label_display),
+.form:has(> #finetune_additional_lora_config_textbox_for_label_display) {
+ box-shadow: none;
+ border-radius: 0;
+ margin-bottom: -8px;
+}
+#finetune_additional_training_arguments_textbox_for_label_display,
+#finetune_additional_lora_config_textbox_for_label_display {
+ padding: 0;
+ margin-bottom: -8px;
+ background: transparent;
+}
+#finetune_additional_training_arguments_textbox_for_label_display textarea,
+#finetune_additional_lora_config_textbox_for_label_display textarea {
+ display: none;
+}
+
+#finetune_training_status > .wrap,
+#finetune_loss_plot_container > .wrap,
+#finetune_loss_plot > .wrap {
+ border: 0;
+ background: transparent;
+ pointer-events: none;
+ top: 0;
+ bottom: 0;
+ left: 0;
+ right: 0;
+}
+#finetune_training_status > .wrap:not(.generating)::after {
+ content: "Refresh the page if this takes too long.";
+ position: absolute;
+ top: 0;
+ left: 0;
+ right: 0;
+ bottom: 0;
+ padding-top: 64px;
+ opacity: 0.5;
+ text-align: center;
+}
+#finetune_training_status > .wrap .meta-text-center {
+ transform: none !important;
+}
+
+#finetune_training_status .progress-block {
+ min-height: 100px;
+ display: flex;
+ flex-direction: column;
+ justify-content: center;
+ align-items: center;
+ background: var(--panel-background-fill);
+ border-radius: var(--radius-lg);
+ border: var(--block-border-width) solid var(--border-color-primary);
+ padding: var(--block-padding);
+}
+#finetune_training_status .progress-block.is_training {
+ min-height: 160px;
+}
+#finetune_training_status .progress-block .empty-text {
+ text-transform: uppercase;
+ font-weight: 700;
+ font-size: 120%;
+ opacity: 0.12;
+}
+#finetune_training_status .progress-block .meta-text {
+ position: absolute;
+ top: 0;
+ right: 0;
+ z-index: var(--layer-2);
+ padding: var(--size-1) var(--size-2);
+ font-size: var(--text-sm);
+ font-family: var(--font-mono);
+ text-align: right;
+}
+#finetune_training_status .progress-block .status {
+ white-space: pre-wrap;
+}
+#finetune_training_status .progress-block .progress-level {
+ flex-grow: 1;
+ display: flex;
+ flex-direction: column;
+ justify-content: center;
+ align-items: center;
+ z-index: var(--layer-2);
+ width: var(--size-full);
+ padding: 8px 0;
+ text-align: center;
+}
+#finetune_training_status .progress-block .progress-level-inner {
+ margin: var(--size-2) auto;
+ color: var(--body-text-color);
+ font-size: var(--text-sm);
+ font-family: var(--font-mono);
+}
+#finetune_training_status .progress-block .progress-bar-wrap {
+ border: 1px solid var(--border-color-primary);
+ background: var(--background-fill-primary);
+ width: 55.5%;
+ height: var(--size-4);
+}
+#finetune_training_status .progress-block .progress-bar {
+ transform-origin: left;
+ background-color: var(--loader-color);
+ width: var(--size-full);
+ height: var(--size-full);
+ transition: all 150ms ease 0s;
+}
+
+#finetune_training_status .progress-block .params-info {
+ font-size: var(--text-sm);
+ font-weight: var(--weight-light);
+ margin-top: 8px;
+ margin-bottom: -4px !important;
+ opacity: 0.4;
+}
+#finetune_training_status .progress-block .progress-level + .params-info {
+ margin-top: -8px;
+}
+
+#finetune_training_status .progress-block .output {
+ display: flex;
+ flex-direction: column;
+ justify-content: center;
+ align-items: center;
+}
+#finetune_training_status .progress-block .output .title {
+ padding: var(--size-1) var(--size-3);
+ font-weight: var(--weight-bold);
+ font-size: var(--text-lg);
+ line-height: var(--line-xs);
+}
+#finetune_training_status .progress-block .output .message {
+ padding: var(--size-1) var(--size-3);
+ color: var(--body-text-color) !important;
+ font-family: var(--font-mono);
+ white-space: pre-wrap;
+}
+
+#finetune_training_status .progress-block .error {
+ display: flex;
+ flex-direction: column;
+ justify-content: center;
+ align-items: center;
+}
+#finetune_training_status .progress-block .error .title {
+ padding: var(--size-1) var(--size-3);
+ color: var(--color-red-500);
+ font-weight: var(--weight-bold);
+ font-size: var(--text-lg);
+ line-height: var(--line-xs);
+}
+#finetune_training_status .progress-block .error .error-message {
+ padding: var(--size-1) var(--size-3);
+ color: var(--body-text-color) !important;
+ font-family: var(--font-mono);
+ white-space: pre-wrap;
+}
+#finetune_training_status .progress-block.is_error {
+ /* background: var(--error-background-fill) !important; */
+ border: 1px solid var(--error-border-color) !important;
+}
+#finetune_loss_plot {
+ padding: var(--block-padding);
+}
+#finetune_loss_plot .altair {
+ overflow: auto !important;
+}
+#finetune_loss_plot .altair > * {
+ margin: auto !important;
+}
+#finetune_loss_plot .vega-embed summary {
+ border: 0;
+ box-shadow: none;
+}
+
+#finetune_training_indicator { display: none; }
diff --git a/llama_lora/ui/finetune/training.py b/llama_lora/ui/finetune/training.py
new file mode 100644
index 0000000000000000000000000000000000000000..e73713a823fdaa1000032d83246adfa18c1be014
--- /dev/null
+++ b/llama_lora/ui/finetune/training.py
@@ -0,0 +1,525 @@
+import os
+import json
+import time
+import math
+import datetime
+import pytz
+import socket
+import threading
+import traceback
+import altair as alt
+import pandas as pd
+import gradio as gr
+
+from huggingface_hub import try_to_load_from_cache, snapshot_download
+from transformers import TrainingArguments
+
+from ...config import Config
+from ...globals import Global
+from ...models import clear_cache, unload_models
+from ...utils.prompter import Prompter
+from ...utils.sample_evenly import sample_evenly
+from ..trainer_callback import (
+ UiTrainerCallback, reset_training_status,
+ update_training_states, set_train_output
+)
+
+from .data_processing import get_data_from_input
+
+
+def status_message_callback(message):
+ if Global.should_stop_training:
+ return True
+
+ Global.training_status_text = message
+
+
+def params_info_callback(all_params, trainable_params):
+ Global.training_params_info_text = f"Params: {trainable_params}/{all_params} ({100 * trainable_params / all_params:.4f}% trainable)"
+
+
+def do_train(
+ # Dataset
+ template,
+ load_dataset_from,
+ dataset_from_data_dir,
+ dataset_text,
+ dataset_text_format,
+ dataset_plain_text_input_variables_separator,
+ dataset_plain_text_input_and_output_separator,
+ dataset_plain_text_data_separator,
+ # Training Options
+ max_seq_length,
+ evaluate_data_count,
+ micro_batch_size,
+ gradient_accumulation_steps,
+ epochs,
+ learning_rate,
+ train_on_inputs,
+ lora_r,
+ lora_alpha,
+ lora_dropout,
+ lora_target_modules,
+ lora_modules_to_save,
+ load_in_8bit,
+ fp16,
+ bf16,
+ gradient_checkpointing,
+ save_steps,
+ save_total_limit,
+ logging_steps,
+ additional_training_arguments,
+ additional_lora_config,
+ model_name,
+ continue_from_model,
+ continue_from_checkpoint,
+ progress=gr.Progress(track_tqdm=False),
+):
+ if Global.is_training or Global.is_train_starting:
+ return render_training_status() + render_loss_plot()
+
+ reset_training_status()
+ Global.is_train_starting = True
+
+ try:
+ base_model_name = Global.base_model_name
+ tokenizer_name = Global.tokenizer_name or Global.base_model_name
+
+ resume_from_checkpoint_param = None
+ if continue_from_model == "-" or continue_from_model == "None":
+ continue_from_model = None
+ if continue_from_checkpoint == "-" or continue_from_checkpoint == "None":
+ continue_from_checkpoint = None
+ if continue_from_model:
+ resume_from_model_path = os.path.join(
+ Config.data_dir, "lora_models", continue_from_model)
+ resume_from_checkpoint_param = resume_from_model_path
+ if continue_from_checkpoint:
+ resume_from_checkpoint_param = os.path.join(
+ resume_from_checkpoint_param, continue_from_checkpoint)
+ will_be_resume_from_checkpoint_file = os.path.join(
+ resume_from_checkpoint_param, "pytorch_model.bin")
+ if not os.path.exists(will_be_resume_from_checkpoint_file):
+ raise ValueError(
+ f"Unable to resume from checkpoint {continue_from_model}/{continue_from_checkpoint}. Resuming is only possible from checkpoints stored locally in the data directory. Please ensure that the file '{will_be_resume_from_checkpoint_file}' exists.")
+ else:
+ will_be_resume_from_checkpoint_file = os.path.join(
+ resume_from_checkpoint_param, "adapter_model.bin")
+ if not os.path.exists(will_be_resume_from_checkpoint_file):
+ # Try to get model in Hugging Face cache
+ resume_from_checkpoint_param = None
+ possible_hf_model_name = None
+ possible_model_info_file = os.path.join(
+ resume_from_model_path, "info.json")
+ if "/" in continue_from_model:
+ possible_hf_model_name = continue_from_model
+ elif os.path.exists(possible_model_info_file):
+ with open(possible_model_info_file, "r") as file:
+ model_info = json.load(file)
+ possible_hf_model_name = model_info.get(
+ "hf_model_name")
+ if possible_hf_model_name:
+ possible_hf_model_cached_path = try_to_load_from_cache(
+ possible_hf_model_name, 'adapter_model.bin')
+ if not possible_hf_model_cached_path:
+ snapshot_download(possible_hf_model_name)
+ possible_hf_model_cached_path = try_to_load_from_cache(
+ possible_hf_model_name, 'adapter_model.bin')
+ if possible_hf_model_cached_path:
+ resume_from_checkpoint_param = os.path.dirname(
+ possible_hf_model_cached_path)
+
+ if not resume_from_checkpoint_param:
+ raise ValueError(
+ f"Unable to continue from model {continue_from_model}. Continuation is only possible from models stored locally in the data directory. Please ensure that the file '{will_be_resume_from_checkpoint_file}' exists.")
+
+ output_dir = os.path.join(Config.data_dir, "lora_models", model_name)
+ if os.path.exists(output_dir):
+ if (not os.path.isdir(output_dir)) or os.path.exists(os.path.join(output_dir, 'adapter_config.json')):
+ raise ValueError(
+ f"The output directory already exists and is not empty. ({output_dir})")
+
+ wandb_group = template
+ wandb_tags = [f"template:{template}"]
+ if load_dataset_from == "Data Dir" and dataset_from_data_dir:
+ wandb_group += f"/{dataset_from_data_dir}"
+ wandb_tags.append(f"dataset:{dataset_from_data_dir}")
+
+ finetune_args = {
+ 'base_model': base_model_name,
+ 'tokenizer': tokenizer_name,
+ 'output_dir': output_dir,
+ 'micro_batch_size': micro_batch_size,
+ 'gradient_accumulation_steps': gradient_accumulation_steps,
+ 'num_train_epochs': epochs,
+ 'learning_rate': learning_rate,
+ 'cutoff_len': max_seq_length,
+ 'val_set_size': evaluate_data_count,
+ 'lora_r': lora_r,
+ 'lora_alpha': lora_alpha,
+ 'lora_dropout': lora_dropout,
+ 'lora_target_modules': lora_target_modules,
+ 'lora_modules_to_save': lora_modules_to_save,
+ 'train_on_inputs': train_on_inputs,
+ 'load_in_8bit': load_in_8bit,
+ 'fp16': fp16,
+ 'bf16': bf16,
+ 'gradient_checkpointing': gradient_checkpointing,
+ 'group_by_length': False,
+ 'resume_from_checkpoint': resume_from_checkpoint_param,
+ 'save_steps': save_steps,
+ 'save_total_limit': save_total_limit,
+ 'logging_steps': logging_steps,
+ 'additional_training_arguments': additional_training_arguments,
+ 'additional_lora_config': additional_lora_config,
+ 'wandb_api_key': Config.wandb_api_key,
+ 'wandb_project': Config.default_wandb_project if Config.enable_wandb else None,
+ 'wandb_group': wandb_group,
+ 'wandb_run_name': model_name,
+ 'wandb_tags': wandb_tags
+ }
+
+ prompter = Prompter(template)
+ data = get_data_from_input(
+ load_dataset_from=load_dataset_from,
+ dataset_text=dataset_text,
+ dataset_text_format=dataset_text_format,
+ dataset_plain_text_input_variables_separator=dataset_plain_text_input_variables_separator,
+ dataset_plain_text_input_and_output_separator=dataset_plain_text_input_and_output_separator,
+ dataset_plain_text_data_separator=dataset_plain_text_data_separator,
+ dataset_from_data_dir=dataset_from_data_dir,
+ prompter=prompter
+ )
+
+ def training():
+ Global.is_training = True
+
+ try:
+ # Need RAM for training
+ unload_models()
+ Global.new_base_model_that_is_ready_to_be_used = None
+ Global.name_of_new_base_model_that_is_ready_to_be_used = None
+ clear_cache()
+
+ train_data = prompter.get_train_data_from_dataset(data)
+
+ if Config.ui_dev_mode:
+ Global.training_args = TrainingArguments(
+ logging_steps=logging_steps, output_dir=""
+ )
+
+ message = "Currently in UI dev mode, not doing the actual training."
+ message += f"\n\nArgs: {json.dumps(finetune_args, indent=2)}"
+ message += f"\n\nTrain data (first 5):\n{json.dumps(train_data[:5], indent=2)}"
+
+ print(message)
+
+ total_epochs = epochs
+ total_steps = len(train_data) * epochs
+ if total_steps < 1500:
+ total_steps = 1500
+ log_history = []
+ initial_loss = 2
+ loss_decay_rate = 0.8
+ for i in range(total_steps):
+ if (Global.should_stop_training):
+ break
+
+ current_step = i + 1
+ current_epoch = i / (total_steps / total_epochs)
+
+ if (current_step % logging_steps == 0):
+ loss = initial_loss * \
+ math.exp(-loss_decay_rate * current_epoch)
+ log_history.append({
+ 'loss': loss,
+ 'learning_rate': 0.0001,
+ 'epoch': current_epoch
+ })
+
+ update_training_states(
+ total_steps=total_steps,
+ current_step=current_step,
+ total_epochs=total_epochs,
+ current_epoch=current_epoch,
+ log_history=log_history
+ )
+ time.sleep(0.01)
+
+ result_message = set_train_output(message)
+ print(result_message)
+ time.sleep(3)
+ Global.is_training = False
+ return
+
+ training_callbacks = [UiTrainerCallback]
+
+ if not os.path.exists(output_dir):
+ os.makedirs(output_dir)
+
+ with open(os.path.join(output_dir, "info.json"), 'w') as info_json_file:
+ dataset_name = "N/A (from text input)"
+ if load_dataset_from == "Data Dir":
+ dataset_name = dataset_from_data_dir
+
+ info = {
+ 'base_model': base_model_name,
+ 'prompt_template': template,
+ 'dataset_name': dataset_name,
+ 'dataset_rows': len(train_data),
+ 'trained_on_machine': socket.gethostname(),
+ 'timestamp': time.time(),
+ }
+ if continue_from_model:
+ info['continued_from_model'] = continue_from_model
+ if continue_from_checkpoint:
+ info['continued_from_checkpoint'] = continue_from_checkpoint
+
+ if Global.version:
+ info['tuner_version'] = Global.version
+
+ json.dump(info, info_json_file, indent=2)
+
+ train_output = Global.finetune_train_fn(
+ train_data=train_data,
+ callbacks=training_callbacks,
+ status_message_callback=status_message_callback,
+ params_info_callback=params_info_callback,
+ additional_wandb_config=info,
+ **finetune_args,
+ )
+
+ result_message = set_train_output(train_output)
+ print(result_message + "\n" + str(train_output))
+
+ clear_cache()
+
+ Global.is_training = False
+
+ except Exception as e:
+ traceback.print_exc()
+ Global.training_error_message = str(e)
+ finally:
+ Global.is_training = False
+
+ training_thread = threading.Thread(target=training)
+ training_thread.daemon = True
+ training_thread.start()
+
+ except Exception as e:
+ Global.is_training = False
+ traceback.print_exc()
+ Global.training_error_message = str(e)
+ finally:
+ Global.is_train_starting = False
+
+ return render_training_status() + render_loss_plot()
+
+
+def render_training_status():
+ if not Global.is_training:
+ if Global.is_train_starting:
+ html_content = """
+ tloen/alpaca-lora-7b).',
+ allowHTML: true,
+ });
+
+ tippy('#inference_prompt_template', {
+ placement: 'top-start',
+ delay: [500, 0],
+ animation: 'scale-subtle',
+ content:
+ 'Templates are loaded from the "templates" folder of your data directory. Be sure to select the template that matches your selected LoRA model to get the best results.',
+ });
+
+ tippy('#inference_reload_selections_button', {
+ placement: 'bottom-end',
+ delay: [500, 0],
+ animation: 'scale-subtle',
+ content: 'Press to reload LoRA Model and Prompt Template selections.',
+ });
+
+ document
+ .querySelector('#inference_preview_prompt_container .label-wrap')
+ .addEventListener('click', function () {
+ tippy('#inference_preview_prompt', {
+ placement: 'right',
+ delay: [500, 0],
+ animation: 'scale-subtle',
+ content: 'This is the prompt that will be sent to the language model.',
+ });
+
+ const update_btn = document.getElementById(
+ 'inference_update_prompt_preview_btn'
+ );
+ if (update_btn) update_btn.click();
+ });
+
+ function setTooltipForOptions() {
+ tippy('#inference_temperature', {
+ placement: 'right',
+ delay: [500, 0],
+ animation: 'scale-subtle',
+ content:
+ 'Controls randomness: Higher values (e.g., 1.0) make the model generate more diverse and random outputs. As the temperature approaches zero, the model will become deterministic and repetitive.0 will enable sampling.',
+ allowHTML: true,
+ });
+
+ tippy('#inference_top_p', {
+ placement: 'right',
+ delay: [500, 0],
+ animation: 'scale-subtle',
+ content:
+ 'Controls diversity via nucleus sampling: only the tokens whose cumulative probability exceeds top_p are considered. 0.5 means half of all likelihood-weighted options are considered.top_k tokens with the highest probabilities. This method can lead to more focused and coherent outputs by reducing the impact of low probability tokens.