Upload 322 files

.gitattributes

@@ -33,3 +33,39 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+documentation/docs/get-started/llm-studio-home-screen.png filter=lfs diff=lfs merge=lfs -text
+documentation/docs/guide/datasets/configure-dataset.png filter=lfs diff=lfs merge=lfs -text
+documentation/docs/guide/datasets/import-h2o-drive-dataset.png filter=lfs diff=lfs merge=lfs -text
+documentation/docs/guide/datasets/import-kaggle-dataset.png filter=lfs diff=lfs merge=lfs -text
+documentation/docs/guide/datasets/import-s3-bucket.png filter=lfs diff=lfs merge=lfs -text
+documentation/docs/guide/datasets/merge-datasets.png filter=lfs diff=lfs merge=lfs -text
+documentation/docs/guide/datasets/upload-dataset.png filter=lfs diff=lfs merge=lfs -text
+documentation/docs/guide/datasets/upload-local-file.png filter=lfs diff=lfs merge=lfs -text
+documentation/docs/guide/datasets/view-imported-dataset.png filter=lfs diff=lfs merge=lfs -text
+documentation/docs/guide/experiments/best-validation-sample.png filter=lfs diff=lfs merge=lfs -text
+documentation/docs/guide/experiments/charts-tab.png filter=lfs diff=lfs merge=lfs -text
+documentation/docs/guide/experiments/chat-tab.png filter=lfs diff=lfs merge=lfs -text
+documentation/docs/guide/experiments/compare-experiments.png filter=lfs diff=lfs merge=lfs -text
+documentation/docs/guide/experiments/delete-experiment.png filter=lfs diff=lfs merge=lfs -text
+documentation/docs/guide/experiments/export-model-to-huggingface.png filter=lfs diff=lfs merge=lfs -text
+documentation/docs/guide/experiments/local-llm-judge-logs.png filter=lfs diff=lfs merge=lfs -text
+documentation/docs/guide/experiments/run-experiment.png filter=lfs diff=lfs merge=lfs -text
+documentation/docs/guide/experiments/set-endpoint.png filter=lfs diff=lfs merge=lfs -text
+documentation/docs/guide/experiments/set-metric-model.png filter=lfs diff=lfs merge=lfs -text
+documentation/docs/guide/experiments/stop-experiment.png filter=lfs diff=lfs merge=lfs -text
+documentation/docs/guide/experiments/view-experiments.png filter=lfs diff=lfs merge=lfs -text
+documentation/docs/guide/experiments/worst-validation-sample.png filter=lfs diff=lfs merge=lfs -text
+documentation/docs/tutorials/tutorial-1a/configure-dataset.png filter=lfs diff=lfs merge=lfs -text
+documentation/docs/tutorials/tutorial-1a/evaluate-experiment.png filter=lfs diff=lfs merge=lfs -text
+documentation/docs/tutorials/tutorial-1a/import-dataset.png filter=lfs diff=lfs merge=lfs -text
+documentation/docs/tutorials/tutorial-2a/configure-dataset.png filter=lfs diff=lfs merge=lfs -text
+documentation/docs/tutorials/tutorial-2a/evaluate-experiment.png filter=lfs diff=lfs merge=lfs -text
+documentation/docs/tutorials/tutorial-2a/import-dataset.png filter=lfs diff=lfs merge=lfs -text
+static/screenshot-1.png filter=lfs diff=lfs merge=lfs -text
+static/screenshot-2.png filter=lfs diff=lfs merge=lfs -text
+static/screenshot-3.png filter=lfs diff=lfs merge=lfs -text
+static/screenshot-4.png filter=lfs diff=lfs merge=lfs -text
+static/screenshot-5.png filter=lfs diff=lfs merge=lfs -text
+static/screenshot-6.png filter=lfs diff=lfs merge=lfs -text
+static/screenshot-7.png filter=lfs diff=lfs merge=lfs -text
+static/screenshot-9.png filter=lfs diff=lfs merge=lfs -text

documentation/.gitignore

@@ -0,0 +1,17 @@
+node_modules
+tmp
+
+# Generated files 
+.docusaurus 
+.cach-loader
+
+# Misc
+.DS_Store
+.env.local
+.env.development.local
+.env.test.local
+.env.production.local
+
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*

documentation/README.md

@@ -0,0 +1,98 @@
+# H2O LLM Studio Documentation
+
+- The LLM Studio documentation is built using [Makersaurus](https://github.com/h2oai/makersaurus/pkgs/npm/makersaurus) which is a very thin wrapper around Facebook's Docusaurus.
+- The documentation is displayed at {{ https://docs.h2o.ai/h2o-llm-studio/ }}
+
+To view, edit, and cut a version of the documentation, the following is required:
+
+- Node.js version 16.14+ (you can check your version by running `node -v`). Use nvm to manage multiple Node versions installed on a single machine.
+
+- To install Node.js and npm with nvm in Mac or Ubuntu, run: `curl -o-
+https://raw.githubusercontent.com/creationix/nvm/v0.33.0/install.sh | bash` and `nvm install node`
+
+- Makersaurus (the H2O themed documentation site) is hosted on H2O's Github npm registry. npm must authenticate to the registry before you can download Makersaurus. Follow the 3 steps below to authenticate the npm package.
+
+ If you have already installed `@h2oai/ui-kit` or any other private `@h2oai`-prefixed npm package you can skip this step.
+
+ **Step 1:** Create a "classic" [personal access token](https://github.com/settings/tokens) (PAT) on Github. Note that you only need to enable the `read:packages` scope for this token. 
+
+ **Step 2:** Add the PAT to your `~/.npmrc` file. Create this file if it doesn't exist yet. 
+ ```
+ @h2oai:registry=https://npm.pkg.github.com/
+ //npm.pkg.github.com/:_authToken=YOUR-GENERATED-TOKEN
+ ```
+ **Step 3:** Verify that it worked by running the following command:
+ ```
+ npm whoami --registry=https://npm.pkg.github.com
+ ```
+ If this command returns your username, you can proceed to the next step. If you get an error, you are not yet authenticated. You might find the [Github registry docs](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-npm-registry#authenticating-with-a-personal-access-token) helpful for debugging.
+
+### Documentation structure
+
+
+```
+├── documentation
+│ ├── docs
+│ ├── tmp
+│ ├── makersaurus.config.js
+│ ├── sidebars.js
+│ ├── package.json
+│ ├── package-lock.json
+```
+
+- `documentation/docs`: Contains Markdown documentation files to edit the next documentation version.
+Customize the order of the docs sidebar in `sidebars.js`
+- `documentation/tmp`: Temporary files generated by Makersaurus. Do not edit these files.
+- `documentation/makersaurus.config.js`: Makersaurus [config file](https://h2oai.github.io/makersaurus/api/config)
+- `documentation/sidebars.js`: Sidebar configuration file
+- `documentation/package.json`: npm configuration file
+- `documentation/package-lock.json`: Generated by npm. Do not edit this file.
+
+
+### Edit locally
+
+To setup the local `env` to view and edit the next or past documentation versions ([first, ensure you install
+Node.js](#requirements)):
+
+1. Enter the documentation folder
+
+`cd documentation`
+
+2. Install dependencies
+
+`npm install`
+
+3. Start Makersaurus
+
+`npm start`
+
+- **Next documentation version**: To view your edits for the next documentation version, navigate to the provided URL.
+Then, select **Next** on the **Versions** dropdown menu.
+- **Debug**
+- If you don't see anything after clicking **Next**, run the following command and try again:
+`make setup-doc`
+- Ensure that the following variable is set to `true` in the `makersaurus.config.js` file (located at `docs`):
+`includeCurrentVersion`
+- **Past documentation versions**: To view your edits for past documentation versions (located at
+`docs/versioned_docs/`), navigate to the provided URL (for example, `http://localhost:3000/h2o-llm-studio/`). 
+Then, select a *version* (for example, v0.2.0) on the **Versions** dropdown menu.
+
+### Cut a version
+
+To cut a new version after making specific changes at `documentation/docs` to align with the next version of the application, consider the following instructions:
+
+1. Before a new version of the documentation is released, and right before we cut a version (`make version-doc`), change the following variable located in the `makersaurus.config.js` file to `false`: `includeCurrentVersion`
+2. Run: `make version-doc` (for example, `make version-doc DOC_VERSION=v0.3.0`)
+3. After the previous steps are executed and all generated files are pushed to the main branch, trigger the following
+script in GitHub actions: `deploy-to-github-pages.yml`
+4. After publishing the new documentation version, change the following variable located in the
+`makersaurus.config.js` file to `true`: `includeCurrentVersion`
+- This ensures the next doc version to edit will be visible while editing locally
+
+
+## More information
+
+Use the [Makersaurus docs](https://h2oai.github.io/makersaurus/) to learn more about how to edit docs, deploy the site, set up versioning and more.
+
+
+

documentation/docs/concepts.md

@@ -0,0 +1,69 @@
+---
+description: Learn about concepts around H2O LLM Studio.
+---
+# Concepts
+
+H2O LLM Studio is based on a few key concepts and uses several key terms across its documentation. Each, in turn, is explained within the sections below.
+
+## LLM
+
+A Large Language Model (LLM) is a type of AI model that uses deep learning techniques and uses massive datasets to analyze and generate human-like language. For example, many AI chatbots or AI search engines are powered by LLMs.  
+
+Generally speaking, LLMs can be characterized by the following parameters: 
+- size of the training dataset
+- cost of training (computational power)
+- size of the model (parameters)
+- performance after training (or how well the model is able to respond to a particular question)
+
+## Parameters and hyperparameters
+
+In the context of an LLM, parameters and hyperparameters are a crucial part of determinining the model's performance and overall behaviour. 
+
+- **Parameters:** The internal variables of the model that are learned during the training process. In the case of an LLM, parameters typically include the weights and biases associated with the neural network layers. The values of parameters directly influence the model's predictions and the quality of generated text.
+
+- **Hyperparameters:** The configuration choices that are set before training the model and are not learned directly from the data (e.g., number of epochs, batch size etc.). These choices impact the learning process and influence the model's overall behavior. Hyperparameters need to be tuned and optimized to achieve the best performance. H2O LLM Studio GUI shows tooltips next to each hyperparameter to explain what each hyperparameter is for. You can also see the following references for more details about hyperparameters in H2O LLM Studio.
+    - Dataset settings
+    - [Experiment settings](./guide/experiments/experiment-settings)
+
+
+## LLM Backbone
+
+LLM Backbone is a key hyperparamter that determines the model's architecture. This option is the most important setting when it comes to experiment creation, as it sets the pretrained model weights. For more information about LLM Backbone, see [Experiment settings](guide/experiments/experiment-settings.md#llm-backbone).
+
+
+## Generative AI
+
+Generative AI refers to AI models that can generate new content, such as images, videos, or text, that did not exist before. These models learn from large datasets and use this knowledge to create new content that is similar in style or content to the original dataset.
+
+
+## Foundation model
+
+A particular adaptive model that has been trained on a large amount of data and starts to derive relationships between words and concepts. Foundation models are fine-tuned to become more specific and adapt to the related domain more efficiently.
+
+## Fine-tuning
+
+Fine-tuning refers to the process of taking a pre-trained language model and further training it on a specific task or domain to improve its performance on that task. It is an important technique used to adapt LLMs to specific tasks and domains. 
+
+## LoRA (Low-Rank Adaptation)
+
+Low-Rank Adapation (LoRa) involves modifying the pre-trained model by adjusting its weights and biases to better fit the new task. This adaptation is done in a way that preserves the pre-trained weights from the original dataset while also adjusting for the new task's specific requirements. This method of training or fine-turning models consumes less memory. By using low rank adaptation, the pre-trained model can be quickly adapted to new tasks, without requiring a large amount of new training data.
+
+## Quantization
+
+Quantization is a technique used to reduce the size and memory requirements of a large language model without sacrificing its accuracy. This is done by converting the floating-point numbers used to represent the model's parameters to lower-precision numbers, such as half-floats or bfloat16. Quantization can be used to make language models more accessible to users with limited computing resources. 
+
+## 8-bit model training with a low memory footprint
+
+8-bit model training with a low memory footprint refers to a fine-tuning technique that reduces the memory requirements for training neural networks by using 8-bit integers instead of 32-bit floating-point numbers. This approach can significantly reduce the amount of memory needed to store the model's parameters and can make it possible to train larger models on hardware with limited memory capacity.
+
+### BLEU 
+
+Bilingual Evaluation Understudy (BLEU) is a model evaluation metric that is used to measure the quality of the predicted text against the input text. 
+
+BLEU: BLEU (Bilingual Evaluation Understudy) measures the quality of machine-generated texts by comparing them to reference texts by calculating a score between 0 and 1, where a higher score indicates a better match with the reference text.  BLEU is based on the concept of n-grams, which are contiguous sequences of words. The different variations of BLEU such as BLEU-1, BLEU-2, BLEU-3, and BLEU-4 differ in the size of the n-grams considered for evaluation.  BLEU-n measures the precision of n-grams (n consecutive words) in the generated text compared to the reference text. It calculates the precision score by counting the number of overlapping n-grams and dividing it by the total number of n-grams in the generated text.
+
+### Perplexity
+
+Perplexity (PPL) is a commonly used evaluation metric. It measures the confidence a model has in its predictions, or in simpler words how 'perplexed' or surprised it is by seeing new data. Perplexity is defined as the exponentiated cross-entropy of a sequence of tokens. Lower perplexity means the model is highly confident and accurate in the sequence of tokens it is responding with.
+
+

documentation/docs/faqs.md

@@ -0,0 +1,181 @@
+---
+description: Learn about frequently asked questions. 
+---
+import Icon from "@material-ui/core/Icon";
+
+# FAQs
+
+The sections below provide answers to frequently asked questions. If you have additional questions, please send them to [[email protected]](mailto:[email protected]).
+
+---
+
+### What are the general recommendations for using H2O LLM Studio?
+
+The recommendation is to always start with the default settings. From there, the parameters that tend to have the largest impact are: 
+- the LLM backbone
+- the number of epochs
+- the learning rate
+- the LoRA settings 
+
+:::info
+For more information on experiment settings, see [Experiment Settings](guide/experiments/experiment-settings). 
+:::
+
+The parameters that have the largest impact on the amount of GPU memory being used are the [backbone dtype](guide/experiments/experiment-settings#backbone-dtype) and the [max length](guide/experiments/experiment-settings#max-length) (the length of the input sequence being used during model training). 
+
+:::info
+For more information, see [this FAQ about GPU out-of-memory issues](#i-encounter-gpu-out-of-memory-issues-what-can-i-change-to-be-able-to-train-large-models). 
+:::
+
+While these parameters will change the behavior of the fine-tuned model, the change that will be most impactful is the actual data used for fine tuning. Having clean data and enough samples (i.e., atleast 1000 records) is imperative.
+
+---
+
+### Is the tool multi-user or single user? 
+
+While it is possible for multiple users to use the same instance, the tool was created for a single user at a time. 
+
+----
+
+### How can human feedback be applied in LLM Studio?
+
+In order to apply human feedback to H2O LLM Studio, there is a problem type called DPO (Direct Preference Optimization), which is specifically used for learning human feedback. For these types of use cases, there would be a selected answer and a rejected answer column to train a reward model. This is a more stable form of the traditional RLHF. For more information, see [this paper about DPO](https://arxiv.org/abs/2305.18290) by Stanford University.
+
+----
+
+### How does H2O LLM Studio evaluate the fine-tuned model? 
+
+The valuation options are [BLEU](concepts#bleu), [Perplexity](concepts#perplexity), and an AI Judge. For more information about the traditional NLP similarity metrics, see [BLEU](concepts#bleu) and [Perplexity](concepts#perplexity) explained on the concepts page. You can also opt to use an AI judge by having an LLM model (ChatGPT or a local LLM) judge the performance of the response. This [sample prompt](https://github.com/h2oai/h2o-llmstudio/blob/main/prompts/general.txt) is an example of a prompt that is used to have the LLM evaluate the response.
+
+----
+
+### Can I use a different AI Judge than ChatGPT? 
+
+Yes. For instructions on how to use a local LLM to evaluate the fine-tuned model, see [Evaluate model using an AI judge](guide/experiments/evaluate-model-using-llm). 
+
+---
+
+### How much data is generally required to fine-tune a model?
+
+There is no clear answer. As a rule of thumb, 1000 to 50000 samples of conversational data should be enough. Quality and diversity is very important. Make sure to try training on a subsample of data using the "sample" parameter to see how big the impact of the dataset size is. Recent studies suggest that less data is needed for larger foundation models.
+
+---
+
+### Are there any recommendations for which backbone to use? Are some backbones better for certain types of tasks?
+
+The majority of the LLM backbones are trained on a very similar corpus of data. The main difference is the size of the model and the number of parameters. Usually, the larger the model, the better they are. The larger models also take longer to train. It is recommended to start with the smallest model and then increase the size if the performance is not satisfactory. If you are looking to train for tasks that are not directly question answering in English, it is also a good idea to look for specialized LLM backbones.
+
+---
+
+### What if my data is not in question-and-answer form and I just have documents? How can I fine-tune the LLM model?
+
+To train a chatbot style model, you need to convert your data into a question and answer format.
+
+If you really want to continue pretraining on your own data without teaching a question-answering style, prepare a dataset with all your data in a single column Dataframe. Make sure that the length of the text in each row is not too long. In the experiment setup, remove all additional tokens (e.g. `<|prompt|>`, `<|answer|>`, for Text Prompt Start and Text Answer Start respectively) and disable **Add Eos Token To Prompt** and **Add Eos Token To Answer**. Deselect everything in the Prompt Column.
+
+There are also other enterprise solutions from H2O.ai that may help you convert your data into a Q&A format. For more information, see [H2O.ai's Generative AI page](https://h2o.ai/) and this blogpost about [H2O LLM DataStudio: Streamlining Data Curation and Data Preparation for LLMs related tasks](https://h2o.ai/blog/2023/streamlining-data-preparation-for-fine-tuning-of-large-language-models/).
+
+---
+
+
+### Can the adapter be downloaded after fine-tuning so that the adapter can be combined with the backbone LLM for deployment?
+
+H2O LLM Studio provides the option to download only the LoRA adapter when a model was trained with LoRA. Once the experiment has finished running, click the **Download adapter** button to download the lora adapter_weights separately from a fine-tuned model. 
+
+---
+
+###  I encounter GPU out-of-memory issues. What can I change to be able to train large models?
+
+There are various parameters that can be tuned while keeping a specific LLM backbone fixed. It is advised to choose 4bit/8bit precision as a backbone dtype to be able to train models >=7B on a consumer type GPU. [LORA](concepts#lora-low-rank-adaptation) should be enabled. Besides that there are the usual parameters such as batch size and maximum sequence length that can be decreased to save GPU memory (please ensure that your prompt+answer text is not truncated too much by checking the train data insights).
+
+---
+
+### When does the model stop the fine-tuning process?
+
+The number of epochs are set by the user.
+
+---
+
+### What is the maximum dataset size that an LLM Studio instance can handle?
+
+The total dataset size is basically unlimited / only bound by disk space as all training is done in batches. There is no specific rule of thumb for maximum batch size - this depends strongly on backbone, context size, use of flash attention 2.0, use of gradient checkpointing, etc.
+We suggest using a batch size that just fills the RAM for maximum efficiency. While testing for maximum memory consumption, set padding quantile to `0`. Make sure to set it back to `1` when you have found a good setting for the batch size to save on runtime.
+
+----
+
+### Where does H2O LLM Studio store its data?
+
+By default, H2O LLM Studio stores its data in two folders located in the root directory in the app. The folders are named `data` and `output`. Here is the breakdown of the data storage structure:
+- `data/dbs`: This folder contains the user database used within the app.
+- `data/user`: This folder is where uploaded datasets from the user are stored.
+- `output/user`: All experiments conducted in H2O LLM Studio are stored in this folder. For each experiment, a separate folder is created within the `output/user` directory, which contains all the relevant data associated with that particular experiment.
+- `output/download`: Utility folder that is used to store data the user downloads within the app. 
+
+It is possible to change the default working directory of H2O LLM Studio by setting the `H2O_LLM_STUDIO_WORKDIR` environment variable. By default, the working directory is set to the root directory of the app.
+
+----
+
+### How can I update H2O LLM Studio?
+
+To update H2O LLM Studio, you have two options:
+
+1. Using the latest main branch: Execute the commands `git checkout main` and `git pull` to obtain the latest updates from the main branch.
+2. Using the latest release tag: Execute the commands `git pull` and `git checkout v0.0.3` (replace 'v0.0.3' with the desired version number) to switch to the latest release branch.
+
+The update process does not remove or erase any existing data folders or experiment records. This means that all your old data, including the user database, uploaded datasets, and experiment results, will still be available to you within the updated version of H2O LLM Studio.
+
+Before updating, it is recommended to run the `git rev-parse --short HEAD` command and save the commit hash. 
+This will allow you to revert to your existing version if needed. 
+
+---
+
+### Once I have the [LoRA](guide/experiments/experiment-settings.md#lora), what is the recommended way of utilizing it with the base model?
+
+You can also export the LoRA weights. You may add them to the files to be exported [here](https://github.com/h2oai/h2o-llmstudio/blob/main/llm_studio/app_utils/sections/experiment.py#L1552). Before exporting, the LoRA weights are merged back into the original LLM backbone weights to make downstream tasks easier. You do not need to have PEFT, or anything else for your deployment.
+
+---
+
+### How to use H2O LLM Studio in Windows? 
+
+Use WSL 2 on Windows 
+
+---
+
+### How can I easily fine-tune a large language model (LLM) using the command-line interface (CLI) of H2O LLM Studio when I have limited GPU memory?
+
+If you have limited GPU memory but still want to fine-tune a large language model using H2O LLM Studio's CLI, there are alternative methods you can use to get started quickly.
+
+- [Using Kaggle kernels](https://www.kaggle.com/code/ilu000/h2o-llm-studio-cli/) 
+- [Using Google Colab](https://colab.research.google.com/drive/1soqfJjwDJwjjH-VzZYO_pUeLx5xY4N1K?usp=sharing)
+
+---
+
+### Can I run a validation metric on a model post-training, optionally on a different validation dataset?
+
+Yes.
+
+1. After you have finished creating an experiment, click on the <Icon>more_vert</Icon> Kebab menu of the relevant experiment and select **New Experiment**. 
+
+2. Enable the **Use previous experiments weight** setting found at the top of the screen. 
+   This will now load the previous weights, and you can now change eval dataset, metric, and anything else as you see fit. To only do evaluation without any retraining, set the **Epochs** to 0.
+
+----
+
+### What are the hardware/infrastructure sizing recommendations for H2O LLM Studio?
+
+When it comes to hardware requirements, it is important to note that the primary demand centers around the GPU and its associated VRAM. In terms of CPUs, most modern choices should suffice as NLP tasks typically do not heavily stress CPU performance. As for RAM, it's advisable to have a minimum of 128GB, with a stronger recommendation of 256GB or more, particularly when dealing with substantial model weights that must be accommodated in the CPU RAM.
+
+----
+
+### I am seeing an OS error during the H2O LLM Studio training session. What should I do? 
+
+If you recieve the following error, it is most likely because of network issues either with your own connection or on the Hugging Face Hub side. 
+
+```title="Error"
+OSError: Consistency check failed: file should be of size 4999819336 but has size 
+14099570832 ((…)ve/main/ model-00002-of-00003.safetensors). 
+```
+
+In most cases, rerunning the experiment will solve it as the download of the model weights will be re-initiated.
+
+---

documentation/docs/get-started/core-features.md

@@ -0,0 +1,34 @@
+---
+description: Learn about the core features of LLM Studio.
+---
+# Core features
+
+## No-code fine-tuning
+
+NLP practitioners can easily fine-tune models without the need for code expertise. The user interface, which is specifically designed for LLMs, allows users to upload large datasets easily and configure [hyperparameters](../concepts#parameters-and-hyperparameters) to fine-tune the model. 
+
+## Highly customizable (wide range of hyperparameters)
+
+H2O LLM Studio supports a wide variety of hyperparameters that can be used to fine-tune the model and supports the following fine-tuning techniques to enable advanced customization:
+
+- [Low-Rank Adaptation (LoRA)](../concepts#lora-low-rank-adaptation)
+- [8-bit model training with a low memory footprint](../concepts#8-bit-model-training-with-a-low-memory-footprint)
+
+## Advanced evaluation metrics and experiment comparison
+
+Advanced evaluation metrics in H2O LLM Studio can be used to validate the answers generated by the LLM. This helps to make data-driven decisions about the model. It also offers visual tracking and comparison of experiment performance, making it easy to analyze and compare different fine-tuned models.You can also visualize how different parameters affect the model performance, and optionally use the [Neptune](https://neptune.ai/) or [W&B](https://wandb.ai/) integration to track and log your experiments. 
+
+## Instant publishing models
+
+H2O LLM Studio enables easy model sharing with the community by allowing you to export the model to the [Hugging Face Hub](https://huggingface.co/h2oai) with a single click. 
+
+## Instant feedback on model performance
+
+Additionally, H2O LLM Studio lets you chat with the fine-tuned model and receive instant feedback about model performance.
+
+
+
+
+
+
+

documentation/docs/get-started/llm-studio-flow.md

@@ -0,0 +1,46 @@
+---
+description: The flow of creating and fine-tuning large language models using H2O LLM Studio.
+---
+# Model flow
+
+The flow of creating and fine-tuning large language models using H2O LLM Studio can be summarized in the following sequential steps:
+
+- [Step 1: Import a dataset](#step-1-import-a-dataset)
+- [Step 2: Create an experiment](#step-2-create-an-experiment)
+- [Step 3: Monitor an experiment](#step-3-monitor-an-experiment)
+- [Step 4: Compare experiments](#step-4-compare-experiments)
+- [Step 5: Export a model to Hugging Face Hub](#step-5-export-a-model-to-hugging-face-hub)
+
+## Step 1: Import a dataset
+
+As the first step in the experiment flow, prep your data and import your dataset to H2O LLM Studio. 
+
+- To learn about supported data connectors and data format, see [Supported data connectors and format](../guide/datasets/data-connectors-format).
+- To learn about how to import a dataset to H2O LLM Studio, see [Import a dataset](../guide/datasets/import-dataset).
+- To learn about reviewing and editing a dataset, see [View and manage dataset](../guide/datasets/view-dataset.md).
+
+## Step 2: Create an experiment
+
+As the second step in the experiment flow, create an experiment using the imported dataset. H2O LLM Studio offers several hyperparameter settings that you can adjust for your experiment model. To ensure that your training process is effective, you may need to specify the [hyperparameters](../concepts#parameters-and-hyperparameters) like learning rate, batch size, and the number of epochs. H2O LLM Studio provides an overview of all the parameters you’ll need to specify for your experiment.
+
+- To learn about creating a new experiment, see [Create an experiment](../guide/experiments/create-an-experiment.md).
+- To learn about the settings available for creating an experiment, see [Experiment settings](../guide/experiments/experiment-settings.md).
+
+## Step 3: Monitor an experiment
+
+As the third step in the experiment flow, monitor the launched experiment. H2O LLM Studio allows you to inspect your experiment (model) during and after model training. Simple interactive graphs in H2O LLM Studio allow you to understand the impact of selected hyperparameter values during and after model training. You can then adjust the [hyperparameters](../concepts#parameters-and-hyperparameters) to further optimize model performance. 
+
+To learn about viewing and monitoring an experiment, see [View and manage experiments](../guide/experiments/view-an-experiment.md).
+
+## Step 4: Compare experiments
+
+The H2O LLM studio provides a useful feature  that allows comparing various experiments and analyzing how different model parameters affect model performance. This feature is a powerful tool for fine-tuning your machine-learning models and ensuring they meet your desired performance metrics.
+
+To learn about comparing multiple experiments, see [Compare experiments](../guide/experiments/compare-experiments.md).
+
+## Step 5: Export a model to Hugging Face Hub
+
+As the final step in the experiment flow, you can export the fine-tuned model to Hugging Face with a single click.
+
+To learn about exporting a trained model to Hugging Face Hub, see, [Export trained model to Hugging Face](../guide/experiments/export-trained-model.md).
+

documentation/docs/get-started/llm-studio-performance.md

@@ -0,0 +1,166 @@
+---
+description: Setting up and running H2O LLM Studio requires the following minimal prerequisites. This page lists out the speed and performance metrics of H2O LLM Studio based on different hardware setups.
+---
+# H2O LLM Studio performance
+
+Setting up and running H2O LLM Studio requires the following minimal [prerequisites](set-up-llm-studio.md#prerequisites). This page lists out the speed and performance metrics of H2O LLM Studio based on different hardware setups.
+
+The following metrics were measured. 
+
+- **Hardware setup:** The type and number of computing devices used to train the model.
+- **LLM backbone:** The underlying architecture of the language model. For more information, see [LLM backbone](concepts.md#llm-backbone).
+- **Quantization:** A technique used to reduce the size and memory requirements of the model. For more information, see [Quantization](concepts.md#quantization).
+- **Train**: The amount of time it took to train the model in hours and minutes.
+- **Validation:** The amount of time it took to validate the mode in hours and minutes. 
+
+| Hardware setup | LLM backbone | Quantization | Train (hh:mm:ss)| Validation (hh:mm:ss) |
+|---|---|---|---|---|
+| 8xA10G | h2oai/h2ogpt-4096-llama2-7b | bfloat16 | 11:35 | 3:32 |
+| 4xA10G | h2oai/h2ogpt-4096-llama2-7b | bfloat16 | 21:13 | 06:35 |
+| 2xA10G | h2oai/h2ogpt-4096-llama2-7b | bfloat16 | 37:04 | 12:21 |
+| 1xA10G | h2oai/h2ogpt-4096-llama2-7b | bfloat16 | 1:25:29 | 15:50 |
+| 8xA10G | h2oai/h2ogpt-4096-llama2-7b | nf4 | 14:26 | 06:13 |
+| 4xA10G | h2oai/h2ogpt-4096-llama2-7b | nf4 | 26:55 | 11:59 |
+| 2xA10G | h2oai/h2ogpt-4096-llama2-7b | nf4 | 48:24 | 23:37 |
+| 1xA10G | h2oai/h2ogpt-4096-llama2-7b | nf4 | 1:26:59 | 42:17 |
+| 8xA10G | h2oai/h2ogpt-4096-llama2-13b | bfloat16 | OOM | OOM |
+| 4xA10G | h2oai/h2ogpt-4096-llama2-13b | bfloat16 | OOM | OOM |
+| 2xA10G | h2oai/h2ogpt-4096-llama2-13b | bfloat16 | OOM | OOM |
+| 1xA10G | h2oai/h2ogpt-4096-llama2-13b | bfloat16 | OOM | OOM |
+| 8xA10G | h2oai/h2ogpt-4096-llama2-13b | nf4 | 25:07 | 10:58 |
+| 4xA10G | h2oai/h2ogpt-4096-llama2-13b | nf4 | 48:43 | 21:25 |
+| 2xA10G | h2oai/h2ogpt-4096-llama2-13b | nf4 | 1:30:45 | 42:06 |
+| 1xA10G | h2oai/h2ogpt-4096-llama2-13b | nf4 | 2:44:36 | 1:14:20 |
+| 8xA10G | h2oai/h2ogpt-4096-llama2-70b | nf4 | OOM | OOM |
+| 4xA10G | h2oai/h2ogpt-4096-llama2-70b | nf4 | OOM | OOM |
+| 2xA10G | h2oai/h2ogpt-4096-llama2-70b | nf4 | OOM | OOM |
+| 1xA10G | h2oai/h2ogpt-4096-llama2-70b | nf4 | OOM | OOM |
+|---|---|---|---|---|
+| 4xA100 80GB | h2oai/h2ogpt-4096-llama2-7b | bfloat16 | 7:04 | 3:55 |
+| 2xA100 80GB | h2oai/h2ogpt-4096-llama2-7b | bfloat16 | 13:14 | 7:23 |
+| 1xA100 80GB | h2oai/h2ogpt-4096-llama2-7b | bfloat16 | 23:36 | 13:25 |
+| 4xA100 80GB | h2oai/h2ogpt-4096-llama2-7b | nf4 | 9:44 | 6:30 |
+| 2xA100 80GB | h2oai/h2ogpt-4096-llama2-7b | nf4 | 18:34 | 12:16 |
+| 1xA100 80GB | h2oai/h2ogpt-4096-llama2-7b | nf4 | 34:06 | 21:51 |
+| 4xA100 80GB | h2oai/h2ogpt-4096-llama2-13b | bfloat16 | 11:46 | 5:56 |
+| 2xA100 80GB | h2oai/h2ogpt-4096-llama2-13b | bfloat16 | 21:54 | 11:17 |
+| 1xA100 80GB | h2oai/h2ogpt-4096-llama2-13b | bfloat16 | 39:10 | 18:55 |
+| 4xA100 80GB | h2oai/h2ogpt-4096-llama2-13b | nf4 | 16:51 | 10:35 |
+| 2xA100 80GB | h2oai/h2ogpt-4096-llama2-13b | nf4 | 32:05 | 21:00 |
+| 1xA100 80GB | h2oai/h2ogpt-4096-llama2-13b | nf4 | 59:11 | 36:53 |
+| 4xA100 80GB | h2oai/h2ogpt-4096-llama2-70b | nf4 | 1:13:33 | 46:02 |
+| 2xA100 80GB | h2oai/h2ogpt-4096-llama2-70b | nf4 | 2:20:44 | 1:33:42 |
+| 1xA100 80GB | h2oai/h2ogpt-4096-llama2-70b | nf4 | 4:23:57 | 2:44:51 |
+
+:::info
+The runtimes were gathered using the default parameters. 
+
+<details>
+<summary>Expand to see the default parameters </summary>
+
+```
+architecture:
+    backbone_dtype: int4
+    gradient_checkpointing: true
+    intermediate_dropout: 0.0
+    pretrained: true
+    pretrained_weights: ''
+augmentation:
+    random_parent_probability: 0.0
+    skip_parent_probability: 0.0
+    token_mask_probability: 0.0
+dataset:
+    add_eos_token_to_answer: true
+    add_eos_token_to_prompt: true
+    add_eos_token_to_system: true
+    answer_column: output
+    chatbot_author: H2O.ai
+    chatbot_name: h2oGPT
+    data_sample: 1.0
+    data_sample_choice:
+    - Train
+    - Validation
+    limit_chained_samples: false
+    mask_prompt_labels: true
+    parent_id_column: None
+    personalize: false
+    prompt_column:
+    - instruction
+    system_column: None
+    text_answer_separator: <|answer|>
+    text_prompt_start: <|prompt|>
+    text_system_start: <|system|>
+    train_dataframe: /data/user/oasst/train_full.pq
+    validation_dataframe: None
+    validation_size: 0.01
+    validation_strategy: automatic
+environment:
+    compile_model: false
+    find_unused_parameters: false
+    gpus:
+    - '0'
+    - '1'
+    - '2'
+    - '3'
+    - '4'
+    - '5'
+    - '6'
+    - '7'
+    huggingface_branch: main
+    mixed_precision: true
+    number_of_workers: 8
+    seed: -1
+    trust_remote_code: true
+experiment_name: default-8-a10g
+llm_backbone: h2oai/h2ogpt-4096-llama2-7b
+logging:
+    logger: None
+    neptune_project: ''
+output_directory: /output/...
+prediction:
+    batch_size_inference: 0
+    do_sample: false
+    max_length_inference: 256
+    metric: BLEU
+    metric_gpt_model: gpt-3.5-turbo-0301
+    metric_gpt_template: general
+    min_length_inference: 2
+    num_beams: 1
+    num_history: 4
+    repetition_penalty: 1.2
+    stop_tokens: ''
+    temperature: 0.0
+    top_k: 0
+    top_p: 1.0
+problem_type: text_causal_language_modeling
+tokenizer:
+    add_prompt_answer_tokens: false
+    max_length: 512
+    padding_quantile: 1.0
+training:
+    batch_size: 2
+    differential_learning_rate: 1.0e-05
+    differential_learning_rate_layers: []
+    drop_last_batch: true
+    epochs: 1
+    evaluate_before_training: false
+    evaluation_epochs: 1.0
+    grad_accumulation: 1
+    gradient_clip: 0.0
+    learning_rate: 0.0001
+    lora: true
+    use_dora: false
+    lora_alpha: 16
+    lora_dropout: 0.05
+    lora_r: 4
+    lora_target_modules: ''
+    loss_function: TokenAveragedCrossEntropy
+    optimizer: AdamW
+    save_checkpoint: "last"
+    schedule: Cosine
+    train_validation_data: false
+    warmup_epochs: 0.0
+    weight_decay: 0.0
+```
+</details>
+:::

documentation/docs/get-started/set-up-llm-studio.md

@@ -0,0 +1,328 @@
+---
+description: Learn how to set up LLM Studio.
+---
+import Tabs from "@theme/Tabs";
+import TabItem from "@theme/TabItem";
+
+# Set up H2O LLM Studio
+
+This page guides you through setting up and installing H2O LLM Studio on your local system. 
+
+First, download the H2O LLM Studio package from the [H2O LLM Studio Github repository](https://github.com/h2oai/h2o-llmstudio). You can use `git clone` or navigate to the [releases page](https://github.com/h2oai/h2o-llmstudio/releases) and download the `.zip` file found within the **Assets** of the relevant release. 
+
+## Prerequisites
+
+H2O LLM Studio requires the following minimum requirements:
+
+- A machine with Ubuntu 16.04+ with atleast one recent Nvidia GPU
+- Have at least 128GB+ of system RAM. Larger models and complex tasks may require 256GB+ or more.
+- Nvidia drivers v470.57.02 or a later version
+- Access to the following URLs:
+  - developer.download.nvidia.com
+  - pypi.org
+  - huggingface.co
+  - download.pytorch.org
+  - cdn-lfs.huggingface.co
+
+:::info Notes
+- Atleast 24GB of GPU memory is recommended for larger models.
+- For more information on performance benchmarks based on the hardware setup, see [H2O LLM Studio performance](llm-studio-performance.md).
+- The required URLs are accessible by default when you start a GCP instance, however, if you have network rules or custom firewalls in place, it is recommended to confirm that the URLs are accessible before running `make setup`.
+:::
+
+## Installation
+
+:::note Installation methods
+
+<Tabs className="unique-tabs">
+  <TabItem
+    value="recommended-install"
+    label="Linux/Ubuntu installation (recommended)"
+    default
+  >
+    <p>
+      The recommended way to install H2O LLM Studio is using pipenv with Python
+      3.10. To install Python 3.10 on Ubuntu 16.04+, execute the following
+      commands.
+    </p>
+    <p>
+      <b>System installs (Python 3.10)</b>
+    </p>
+    <pre>
+      <code>
+        sudo add-apt-repository ppa:deadsnakes/ppa <br></br>
+        sudo apt install python3.10 <br></br>
+        sudo apt-get install python3.10-distutils <br></br>
+        curl -sS https://bootstrap.pypa.io/get-pip.py | python3.10
+      </code>
+    </pre>
+    <p>
+      <b>Install NVIDIA drivers (if required)</b>
+      <br></br>
+      If you are deploying on a 'bare metal' machine running Ubuntu, you may need
+      to install the required Nvidia drivers and CUDA. The following commands show
+      how to retrieve the latest drivers for a machine running Ubuntu 20.04 as an
+      example. You can update the following based on your respective operating system.
+    </p>
+    <pre>
+      <code>
+        wget
+        https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin{" "}
+        <br></br>
+        sudo mv cuda-ubuntu2004.pin
+        /etc/apt/preferences.d/cuda-repository-pin-600 <br></br>
+        wget
+        https://developer.download.nvidia.com/compute/cuda/11.4.3/local_installers/cuda-repo-ubuntu2004-11-4-local_11.4.3-470.82.01-1_amd64.deb{" "}
+        <br></br>
+        sudo dpkg -i
+        cuda-repo-ubuntu2004-11-4-local_11.4.3-470.82.01-1_amd64.deb <br></br>
+        sudo apt-key add /var/cuda-repo-ubuntu2004-11-4-local/7fa2af80.pub <br></br>
+        sudo apt-get -y update <br></br>
+        sudo apt-get -y install cuda
+      </code>
+    </pre>
+    <p>
+      <b>Create virtual environment (pipenv) </b>
+      <br></br>
+      The following command creates a virtual environment using pipenv and will install
+      the dependencies using pipenv.
+      <pre>
+        <code>make setup</code>
+      </pre>
+    </p>
+  </TabItem>
+  <TabItem value="using-requirements" label="Using requirements.txt">
+    <p>
+      If you wish to use conda or another virtual environment, you can also
+      install the dependencies using the <code>requirements.txt</code>{" "}
+      file.{" "}
+    </p>
+    <pre>
+      <code>pip install -r requirements.txt</code>
+    </pre>
+  </TabItem>
+  <TabItem value="wsl2-install" label="Windows installation" default>
+    <p>
+      Follow the steps below to install H2O LLM Studio on a Windows machine
+      using Windows Subsystem for Linux{" "}
+      <a href="https://learn.microsoft.com/en-us/windows/wsl/">WSL2</a>
+    </p>
+    <p>
+      1. Download the{" "}
+      <a href="https://www.nvidia.com/download/index.aspx">
+        latest nvidia driver
+      </a>{" "}
+      for Windows.{" "}
+    </p>
+    <p>
+      2. Open PowerShell or a Windows Command Prompt window in administrator
+      mode.{" "}
+    </p>
+    <p>
+      3. Run the following command to confirm that the driver is installed
+      properly and see the driver version.
+      <pre>
+        <code>nvidia-smi</code>
+      </pre>
+    </p>
+    <p>
+      4. Run the following command to install WSL2.
+      <pre>
+        <code>wsl --install</code>
+      </pre>
+    </p>
+    <p>5. Launch the WSL2 Ubuntu installation. </p>
+    <p>
+      6. Install the{" "}
+      <a href="https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=WSL-Ubuntu&target_version=2.0">
+        WSL2 Nvidia Cuda Drivers
+      </a>
+      .
+      <pre>
+        <code>
+          wget
+          https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/cuda-wsl-ubuntu.pin{" "}
+          <br></br>
+          sudo mv cuda-ubuntu2004.pin
+          /etc/apt/preferences.d/cuda-repository-pin-600 <br></br>
+          wget
+          https://developer.download.nvidia.com/compute/cuda/12.2.0/local_installers/cuda-repo-wsl-ubuntu-12-2-local_12.2.0-1_amd64.deb{" "}
+          <br></br>
+          sudo dpkg -i cuda-repo-wsl-ubuntu-12-2-local_12.2.0-1_amd64.deb <br></br>
+          sudo cp /var/cuda-repo-wsl-ubuntu-12-2-local/cuda-*-keyring.gpg
+          /usr/share/keyrings/ <br></br>
+          sudo apt-get update <br></br>
+          sudo apt-get -y install cuda
+        </code>
+      </pre>
+    </p>
+    <p>
+      7. Set up the required python system installs (Python 3.10).
+      <pre>
+        <code>
+          sudo add-apt-repository ppa:deadsnakes/ppa <br></br>
+          sudo apt install python3.10 <br></br>
+          sudo apt-get install python3.10-distutils <br></br>
+          curl -sS https://bootstrap.pypa.io/get-pip.py | python3.10
+        </code>
+      </pre>
+    </p>
+    <p>
+      8. Create the virtual environment.
+      <pre>
+        <code>
+          sudo apt install -y python3.10-venv<br></br>
+          python3 -m venv llmstudio<br></br>
+          source llmstudio/bin/activate<br></br>
+        </code>
+      </pre>
+    </p>
+    <p>
+      9.Clone the H2O LLM Studio repository locally.
+      <pre>
+        <code>
+          git clone https://github.com/h2oai/h2o-llmstudio.git<br></br>
+          cd h2o-llmstudio
+        </code>
+      </pre>
+    </p>
+    <p>
+      10. Install H2O LLM Studio using the `requirements.txt`.
+      <pre>
+        <code>pip install -r requirements.txt</code>
+      </pre>
+    </p>
+    <p>
+      11. Run the H2O LLM Studio application.
+      <pre>
+        <code>
+          H2O_WAVE_MAX_REQUEST_SIZE=25MB \ <br></br>
+          H2O_WAVE_NO_LOG=True \ <br></br>
+          H2O_WAVE_PRIVATE_DIR="/download/@output/download" \ <br></br>
+          wave run llm_studio.app
+        </code>
+      </pre>
+    </p>
+    <p>
+      This will start the H2O Wave server and the H2O LLM Studio app. Navigate
+      to <a>http://localhost:10101/</a> (we recommend using Chrome) to access
+      H2O LLM Studio and start fine-tuning your models.
+    </p>
+  </TabItem>
+</Tabs>
+:::
+
+## Install custom package
+
+If required, you can install additional Python packages into your environment. This can be done using pip after activating your virtual environment via `make shell`. For example, to install flash-attention, you would use the following commands:
+
+```bash
+make shell
+pip install flash-attn --no-build-isolation
+pip install git+https://github.com/HazyResearch/flash-attention.git#subdirectory=csrc/rotary
+```
+
+Alternatively, you can also directly install the custom package by running the following command.
+
+```bash
+pipenv install package_name
+```
+
+## Run H2O LLM Studio
+
+There are several ways to run H2O LLM Studio depending on your requirements.
+
+1. [Run H2O LLM Studio GUI](#run-h2o-llm-studio-gui)
+2. [Run using Docker](#run-using-docker)
+3. [Run by building your own Docker image](#run-by-building-your-own-docker-image)
+4. [Run with the CLI (command-line interface)](#run-with-command-line-interface-cli)
+
+### Run H2O LLM Studio GUI
+
+Run the following command to start the H2O LLM Studio.
+
+```sh
+make llmstudio
+```
+
+This will start the H2O Wave server and the H2O LLM Studio app. Navigate to [http://localhost:10101/](http://localhost:10101/) (we recommend using Chrome) to access H2O LLM Studio and start fine-tuning your models.
+
+![home-screen](llm-studio-home-screen.png)
+
+If you are running H2O LLM Studio with a custom environment other than Pipenv, start the app as follows:
+
+```sh
+H2O_WAVE_MAX_REQUEST_SIZE=25MB \
+H2O_WAVE_NO_LOG=True \
+H2O_WAVE_PRIVATE_DIR="/download/@output/download" \
+wave run llm_studio.app
+```
+
+### Run using Docker
+
+First, install Docker by following the instructions from the [NVIDIA Container Installation Guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#docker). H2O LLM Studio images are stored in the `h2oai dockerhub` container repository.
+
+```sh
+mkdir -p `pwd`/llmstudio_mnt
+docker run \
+    --runtime=nvidia \
+    --shm-size=64g \
+    --init \
+    --rm \
+    -it \
+    -p 10101:10101 \
+    -v `pwd`/llmstudio_mnt:/home/llmstudio/mount \
+    -v ~/.cache:/home/llmstudio/.cache \
+    h2oairelease/h2oai-llmstudio-app:latest
+```
+
+Navigate to [http://localhost:10101/](http://localhost:10101/) (we recommend using Chrome) to access H2O LLM Studio and start fine-tuning your models.
+
+:::info
+Other helpful docker commands are `docker ps` and `docker kill`.
+:::
+
+### Run by building your own Docker image
+
+```sh
+docker build -t h2o-llmstudio .
+mkdir -p `pwd`/llmstudio_mnt
+docker run \
+    --runtime=nvidia \
+    --shm-size=64g \
+    --init \
+    --rm \
+    -it \
+    -p 10101:10101 \
+    -v `pwd`/llmstudio_mnt:/home/llmstudio/mount \
+    -v ~/.cache:/home/llmstudio/.cache \
+    h2o-llmstudio
+```
+
+### Run with command line interface (CLI)
+
+You can also use H2O LLM Studio with the command line interface (CLI) and specify the configuration .yaml file that contains all the experiment parameters. To finetune using H2O LLM Studio with CLI, activate the pipenv environment by running `make shell`.
+
+To specify the path to the configuration file that contains the experiment parameters, run:
+
+```sh
+python llm_studio/train.py -Y {path_to_config_yaml_file}
+```
+
+To run on multiple GPUs in DDP mode, run:
+
+```sh
+bash distributed_train.sh {NR_OF_GPUS} -Y {path_to_config_yaml_file}
+```
+
+:::info
+By default, the framework will run on the first `k` GPUs. If you want to specify specific GPUs to run on, use the `CUDA_VISIBLE_DEVICES` environment variable before the command.
+:::
+
+To start an interactive chat with your trained model, run:
+
+```sh
+python llm_studio/prompt.py -e {experiment_name}
+```
+
+`experiment_name` is the output folder of the experiment you want to chat with. The interactive chat will also work with models that were fine-tuned using the GUI.

documentation/docs/get-started/videos.md

@@ -0,0 +1,49 @@
+---
+description: Learn from a collection of videos about LLM Studio.
+---
+import ReactPlayer from 'react-player'
+
+
+# Videos
+
+## Discovering the Potential of LLMs
+
+<iframe width="930" height="515" src="https://www.youtube.com/embed/u48QaIAIFw4" title="Discovering the Potential of LLMs: A Journey through H2O.ai's LLM Studio!" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
+
+
+:::info Note 
+  In this video, Andreea Turcu delves in-depth into the world of language models, showcasing how users can use H2O.ai's LLM Studio to their full advantage.
+:::
+
+---
+
+## The Fine Art of Fine-Tuning Large Language Models
+
+<iframe width="930" height="515" src="https://www.youtube.com/embed/YWAS3QDFg40" title="The Fine Art of Fine-Tuning LLMs" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
+
+
+:::info Note 
+  In this video, Pascal Pfeiffer, Principal Data Scientist at H2O.ai and Kaggle Grandmaster, announces the release of H2O LLM Studio and talks about fine-tuning LLMs using H2O LLM Studio at H2O World India 2023. 
+:::
+
+---
+
+## Basic introduction to H2O LLM Studio 
+
+<iframe width="930" height="515" src="https://www.youtube.com/embed/aFU3VRGE2gk" title="Introduction to H2O LLM Studio" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
+
+
+:::info Note 
+  In this video, Avkash Chauhan, founder of Prodramp Inc, gives a basic introduction about H2O LLM Studio. 
+:::
+
+----
+
+## LLM Fine-Tuning, Falcon 40b, and the State of Open-Source
+
+<iframe width="930" height="515" src="https://www.youtube.com/embed/Ur-1PI9SMfw" title="Pascal Pfeiffer - Kaggle, Fine-Tuning, H2O.ai, GPT4, Falcon 40b, Open Source" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
+
+
+:::info Note 
+  In this video, Pascal Pfeiffer, the Principal Data Scientist at h2o.ai is interviewed about LLM fine-tuning, being a Kaggle Grandmaster, H2O.ai, Falcon 40b, the state of open-source, and more. 
+:::

documentation/docs/get-started/what-is-h2o-llm-studio.md

@@ -0,0 +1,16 @@
+---
+description: H2O LLM Studio is an open-source, no-code LLM graphical user interface (GUI) designed for fine-tuning state-of-the-art large language models. 
+---
+# What is H2O LLM Studio?
+
+H2O LLM Studio is an open-source, no-code [LLM](../concepts#llm) graphical user interface (GUI) designed for fine-tuning state-of-the-art large language models. 
+
+[Fine-tuning](../concepts#fine-tuning) a pretrained language model requires coding expertise and extensive knowledge about the model and its [hyperparameters](../concepts#parameters-and-hyperparameters), however  H2O LLM Studio enables NLP practitioners to fine-tune their LLMs easily with no need for coding and better flexibility over customization. 
+
+H2O LLM Studio also lets you chat with the fine-tuned model and receive instant feedback about model performance.
+
+## Who is H2O LLM Studio for?
+
+H2O LLM Studio is a free and open-source tool that is designed for anyone who wants to fine-tune their own language models. It is designed to be easy to use and accessible to everyone regardless of their technical expertise.
+
+NLP practitioners and data scientists in particular may find it useful to easily and effectively create and fine-tune large language models. 

documentation/docs/guide/datasets/data-connectors-format.md

@@ -0,0 +1,35 @@
+# Supported data connectors and format
+
+## Data connectors
+
+H2O LLM Studio supports the following data connectors to access or upload external data sources.
+
+- **Upload**: Upload a local dataset from your machine. 
+- **Local**: Specify the file location of the dataset on your machine. 
+- **AWS S3 (Amazon AWS S3)**: Connect to an Amazon AWS S3 data bucket. 
+- **Azure Datalake**: Connect to a dataset in Azure Datalake. 
+- **H2O Drive**: Upload a dataset from H2O Drive.
+- **Kaggle**: Connect to a dataset hosted on Kaggle.
+- **Hugging Face**: Connect to a dataset on Hugging Face.
+
+## Data format 
+
+- Each data connector requires either a single `.csv` or `.pq` file, or the data to be in a `.zip` file for a successful import. 
+
+- H2O LLM studio requires a `.csv` file with a minimum of two columns, where one contains the instructions and the other has the model’s expected output. You can also include an additional validation dataframe in the same format or allow for an automatic train/validation split to assess the model’s performance.
+
+- Optionally, a **Parent Id** can be used for training nested data prompts that are linked to a parent question. 
+
+- During an experiment you can adapt the data representation with the following settings:
+    - **Prompt Column:** The column in the dataset containing the user prompt.
+    - **Answer Column:** The column in the dataset containing the expected output.
+    - **Parent Id Column:** An optional column specifying the parent id to be used for chained conversations. The value of this column needs to match an additional column with the name `id`. If provided, the prompt will be concatenated after preceding parent rows.
+    - **Id Column:**  This column is only required when the `Parent Id` is set. It identifies the record ID used for linked conversations. The value in the `Parent Id` column must match this column's value.
+
+:::info
+To train a chatbot style model, you need to convert your data into a question and answer format. There are other enterprise solutions by H2O.ai that may help you prep your data. For more information, see [H2O.ai's Generative AI page](https://h2o.ai/) and this blogpost about [H2O LLM DataStudio: Streamlining Data Curation and Data Preparation for LLMs related tasks](https://h2o.ai/blog/2023/streamlining-data-preparation-for-fine-tuning-of-large-language-models/).
+
+## Example data
+
+H2O LLM Studio provides a sample dataset (converted dataset from [OpenAssistant/oasst2](https://huggingface.co/datasets/OpenAssistant/oasst2))
+that can be downloaded [here](https://www.kaggle.com/code/philippsinger/openassistant-conversations-dataset-oasst2?scriptVersionId=160485459). It is recommended to use `train_full.csv` for training. This dataset is also downloaded and prepared by default when first starting the GUI. Multiple dataframes can be uploaded into a single dataset by uploading a `.zip` archive.

documentation/docs/guide/datasets/import-dataset.md

@@ -0,0 +1,222 @@
+---
+description: H2O LLM Studio provides a number of data connectors to support importing data from local or external sources and requires your data to be in a certain format for successful importing of data.
+---
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import Admonition from '@theme/Admonition';
+import upload_dataset from './upload-dataset.png';
+import upload_local_file from './upload-local-file.png';
+import import_s3_bucket from './import-s3-bucket.png';
+import import_kaggle_dataset from './import-kaggle-dataset.png';
+import import_h2odrive_dataset from './import-h2o-drive-dataset.png';
+import DatasetNameTooltip from '../../tooltips/experiments/_dataset-name.mdx';
+import ProblemTypeTooltip from '../../tooltips/experiments/_problem-type.mdx';
+import TrainDataframeTooltip from '../../tooltips/experiments/_train-dataframe.mdx';
+import ValidationDataframeTooltip from '../../tooltips/experiments/_validation-dataframe.mdx';
+import SystemColumnTooltip from '../../tooltips/experiments/_system-column.mdx';
+import PromptColumnTooltip from '../../tooltips/experiments/_prompt-column.mdx';
+import RejectedPromptColumnTooltip from '../../tooltips/experiments/_rejected-prompt-column.mdx';
+import AnswerColumnTooltip from '../../tooltips/experiments/_answer-column.mdx';
+import RejectedAnswerColumnTooltip from '../../tooltips/experiments/_rejected-answer-column.mdx';
+import ParentIdColumnTooltip from '../../tooltips/experiments/_parent-id-column.mdx';
+
+# Import a dataset
+
+H2O LLM Studio provides a number of data connectors to support importing data from local or external sources and requires your data to be in a certain format for successful importing of data. 
+
+For more information, see [Supported data connectors and format](data-connectors-format).
+
+## Import data
+
+Follow the relevant steps below to import a dataset to H2O LLM Studio.
+
+1. On the H2O LLM Studio left-navigation pane, click **Import dataset**.
+2. Select the relevant **Source** (data connector) that you want to use from the dropdown list .
+    :::note Data sources
+    <Tabs className="unique-tabs">
+    <TabItem value="upload" label="Upload" default>
+        <ol>
+        <li>
+        Drag and drop the file, or click <b>Browse</b> and select the file you want to upload.
+        </li>
+        <li>
+        Click <b>Upload</b>.
+        <img src={upload_dataset} alt="upload-dataset" />
+        </li>
+        </ol>
+    </TabItem>
+    <TabItem value="local" label="Local">
+        <ol>
+        <li>
+        Enter the file path as the <b>File Location</b> or select the relevant local directory that the dataset is located in. 
+        </li>
+        <li>
+        Click <b>Continue</b>.
+        <img src={upload_local_file} alt="upload-local-file" />
+        </li>
+        </ol>
+    </TabItem>
+    <TabItem value="aws" label="AWS S3">
+        <ol>
+        <li>
+        Enter values for the following fields:
+            <ul>
+            <li>
+            <b>S3 bucket name: </b> <br></br>
+            The name of the S3 bucket including the reletive file paths.
+            </li>
+            <li>
+            <b>AWS access key: </b><br></br>
+            The access key associated with your S3 bucket. This field is optional. If the S3 bucket is public, you can leave this empty for anonymous access. 
+            </li>
+            <li>
+            <b>AWS access secret: </b><br></br>
+            The access secret associated with your S3 bucket. This field is optional. If the S3 bucket is public, you can leave this empty for anonymous access.
+            </li>
+            <li>
+            <b>File name: </b><br></br>
+            Enter the file name of the dataset that you want to import. 
+            </li>
+            </ul>
+            <div>
+            <Admonition type="info" title="Note">
+                <p>For more information, see <a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html#access-keys-and-secret-access-keys">AWS credentials</a> and <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-bucket-intro.html">Methods for accessing a bucket</a> in the AWS Documentation.</p>
+            </Admonition>
+            </div>
+        </li>
+        <li>
+        Click <b>Continue</b>.
+        <img src={import_s3_bucket} alt="import-s3-bucket" />
+        </li>
+        </ol>
+    </TabItem>
+    <TabItem value="azure datalake" label="Azure Datalake">
+        <ol>
+        <li>
+        Enter values for the following fields:
+            <ul>
+            <li>
+            <b>Datalake connection string: </b><br></br>
+            Enter your Azure connection string to connect to Datalake storage.
+            </li>
+            <li>
+            <b>Datalake container name: </b><br></br>
+            Enter the name of the Azure Data Lake container where your dataset is stored, including the relative path to the file within the container. 
+            </li>
+            <li>
+            <b>File name: </b><br></br>
+            Specify the exact name of the file you want to import.
+            </li>
+            </ul>
+        </li>
+        <li>
+        Click <b>Continue</b>.
+        </li>
+        </ol>
+    </TabItem>
+    <TabItem value="h2o-drive" label="H2O-Drive">
+        <ol>
+        <li>
+        Select the dataset you want to upload from the list of datasets in H2O Drive.
+        </li>
+        <li>
+        Click <b>Continue</b>.
+        <img src={import_h2odrive_dataset} alt="import-h2odrive-dataset" />
+        </li>
+        </ol>
+    </TabItem>
+    <TabItem value="kaggle" label="Kaggle">
+        <ol>
+        <li>
+        Enter values for the following fields:
+            <ul>
+            <li>
+            <b>Kaggle API command: </b><br></br>
+            Enter the Kaggle API command that you want to execute.
+            </li>
+            <li>
+            <b>Kaggle username: </b><br></br>
+            Your Kaggle username for API authentication
+            </li>
+            <li>
+            <b>Kaggle secret key: </b><br></br>
+            Your Kaggle secret key for API authentication.
+            </li>
+            </ul>
+        </li>
+        <li>
+        Click <b>Continue</b>.
+        <img src={import_kaggle_dataset} alt="import-kaggle-dataset" />
+        </li>
+        </ol>
+    </TabItem>
+    <TabItem value="hugging face" label="Hugging Face">
+        <ol>
+        <li>
+        Enter values for the following fields:
+            <ul>
+            <li>
+            <b>Hugging Face dataset: </b><br></br>
+            Enter the name of the Hugging Face dataset.
+            </li>
+            <li>
+            <b>Split: </b><br></br>
+            Enter the specific data split you want to import (e.g., "train", "test").
+            </li>
+            <li>
+            <b>Hugging Face API token (optional): </b><br></br>
+            Enter your Hugging Face API token to authenticate access to private datasets or datasets with gated access.
+            </li>
+            </ul>
+        </li>
+        <li>
+        Click <b>Continue</b>.
+        </li>
+        </ol>
+    </TabItem>
+    </Tabs>
+    :::
+ 
+## Configure dataset
+
+Once you have successfully uploaded or imported your dataset, you can configure the dataset settings. Depending on the problem type, you may need to specify the following:
+
+:::info Tip
+You can upload a `.zip` file with both training and validation sets to avoid having to separately upload files.
+:::
+
+- **Dataset name:** <DatasetNameTooltip/>
+
+- **Problem Type:** <ProblemTypeTooltip/>
+
+- **Train Dataframe:** <TrainDataframeTooltip/>
+
+- **Validation Dataframe:** <ValidationDataframeTooltip/>
+
+- **System Column:** <SystemColumnTooltip/>
+
+- **Prompt Column:** <PromptColumnTooltip/>
+
+- **Rejected Prompt Column:** <RejectedPromptColumnTooltip/>
+
+- #### **Answer Column:** 
+
+  <AnswerColumnTooltip/>
+
+- **Rejected Answer Column:** <RejectedAnswerColumnTooltip/>
+
+- **Parent Id Column:** <ParentIdColumnTooltip/>
+
+![configure-dataset](configure-dataset.png)
+
+## Data validity check
+
+H2O LLM Studio will provide a preview of the dataset input (sample questions) and output (sample answers) according to the content of the imported dataset. Review the text to ensure that the input and output is as intended, and then click **Continue**. 
+
+## View dataset
+
+You will now be redirected to the **View datasets** screen. You should be able to see the dataset you just imported listed on the screen. 
+
+![view-dataset](view-imported-dataset.png)
+
+For more information about viewing dataset summary and statistics, see [View and manage datasets](view-dataset)

documentation/docs/guide/datasets/merge-datasets.md

@@ -0,0 +1,34 @@
+---
+description: H2O LLM Studio enables you to merge imported datasets into one main dataset. This functionality can be used to merge training and validation data together into one dataset or extend your existing dataset with more data and increase your dataset size. 
+---
+import Icon from "@material-ui/core/Icon";
+
+# Merge datasets
+
+H2O LLM Studio enables you to merge imported datasets into one main dataset. This functionality can be used to merge training and validation data together into one dataset or extend your existing dataset with more data and increase your dataset size. 
+
+:::info
+H2O LLM Studio does not merge dataset files in the sense that rows are combined, and duplicate rows are removed. "Merge", in this case, refers to bringing the dataset files a dataset might have to a single dataset (another dataset), continuing other dataset files already.
+:::
+
+Generally, you might want to merge datasets in H2O LLM Studio to have both the training data .csv and validation data .csv in one final dataset. 
+
+1. On the H2O LLM Studio left-navigation pane, click **View datasets**.  
+2. Click the <Icon>more_vert</Icon> Kebab menu of the dataset you want to merge with. 
+3. Click **Edit dataset**. 
+4. Click **Merge with existing dataset**.
+5. Select the dataset you want that you want to merge with. 
+    ![merge-datasets](merge-datasets.png)
+6. Click **Merge**.
+7. Adjust the dataset configuration if needed. For more information about the configurations, see [Configure dataset](./import-dataset#configure-dataset). 
+8. Click **Continue**.
+9. Review the text to ensure that the input and output is as intended, and then click **Continue**.
+
+Your datasets are now merged. 
+
+:::info
+Alternatively, you can also merge datasets at the point of [importing a dataset](./import-dataset) or combine both datasets (.csv files) into a `.zip` file before uploading it as a whole dataset. 
+:::
+
+
+

documentation/docs/guide/datasets/view-dataset.md

@@ -0,0 +1,74 @@
+---
+description: You can view, review, edit, or delete your datasets once you have imported them. You can also start a new experiment using a dataset you have imported.
+---
+import Icon from "@material-ui/core/Icon";
+
+# View and manage dataset
+
+You can view, review, edit, or delete your datasets once you have imported them. You can also start a new experiment using a dataset you have imported. 
+
+## View a dataset
+
+To view an imported dataset:
+
+1. On the H2O LLM Studio left-navigation pane, click **View datasets**.
+
+2. You will see the datasets table with a list of all the datasets you have imported so far. Click the name of the dataset that you want to view.
+
+    ![view-datasets](view-imported-dataset.png)
+
+    :::info
+    For more information about the dataset details you see on the table above, see [dataset configurations](import-dataset.md#configure-a-dataset).
+    :::
+
+## Dataset tabs
+
+You will see the following tabs that provide details and different aspects of your dataset.
+
+- **Sample train data** : This tab contains sample training data from the imported dataset.
+
+- **Sample train visualization:** This tab visualizes a few sample training data from the imported dataset in a question-answer format; simulating the way the chatbot would answer questions based on the training data. 
+
+- **Train data statistics:** This tab contains metrics about the training data (e.g., unique values) from the imported dataset.
+
+- **Summary:** This tab contains the following details about the dataset. 
+
+    | Name      | Description                          |
+    | ----------- | ------------------------------------ |
+    | **Name**        | Name of the dataset.  |
+    | **Problem type**        | Problem type of the dataset. |
+    | **Train dataframe**   | Name of the training dataframe in the imported dataset. An imported dataset can contain train, test, and validation dataframes.  |
+    | **Train rows**       | The number of rows the train dataframe contains.  |
+    | **Validation dataframe**       | Name of the validation dataframe in the imported dataset. An imported dataset can contain train, test, and validation dataframes.  |
+    | **Validation rows**         | The number of rows the validation dataframe contains. |
+    | **Labels**       | The labels the imported dataset contains.  |
+
+
+## Edit a dataset
+
+To edit an imported dataset,
+
+1. On the H2O LLM Studio left-navigation pane, click **View datasets**. You will see the datasets table with a list of all the datasets you have imported so far.
+2. Locate the row of the dataset you want to edit and click the <Icon>more_vert</Icon> Kebab menu.
+3. Select **Edit dataset**.
+4. Make the desired changes to the dataset configuration. You can also [merge the dataset with an existing dataset](merge-datasets) at this point.
+5. Click **Continue** and review the dataset with your changes. 
+
+<!-- 
+## Start a new experiment
+
+
+link to start a new experiment page in the experiments sub page.  -->
+
+## Delete a dataset
+
+When a dataset is no longer needed, you can delete it. Deleted datasets are permanently removed from the H2O LLM Studio instance.
+
+:::caution
+You can only delete datasets that are not linked to any experiments. If you wish to delete a dataset that is linked to an experiment, first [delete the experiment](../experiments/view-an-experiment#delete-an-experiment), and then delete the dataset. 
+:::
+
+1. On the H2O LLM Studio left-navigation pane, click **View datasets**.
+2. Click **Delete datasets**.
+3. Select the dataset(s) that you want to delete.
+4. Click **Delete** to confirm deletion.

documentation/docs/guide/experiments/compare-experiments.md

@@ -0,0 +1,21 @@
+---
+description: Using H2O LLM Studio, you can compare experiments and analyze how different model parameters affect model performance. 
+---
+# Compare experiments
+
+Using H2O LLM Studio, you can compare experiments and analyze how different model parameters affect model performance. 
+
+Follow the relevant steps below to compare experiments in H2O LLM Studio.
+
+1. On the H2O LLM Studio left-navigation pane, click **View experiments**.
+2. Click **Compare experiments**.
+3. Select the experiments you want to compare.
+4. Click **Compare experiments**.
+
+    ![compare experiments](compare-experiments.png)
+
+    The **Charts** tab visually represents the comparison of train/validation loss, metrics, and learning rate of selected experiments. The **Config** tab compares the configuration settings of selected experiments.  
+
+:::info note
+In addition, H2O LLM Studio also integrates with [Neptune](https://neptune.ai/) and [W&B](https://wandb.ai/), two powerful experiment tracking platforms. By enabling Neptune or W&B logging when starting an experiment, you can easily track and visualize all aspects of your experiment in real time. This includes model performance, hyperparameter tuning, and other relevant metrics.
+:::

documentation/docs/guide/experiments/create-an-experiment.md

@@ -0,0 +1,68 @@
+---
+description: This page highlights the steps needed to create an experiment in H2O LLM Studio.
+---
+import Icon from "@material-ui/core/Icon";
+
+# Create an experiment
+
+Follow the relevant steps below to create an experiment in H2O LLM Studio.
+
+1. On the H2O LLM Studio left-navigation pane, click **Create experiment**. Alternatively, you can click **New experiment** on the <Icon>more_vert</Icon> Kebab menu of the [View datasets](../datasets/view-dataset.md) page.
+
+2. Select the **Dataset** you want to use to fine-tune an LLM model.
+
+3. Select the **Problem type**.
+
+4. Provide a meaningful **Experiment name**.
+
+5. Define the parameters. The most important parameters are:
+    - **LLM Backbone**: This parameter determines the LLM architecture to use. It is the foundation model that you continue training. H2O LLM Studio has a predefined list of recommended foundation models available in the dropdown list. You can also type in the name of a [Hugging Face model](https://huggingface.co/models) that is not in the list, for example: `h2oai/h2o-danube2-1.8b-sft` or the path of a local folder that has the model you would like to fine-tune.
+    - **Mask Prompt Labels**: This option controls whether to mask the prompt labels during training and only train on the loss of the answer.
+    - Hyperparameters such as **Learning rate**, **Batch size**, and number of epochs determine the training process. You can refer to the tooltips that are shown next to each hyperparameter in the GUI to learn more about them.
+    - **Evaluate Before Training**: This option lets you evaluate the model before training, which can help you judge the quality of the LLM backbone before fine-tuning. 
+
+    H2O LLM Studio provides several metric options for evaluating the performance of your model. In addition to the BLEU score, H2O LLM Studio also offers the GPT3.5 and GPT4 metrics that utilize the OpenAI API to determine whether the predicted answer is more favorable than the ground truth answer. To use these metrics, you can either export your OpenAI API key as an environment variable before starting LLM Studio, or you can specify it in the **Settings** menu within the UI.
+
+    :::info note
+    H2O LLM Studio provides an overview of all the parameters you need to specify for your experiment. The default settings are suitable when you first start an experiment. To learn more about the parameters, see [Experiment settings](experiment-settings.md).
+    :::
+
+6. Click **Run experiment**.
+
+    ![run-experiment](run-experiment.png)
+
+## Run an experiment on the OASST data via CLI
+
+The steps below provide an example of how to to run an experiment on [OASST](https://huggingface.co/OpenAssistant) data via the command line interface (CLI).
+
+1. Get the training dataset (`train_full.csv`), [OpenAssistant Conversations Dataset OASST2](https://www.kaggle.com/code/philippsinger/openassistant-conversations-dataset-oasst2?scriptVersionId=160485459) and place it into the `examples/data_oasst2` folder; or download it directly using the [Kaggle API](https://www.kaggle.com/docs/api) command given below.
+
+ ```bash
+ kaggle kernels output philippsinger/openassistant-conversations-dataset-oasst2 -p examples/data_oasst2/
+ ```
+
+2. Go into the interactive shell or open a new terminal window. Install the dependencies first, if you have not installed them already. 
+
+ ```bash
+ make setup  # installs all dependencies
+ make shell
+ ```
+
+3. Run the following command to run the experiment. 
+
+ ```bash
+ python llm_studio/train.py -Y examples/example_oasst2.yaml
+ ```
+
+After the experiment is completed, you can find all output artifacts in the `examples/output_oasst2` folder.
+You can then use the `prompt.py` script to chat with your model.
+
+```bash
+python llm_studio/prompt.py -e examples/output_oasst2
+```
+
+4. To publish the model to Hugging Face, use the following command:
+
+ ```bash
+ python llm_studio/publish_to_hugging_face.py -p {path_to_experiment} -d {device} -a {api_key} -u {user_id} -m {model_name} -s {safe_serialization}
+ ```

documentation/docs/guide/experiments/evaluate-model-using-llm.md

@@ -0,0 +1,29 @@
+# Evaluate model using an AI judge
+
+H2O LLM Studio provides the option to use an AI Judge like ChatGPT or a local LLM deployment to evaluate a fine-tuned model. 
+
+Follow the instructions below to specify a local LLM to evaluate the responses of the fine-tuned model.
+
+1. Have an endpoint running of the local LLM deployment, which supports the OpenAI API format; specifically the [Chat Completions API](https://platform.openai.com/docs/guides/text-generation/chat-completions-api).
+
+2. Start the H2O LLM Studio server with the following environment variable that points to the endpoint. 
+    ```
+    OPENAI_API_BASE="http://111.111.111.111:8000/v1"
+    ```
+
+3. Once H2O LLM Studio is up and running, click **Settings** on the left navigation panel to validate that the endpoint is being used correctly. The **Use OpenAI API on Azure** setting must be set to Off, and the environment variable that was set above should be the **OpenAI API Endpoint** value as shown below. 
+    ![set-endpoint](set-endpoint.png)
+
+    :::info
+    Note that changing the value of this field here on the GUI has no effect. This is only for testing the correct setting of the environment variable.
+    :::
+
+4. Run an experiment using `GPT` as the **Metric** and the relevant model name available at your endpoint as the **Metric Gpt Model**.
+    ![set-metric-model](set-metric-model.png)
+
+5. Validate that it is working as intended by checking the logs. Calls to the LLM judge should now be directed to your own LLM endpoint. 
+    ![local-llm-judge-logs](local-llm-judge-logs.png)
+
+
+
+

documentation/docs/guide/experiments/experiment-settings.md

@@ -0,0 +1,468 @@
+---
+description: All the settings needed for creating an experiment are explored in this page.
+---
+import GeneralSettingsDataset from '../../tooltips/experiments/_dataset.mdx';
+import GeneralSettingsProblemType from '../../tooltips/experiments/_problem-type.mdx';
+import GSImportConfigFromYaml from '../../tooltips/experiments/_import-config-from-yaml.mdx';
+import GSExperimentName from '../../tooltips/experiments/_experiment-name.mdx';
+import GSLLMBackbone from '../../tooltips/experiments/_llm-backbone.mdx';
+import DSTrainDataframe from '../../tooltips/experiments/_train-dataframe.mdx';
+import DSvalidationStrategy from '../../tooltips/experiments/_validation-strategy.mdx';
+import DSvalidationSize from '../../tooltips/experiments/_validation-size.mdx';
+import DSdataSample from '../../tooltips/experiments/_data-sample.mdx';
+import DSpromptColumn from '../../tooltips/experiments/_prompt-column.mdx';
+import DSPromptColumnSeparator from '../../tooltips/experiments/_prompt-column-separator.mdx';
+import DSsystemColumn from '../../tooltips/experiments/_system-column.mdx';
+import DSanswerColumn from '../../tooltips/experiments/_answer-column.mdx';
+import DSparentIdColumn from '../../tooltips/experiments/_parent-id-column.mdx';
+import DStextPromptStart from '../../tooltips/experiments/_text-prompt-start.mdx';
+import DStextAnswerSeparator from '../../tooltips/experiments/_text-answer-separator.mdx';
+import DSaddEosTokentoprompt from '../../tooltips/experiments/_add-eos-token-to-prompt.mdx';
+import DSaddEosTokentoanswer from '../../tooltips/experiments/_add-eos-token-to-answer.mdx';
+import DSmaskPromptlabels from '../../tooltips/experiments/_mask-prompt-labels.mdx';
+import TSmaxLength from '../../tooltips/experiments/_max-length.mdx';
+import TSaddpromptanswertokens from '../../tooltips/experiments/_add-prompt-answer-tokens.mdx';
+import TSpaddingQuantile from '../../tooltips/experiments/_padding-quantile.mdx';
+import ASBackboneDtype from '../../tooltips/experiments/_backbone-dtype.mdx';
+import ASGradientcheckpointing from '../../tooltips/experiments/_gradient-checkpointing.mdx';
+import ASintermediateDropout from '../../tooltips/experiments/_intermediate-dropout.mdx';
+import ASpretrainedWeights from '../../tooltips/experiments/_pretrained-weights.mdx';
+import TSoptimizer from '../../tooltips/experiments/_optimizer.mdx';
+import TSlossfunction from '../../tooltips/experiments/_loss-function.mdx';
+import TSlearningRate from '../../tooltips/experiments/_learning-rate.mdx';
+import TSdifferentialLearningRateLayers from '../../tooltips/experiments/_differential-learning-rate-layers.mdx';
+import TSfreezeLayers from '../../tooltips/experiments/_freeze-layers.mdx';
+import TSattentionImplementation from '../../tooltips/experiments/_attention-implementation.mdx';
+import TSbatchSize from '../../tooltips/experiments/_batch-size.mdx';
+import TSepochs from '../../tooltips/experiments/_epochs.mdx';
+import TSschedule from '../../tooltips/experiments/_schedule.mdx';
+import TSminLearningRateRatio from '../../tooltips/experiments/_min-learning-rate-ratio.mdx';
+import TSwarmupEpochs from '../../tooltips/experiments/_warmup-epochs.mdx';
+import TSweightDecay from '../../tooltips/experiments/_weight-decay.mdx';
+import TSGradientclip from '../../tooltips/experiments/_gradient-clip.mdx';
+import TSgradAccumulation from '../../tooltips/experiments/_grad-accumulation.mdx';
+import TSlora from '../../tooltips/experiments/_lora.mdx';
+import TSuseDora from '../../tooltips/experiments/_use-dora.mdx';
+import TSloraR from '../../tooltips/experiments/_lora-r.mdx';
+import TSloraAlpha from '../../tooltips/experiments/_lora-alpha.mdx';
+import TSloraDropout from '../../tooltips/experiments/_lora-dropout.mdx';
+import TSuseRSlora from '../../tooltips/experiments/_use-rslora.mdx';
+import TSloraTargetModules from '../../tooltips/experiments/_lora-target-modules.mdx';
+import TSloraUnfreezeLayers from '../../tooltips/experiments/_lora-unfreeze-layers.mdx';
+import TSsavecheckpoint from '../../tooltips/experiments/_save-checkpoint.mdx';
+import TSevaluationepochs from '../../tooltips/experiments/_evaluation-epochs.mdx';
+import TSevaluationbeforetraining from '../../tooltips/experiments/_evaluate-before-training.mdx';
+import TStrainvalidationdata from '../../tooltips/experiments/_train-validation-data.mdx';
+import AStokenmaskprobability from '../../tooltips/experiments/_token-mask-probability.mdx';
+import ASskipParentprobability from '../../tooltips/experiments/_skip-parent-probability.mdx';
+import ASrandomparentprobability from '../../tooltips/experiments/_random-parent-probability.mdx';
+import ASneftunenoisealpha from '../../tooltips/experiments/_neftune_noise_alpha.mdx';
+import PSmetric from '../../tooltips/experiments/_metric.mdx';
+import PSmetricgptmodel from '../../tooltips/experiments/_metric-gpt-model.mdx';
+import PSmetricgpttemplate from '../../tooltips/experiments/_metric-gpt-template.mdx';
+import PSminlengthinference from '../../tooltips/experiments/_min-length-inference.mdx';
+import PSmaxlengthinference from '../../tooltips/experiments/_max-length-inference.mdx';
+import PSbatchsizeinference from '../../tooltips/experiments/_batch-size-inference.mdx';
+import PSdosample from '../../tooltips/experiments/_do-sample.mdx';
+import PSnumbeams from '../../tooltips/experiments/_num-beams.mdx';
+import PStemperature from '../../tooltips/experiments/_temperature.mdx';
+import PSrepetitionpenalty from '../../tooltips/experiments/_repetition-penalty.mdx';
+import PSstoptokens from '../../tooltips/experiments/_stop-tokens.mdx';
+import PStopk from '../../tooltips/experiments/_top-k.mdx';
+import PStopp from '../../tooltips/experiments/_top-p.mdx';
+import ESgpus from '../../tooltips/experiments/_gpus.mdx';
+import ESmixedprecision from '../../tooltips/experiments/_mixed-precision.mdx';
+import EScompilemodel from '../../tooltips/experiments/_compile-model.mdx';
+import ESfindunusedparameters from '../../tooltips/experiments/_find-unused-parameters.mdx';
+import EStrustremotecode from '../../tooltips/experiments/_trust-remote-code.mdx';
+import EShuggingfacebranch from '../../tooltips/experiments/_huggingface-branch.mdx';
+import ESnumofworkers from '../../tooltips/experiments/_number-of-workers.mdx';
+import ESseed from '../../tooltips/experiments/_seed.mdx';
+import LSlogstepsize from '../../tooltips/experiments/_log-step-size.mdx';
+import LSlogallranks from '../../tooltips/experiments/_log-all-ranks.mdx';
+import LSlogger from '../../tooltips/experiments/_logger.mdx';
+import LSneptuneproject from '../../tooltips/experiments/_neptune-project.mdx';
+import LSwandbproject from '../../tooltips/experiments/_wandb-project.mdx';
+import LSwandbentity from '../../tooltips/experiments/_wandb-entity.mdx';
+import NumClasses from '../../tooltips/experiments/_num-classes.mdx';
+
+# Experiment settings
+
+The settings for creating an experiment are grouped into the following sections: 
+ - [General settings](#general-settings) 
+ - [Dataset settings](#dataset-settings)
+ - [Tokenizer settings](#tokenizer-settings)
+ - [Architecture settings](#architecture-settings)
+ - [Training settings](#training-settings)
+ - [Augmentation settings](#augmentation-settings)
+ - [Prediction settings](#prediction-settings)
+ - [Environment settings](#environment-settings)
+ - [Logging settings](#logging-settings)
+
+The settings under each category are listed and described below.
+
+## General settings 
+
+### Dataset
+
+<GeneralSettingsDataset/>
+
+### Problem type
+
+<GeneralSettingsProblemType/>
+
+### Import config from YAML
+
+<GSImportConfigFromYaml/>
+
+### Experiment name
+
+<GSExperimentName/>
+
+### LLM backbone
+
+<GSLLMBackbone/>
+
+## Dataset settings
+
+### Train dataframe
+
+<DSTrainDataframe/>
+
+### Validation strategy
+
+<DSvalidationStrategy/>
+
+### Validation size
+
+<DSvalidationSize/>
+
+### Data sample
+
+<DSdataSample/>
+
+### System column
+
+<DSsystemColumn/>
+
+### Prompt column
+
+<DSpromptColumn/>
+
+### Prompt column separator
+
+<DSPromptColumnSeparator/>
+
+### Answer column
+
+<DSanswerColumn/>
+
+### Parent ID column
+
+<DSparentIdColumn/>
+
+### ID column
+
+<DSidColumn/>
+
+### Text prompt start
+
+<DStextPromptStart/>
+
+### Text answer separator
+
+<DStextAnswerSeparator/>
+
+### Add EOS token to prompt
+
+<DSaddEosTokentoprompt/>
+
+### Add EOS token to answer
+
+<DSaddEosTokentoanswer/>
+
+### Mask prompt labels
+
+<DSmaskPromptlabels/>
+
+### Num classes 
+
+<NumClasses/>
+
+The **Num classes** field should be set to the total number of classes in the [answer column](../datasets/import-dataset.md#answer-column) of the dataset.
+
+## Tokenizer settings
+
+### Max length 
+
+<TSmaxLength/>
+
+### Add prompt answer tokens
+
+<TSaddpromptanswertokens/>
+
+### Padding quantile
+
+<TSpaddingQuantile/>
+
+## Architecture settings
+
+### Backbone Dtype
+
+<ASBackboneDtype/>
+
+### Gradient Checkpointing
+
+<ASGradientcheckpointing/>
+
+### Intermediate dropout
+
+<ASintermediateDropout/>
+
+### Pretrained weights
+
+<ASpretrainedWeights/>
+
+## Training settings
+
+### Loss function
+
+<TSlossfunction/>
+
+For multiclass classification problems, set the loss function to **Cross-entropy**.
+
+### Optimizer
+
+<TSoptimizer/>
+
+### Learning rate
+
+<TSlearningRate/>
+
+### Differential learning rate layers
+
+<TSdifferentialLearningRateLayers/>
+
+By default, H2O LLM Studio applies **Differential learning rate Layers**, with the learning rate for the `classification_head` being 10 times smaller than the learning rate for the rest of the model.
+
+### Freeze layers
+
+<TSfreezeLayers/>
+
+### Attention Implementation
+
+<TSattentionImplementation/>
+
+### Batch size
+
+<TSbatchSize/>
+
+### Epochs
+
+<TSepochs/>
+
+### Schedule
+
+<TSschedule/>
+
+### Min Learning Rate Ratio
+
+<TSminLearningRateRatio/>
+
+### Warmup epochs
+
+<TSwarmupEpochs/>
+
+### Weight decay
+
+<TSweightDecay/>
+
+### Gradient clip
+
+<TSGradientclip/>
+
+### Grad accumulation
+
+<TSgradAccumulation/>
+
+### Lora
+
+<TSlora/>
+
+### Use Dora
+
+<TSuseDora/>
+
+### Lora R
+
+<TSloraR/>
+
+### Lora Alpha
+
+<TSloraAlpha/>
+
+### Lora dropout
+
+<TSloraDropout/>
+
+### Use RS Lora
+
+<TSuseRSlora/>
+
+### Lora target modules
+
+<TSloraTargetModules/>
+
+### Lora unfreeze layers
+
+<TSloraUnfreezeLayers/>
+
+### Save checkpoint
+
+<TSsavecheckpoint/>
+
+### Evaluation epochs
+
+<TSevaluationepochs/>
+
+### Evaluate before training
+
+<TSevaluationbeforetraining/>
+
+### Train validation data
+
+<TStrainvalidationdata/>
+
+## Augmentation settings
+
+### Token mask probability
+
+<AStokenmaskprobability/>
+
+### Skip parent probability
+
+<ASskipParentprobability/>
+
+### Random parent probability
+
+<ASrandomparentprobability/>
+
+### Neftune noise alpha
+
+<ASneftunenoisealpha/>
+
+## Prediction settings
+
+### Metric
+
+<PSmetric/>
+
+### Metric GPT model
+
+<PSmetricgptmodel/>
+
+### Metric GPT template
+
+<PSmetricgpttemplate/>
+
+### Min length inference
+
+<PSminlengthinference/>
+
+### Max length inference
+
+<PSmaxlengthinference/>
+
+### Batch size inference
+
+<PSbatchsizeinference/>
+
+### Do sample
+
+<PSdosample/>
+
+### Num beams
+
+<PSnumbeams/>
+
+### Temperature 
+
+<PStemperature/>
+
+### Repetition penalty
+
+<PSrepetitionpenalty/>
+
+### Stop tokens
+
+<PSstoptokens/>
+
+### Top K
+
+<PStopk/>
+
+### Top P
+
+<PStopp/>
+
+## Environment settings
+
+### GPUs
+
+<ESgpus/>
+
+### Mixed precision
+
+<ESmixedprecision/>
+
+### Compile model
+
+<EScompilemodel/>
+
+### Find unused parameters
+
+<ESfindunusedparameters/>
+
+### Trust remote code
+
+<EStrustremotecode/>
+
+### Hugging Face branch
+
+<EShuggingfacebranch/>
+
+### Number of workers
+
+<ESnumofworkers/>
+
+### Seed
+
+<ESseed/>
+
+## Logging settings
+
+### Log step size
+
+<LSlogstepsize/>
+
+### Log all ranks
+
+<LSlogallranks/>
+
+### Logger
+
+<LSlogger/>
+
+### Neptune project
+
+<LSneptuneproject/>
+
+### W&B project
+
+<LSwandbproject/>
+
+### W&B entity
+
+<LSwandbentity/>
+
+

documentation/docs/guide/experiments/export-trained-model.md

@@ -0,0 +1,61 @@
+---
+description: If you are ready to share your trained model with a broader community, H2O LLM Studio allows you to export the fine-tuned model to Hugging Face Hub with a single click.
+---
+# Publish model to HuggingFace
+
+If you are ready to share your trained model with a broader community, H2O LLM Studio allows you to export the fine-tuned model to [Hugging Face](https://huggingface.co/) with a single click.
+
+:::info note
+Before publishing your model to the Hugging Face Hub, you need to have an API key with write access. To obtain an API token with write access, follow the [instructions provided by Hugging Face](https://huggingface.co/docs/hub/security-tokens), which involve creating an account, logging in, and generating an access token with the appropriate permission.
+:::
+
+To publish a trained model to Hugging Face Hub:
+
+1. On the H2O LLM Studio left-navigation pane, click **View experiments**. You will see the experiments table with a list of all the experiments you have launched so far. 
+
+2. Click the name of the experiment that you want to export as a model.
+
+3. Click **Push checkpoint to huggingface**.
+
+4. Enter the **Account name** on Hugging Face to push the model to a particular account. Leaving it empty will push it to the default user account.
+
+5. Enter the **Hugging Face API** Key with write access.
+
+6. Click **Export**.
+
+    ![export model to Hugging Face](export-model-to-huggingface.png)
+
+## Download a model
+
+Click **Download model** on the **View experiments** page to download the model locally.
+
+Use the following code snippet to utilize the converted model in Jupyter Notebook or Google Colab.
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_name = "path_to_downloaded_model"  # either local folder or Hugging Face model name
+
+# Important: The prompt needs to be in the same format the model was trained with.
+# You can find an example prompt in the experiment logs.
+prompt = "<|prompt|>How are you?<|endoftext|><|answer|>"
+
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForCausalLM.from_pretrained(model_name)
+model.cuda().eval()
+
+inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to("cuda")
+# generate configuration can be modified to your needs
+tokens = model.generate(
+    **inputs,  # Input any question for the model. Ex: "What is the capital of USA?"
+    max_new_tokens=256,
+    temperature=0.3,
+    repetition_penalty=1.2,
+    num_beams=1
+)[0]
+tokens = tokens[inputs["input_ids"].shape[1]:]
+answer = tokenizer.decode(tokens, skip_special_tokens=True)
+print(answer)
+```
+
+You can enter any question for the model and change the parameters to get different outputs. 

documentation/docs/guide/experiments/import-to-h2ogpt.md

@@ -0,0 +1,48 @@
+---
+description: Once the model has been fine-tuned using H2O LLM Studio, you can then use h2oGPT to query, summarize, and chat with your model.
+---
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Import a model to h2oGPT
+
+Once the model has been fine-tuned using H2O LLM Studio, you can then use [h2oGPT](https://github.com/h2oai/h2ogpt/blob/main/README.md) to query, summarize, and chat with your model. 
+
+The most common method to get the model from H2O LLM Studio over to h2oGPT, is to import it into h2oGPT via HuggingFace. However, if your data is sensitive, you can also choose to download the model locally to your machine, and then import it directly into h2oGPT. 
+
+You can use any of the following methods: 
+
+- Publish the model to HuggingFace and import the model from HuggingFace
+- Download the model and import it to h2oGPT by specifying the local folder path
+- Download the model and upload it to h2oGPT using the file upload option on the UI
+- Pull a model from a Github repository or a resolved web link
+
+## Steps
+
+1. [Publish the model to HuggingFace](export-trained-model.md) or [download the model locally](export-trained-model.md#download-a-model). 
+
+2. If you opt to download the model, make sure you extract the downloaded .zip file. 
+
+3. Use the following command to import it into h2oGPT.
+    ```
+    python generate.py --base_model=[link_or_path_to_folder]
+    ```
+
+    :::note Examples
+    <Tabs className="unique-tabs">
+    <TabItem value="Example1" label="From HuggingFace">
+    <pre><code>python generate.py --base_model=HuggingFaceH4/zephyr-7b-beta</code></pre>
+    </TabItem>
+    <TabItem value="Example2" label="From a Local File">
+    <pre><code>python generate.py --base_model=zephyr-7b-beta.Q5_K_M.gguf</code></pre>
+    </TabItem>
+    <TabItem value="Example3" label="From a Repository">
+    <pre><code>python generate.py --base_model=TheBloke/zephyr-7B-beta-AWQ</code></pre>
+    </TabItem>
+    </Tabs>
+    :::
+
+:::info
+For more information, see the [h2oGPT documentation](https://github.com/h2oai/h2ogpt/blob/main/docs/FAQ.md#adding-models). 
+:::
+

documentation/docs/guide/experiments/supported-problem-types.md

@@ -0,0 +1,30 @@
+# Supported problem types
+
+## Overview
+
+H2O LLM Studio supports various problem types that allow users to fine-tune models for different tasks. The five supported problem types are explained below.
+
+## Causal language modeling 
+
+- **Description:** Causal language modeling involves predicting the next token in a sequence, based only on the preceding tokens (i.e., the left side of the sequence). It is commonly used for tasks such as text generation. It is used to fine-tune large language models.
+
+## Causal classification modeling
+
+- **Description:** Causal classification modeling involves assigning one or more categorical target labels to an input text. It is used for fine-tuning models to perform text classification tasks.
+
+- **Supported classification tasks:** Binary, multi-class, and multi-label classification.
+
+## Causal regression modeling
+
+- **Description:** Causal regression modeling assigns one or more continuous target labels to an input text. It is used to fine-tune models for text regression tasks. 
+
+- **Supported regression tasks:** Multi-label regression.
+
+## Sequence to sequence modeling
+
+- **Description:** A type of machine learning architecture designed to transform one sequence into another. It is commonly used for tasks like machine translation, text summarization, and speech recognition.
+
+## DPO modeling
+
+- **Description:** The DPO modeling is used to fine-tune large language models using Direct Preference Optimization (DPO), a method that helps large, unsupervised language models better match human preferences using a simple classification approach. 
+

documentation/docs/guide/experiments/view-an-experiment.md

@@ -0,0 +1,85 @@
+---
+description: You can view, rename, stop, or delete your experiments once you launch them. You can also create a new experiment based on an experiment you have already launched.
+---
+import Icon from "@material-ui/core/Icon";
+
+# View and manage experiments
+
+You can view, rename, stop, or delete your experiments once you launch them. You can also create a new experiment based on an experiment you have already launched.
+
+## View an experiment
+
+To view an experiment:
+
+1. On the H2O LLM Studio left-navigation pane, click **View experiments**.
+
+2. You will see the experiments table with a list of all the experiments you have launched so far. Click the name of the experiment that you want to view.
+
+ ![view-experiments](view-experiments.png)
+
+## Experiment tabs
+
+Once you click the name of the experiment, you will see the following tabs that provide details and different aspects of your experiment.
+
+- **Charts** : This tab visually represents the train/validation loss, metrics, and learning rate. These charts allow you to easily track your model’s performance as it trains.
+
+    ![charts tab](charts-tab.png)
+
+- **Summary** : This tab contains the following details about an experiment.  
+
+    | Name      | Description                          |
+    | ----------- | ------------------------------------ |
+    | **Name**        | Name of the experiment.  |
+    | **Dataset**  | Name of the dataset. |
+    | **Problem type** | The problem type of the experiment. |
+    | **Seed** | The random seed value that H2O LLM Studio uses during model training. |
+    | **GPU list** | The list of GPUs H2O LLM Studio can use for the experiment. |
+    | **Loss** | The loss function. |
+    | **Metric** | The metric to evaluate the model’s performance. |
+    | **Val metric** | The measure of how well the experiment was performed. |
+
+- **Train data insights** : This tab displays the model’s first batch, so you can verify that the input data representation is correct. Also, it provides insight into how your data is being processed and can help identify potential issues early on in the experiment.
+
+- **Validation prediction insights** : This tab displays model predictions for random, best, and worst validation samples. This tab becomes available after the first validation run and allows you to evaluate how well your model generalizes to new data. 
+
+    ![best validation sample](best-validation-sample.png)
+
+    ![worst validation sample](worst-validation-sample.png)
+
+    The **Worst validation samples** give you an idea of where the model is having issues, and the model can be used to fine-tune further.
+
+- **Logs and Config tabs** : These two tabs show you the logs and configuration of the experiment. You can keep track of any changes made and quickly troubleshoot the issues that arise.
+
+- **Chat** : This tab provides a unique opportunity to interact with your trained model and get instant feedback on its performance. The **Chat** tab becomes available after the training is completed and can be used to evaluate how well your model performs in a conversational setting. 
+
+ :::info note
+ You can use the **Chat** feature only when there are no other experiments running. The chatbot is unavailable if the GPU is occupied by another experiment.
+ :::
+
+ ![chat tab](chat-tab.png)
+
+## Stop an experiment
+
+You can stop a running experiment if you no longer need it to be completed. 
+
+1. On the H2O LLM Studio left-navigation pane, click **View experiments**.
+2. Click **Stop experiments**.
+3. Select the experiment(s) that you want to stop. 
+4. Click **Stop experiments**.
+
+You can also click **Stop experiment** on the <Icon>more_vert</Icon> Kebab menu of the relevant experiment row to stop an experiment from running. 
+
+  ![stop experiment](stop-experiment.png)
+
+## Delete an experiment
+
+When an experiment is no longer needed, you can delete it. Deleted experiments are permanently removed from the H2O LLM Studio instance.
+
+1. On the H2O LLM Studio left-navigation pane, click **View experiments**.
+2. Click **Delete experiments**.
+3. Select the experiment(s) that you want to delete and click **Delete experiments**.
+4. Click **Delete** to confirm deletion.
+
+You can also click **Delete experiment** in the kebab menu of the relevant experiment row to delete an experiment. 
+
+  ![delete experiment](delete-experiment.png)

documentation/docs/index.md

@@ -0,0 +1,110 @@
+---
+slug: /
+displayed_sidebar: defaultSidebar
+title: H2O LLM Studio | Docs
+hide_table_of_contents: true
+hide_title: true
+description: 'A framework and no-code GUI designed for fine-tuning state-of-the-art large language models (LLMs)'
+---
+
+import H2OHome from '@site/src/components/H2OHome';
+
+<H2OHome title="H2O LLM Studio" description="A framework and no-code GUI designed for fine-tuning state-of-the-art large language models (LLMs)" sections={[
+  {
+    subsections: [
+      {
+        title: "Get started",
+        icon: "rocket_launch",
+        items: [
+          {
+            label: "What is H2O LLM Studio?",
+            to: "/get-started/what-is-h2o-llm-studio",
+          },
+          {
+            label: "Set up H2O LLM Studio",
+            to: "/get-started/set-up-llm-studio",
+          },
+          {
+            label: "Core features",
+            to: "/get-started/core-features",
+          },
+          {
+            label: "Model flow",
+            to: "/get-started/llm-studio-flow",
+          },
+        ],
+      },
+      {
+        title: "Datasets",
+        icon: "dataset",
+        subtitle: null,
+        items: [
+          {
+            label: "Data connectors and data format",
+            to: "/guide/datasets/data-connectors-format",
+          },
+          {
+            label: "Import a dataset",
+            to: "/guide/datasets/import-dataset",
+          },
+          {
+            label: "View and manage a dataset",
+            to: "/guide/datasets/view-dataset",
+          },
+          {
+            label: "Merge datasets",
+            to: "/guide/datasets/merge-datasets",
+          },
+        ],
+      },
+      {
+        title: "Experiments",
+        icon: "timeline",
+        items: [
+          {
+            label: "Experiment settings",
+            to: "/guide/experiments/experiment-settings",
+          },
+          {
+            label: "Create an experiment",
+            to: "/guide/experiments/create-an-experiment",
+          },
+          {
+            label: "Monitor experiments",
+            to: "/guide/experiments/view-an-experiment",
+          },
+          {
+            label: "Compare experiments",
+            to: "/guide/experiments/compare-experiments",
+          },
+          {
+            label: "Publish model",
+            to: "/guide/experiments/export-trained-model",
+          },
+        ],
+      },
+            {
+        title: "Helpful links",
+        icon: "link",
+        items: [
+          {
+            label: "FAQs",
+            to: "/faqs",
+          },
+          {
+            label: "Concepts",
+            to: "/concepts",
+          },
+          {
+            label: "Videos",
+            to: "/get-started/videos",
+          },
+          {
+            label: "Key Terms",
+            to: "/key-terms",
+          },
+        ],
+      },
+    ],
+  },
+]} />

documentation/docs/key-terms.md

@@ -0,0 +1,78 @@
+---
+description: H2O LLM Studio uses several key terms across its documentation. 
+---
+# Key terms 
+
+H2O LLM Studio uses several key terms across its documentation, and each, in turn, is explained in the sections below.
+
+## Prompt Engineering
+
+Prompt engineering involves crafting precise and effective input queries to guide language models in generating desired outputs or responses.
+
+## Agents
+
+Software entities or components that interact with data or perform tasks within a system.
+
+## ELO
+
+An algorithm or method used to assess and rank the performance or accuracy of language models based on their proficiency in understanding and processing textual data.
+
+## Vector Store
+
+A Vector Store stores numerical representations of text for fast access in language models.
+
+## Pre-training 
+
+The initial phase of training a machine learning model on a large dataset to learn general features before fine-tuning on a specific task.
+
+## Attention
+
+A mechanism that enables models to focus on specific parts of input data relevant to the task at hand, enhancing their understanding and performance.
+
+## Embedding
+
+Embedding refers to a mathematical representation of words or tokens in a numerical vector space, enabling machine learning models to understand and process language based on their context and relationships.
+
+## Language Model
+
+A language model is an AI system that understands and generates human language, predicting and generating text based on patterns and context within a given sequence of words.
+
+## Transformer
+
+A Transformer refers to a neural network architecture specifically designed for processing sequential data like text, using attention mechanisms to learn contextual relationships between words or tokens.
+
+## Encoders and Decoders
+
+Encoders and decoders are vital parts of sequence-to-sequence models used in natural language processing. Encoders process input data into a fixed-size representation, while decoders generate an output sequence based on that representation.
+
+##  Text generation
+
+Text generation is the process of creating written content, such as sentences or paragraphs, using machine learning or AI algorithms based on patterns learned from existing text data.
+
+## In-context learning
+
+In-context learning refers to the process where a machine learning model continuously improves and adapts by considering the context of new information within its existing knowledge, enhancing its accuracy and understanding over time.
+
+## Few-shot learning
+
+Few-shot learning refers to a machine learning technique where a model can learn from a very small amount of labeled data to generalize and make predictions accurately on new, unseen data.
+
+## Summarization
+
+Summarization is the process of condensing a larger piece of text into a shorter, coherent version while retaining its essential information.
+
+## Fine-tuning
+
+Fine-tuning refers to adjusting and optimizing a pre-trained machine learning model using specific data to enhance its performance for a particular task.
+
+## GPT
+
+GPT stands for "Generative Pre-trained Transformer," a type of language model that uses transformers to understand and generate human-like text based on vast amounts of training data.
+
+# GPU deployment
+
+GPU deployment is the utilization of graphics processing units (GPUs) to execute and accelerate the computations involved in deploying machine learning models, improving speed and efficiency in model inference or training.
+
+# Tokenization
+
+Tokenization is the process of breaking text into smaller units, typically words or phrases, to analyze or process them individually within a natural language processing system.

documentation/docs/tooltips/experiments/_add-eos-token-to-answer.mdx

@@ -0,0 +1 @@
+Adds EOS token at end of answer.

documentation/docs/tooltips/experiments/_add-eos-token-to-prompt.mdx

@@ -0,0 +1 @@
+Adds EOS token at end of prompt.