Update README.md

README.md

@@ -1,3 +1,99 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+language:
+- en
+base_model:
+- mistralai/Mistral-7B-Instruct-v0.3
+pipeline_tag: text2text-generation
+library_name: unsloth
+---
+
+# ModelCard for UnGPT-v1
+
+## Model Details
+- **Name:** UnGPT-v1
+- **Foundation Model:** Mistral v0.3 (7B parameters)
+- **Recommended Context Length:** 16k tokens
+- **Fine-tuning Methodology:** LoRA-based training with Odds Ratio Preference Optimization method, using a combination of ebooks and synthetic data.
+
+## Usage Instructions
+Use the Alpaca format for prompts:
+```
+### Instruction:
+{instruction}
+
+### Input:
+{input}
+
+### Response:
+```
+**Example prompts**
+
+For instructions, it is not recommended to deviate from the provided examples. For the input, a minimum is 10 sentences, but more can work as the model can handle longer context sizes (Thanks to the Mistral 7B v0.3 base model).
+
+  1. **Completion Prompt:**
+      ```
+      ### Instruction:
+      Continue writing the story while retaining writing style. Write about 10 sentences.
+    
+      ### Input:
+      It was a dark and stormy night...
+    
+      ### Response:
+      ```
+
+ 2. **Fill-in-the-middle Prompt:**
+     ```
+     ### Instruction:
+     Fill in the missing part of the story ({{FILL_ME}}) with about 10 sentences while retaining the writing style.
+    
+     ### Input:
+     The bus was speeding down the road, cops chasing after it. 
+     {{FILL_ME}}
+     She woke up to find herself in an unfamiliar room...
+    
+      ### Response:
+     ```
+
+## Dataset Preparation
+
+For dataset acquisition and cleanup please refer steps 1 and 2 of my text-completion example, [molbal/llm-text-completion-finetune](https://github.com/molbal/llm-text-completion-finetune/).
+
+Chunking: Split texts into chunks based on sentence boundaries, aiming for 100 sentences per example.
+   - For completion examples, 90 sentences were used as input, 10 sentences as response.
+   - For fill-in-the-middle examples, 80 + 10 sentences as input (before and after the {{FILL_ME}} placeholder, respectively), and 10 sentences as response.
+
+The beauty of the ORPO method is that for a single prompt we can set both a positive and a negative example. I wanted the model to avoid 'GPTisms' so I had gpt4o-mini generate answers both for completion and FOM tasks and added them as a neative example.
+
+The dataset used is ~15k examples, each approximately 9000 characters long including input, accepted and refused response. (Note these are characters not tokens)
+
+## Training setup
+
+- Fine-tuned the Mistral v0.3 foundation model using Unsloth and ORPO trainer.
+
+- Training configuration:
+  - Batch size: 1
+  - Gradient accumulation steps: 4
+  - Learning rate scheduler type: Linear
+  - Optimizer: AdamW (8-bit)
+  - Number of training epochs: 1
+
+- Hardware
+  - I used GPU accelerated containers from the provider vast.ai (My referral link: https://cloud.vast.ai/?ref_id=123492 ) and executed training for ~8 hours on a single RTX 4090. 
+
+- Training costs
+  - ~5€ for renting a GPU pod (+15€ in unsuccessful attempts)
+  - ~5€ in OpenAI API costs for generating refusals
+
+**Licensing and Citation**
+
+- **License:** This model is licensed under the Apache License 2.0.
+- **Citation:**
+```bibtex
+@misc{ungpt-v1,
+  author = Bálint Molnár-Kaló,
+  title = {UnGPT-v1: A Fine-tuned Mistral Model for Story Continuation},
+  howpublished = {\url{https://huggingface.co/models/molbal/UnGPT-v1}},
+  year = 2024
+}
+```