---
license: apache-2.0
library_name: peft
tags:
- trl
- sft
- generated_from_trainer
base_model: TinyLlama/TinyLlama-1.1B-Chat-v0.3
model-index:
- name: tinyllama-colorist-lora-v0.3
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

<h1 align="center"><font color="red">tinyllama-colorist-lora-v0.3</font></h1>


![image/png](https://cdn-uploads.huggingface.co/production/uploads/628fcb73267c3813eb5ae99d/UMg3Uviv6JcwD4D6Vil7o.png)

This model, `tinyllama-colorist-lora-v0.3`, is a fine-tuned version of [TinyLlama/TinyLlama-1.1B-Chat-v0.3](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v0.3) on the color dataset.

## <font color="yellow">Study Motivation</font>

To study this new TinyLlama model as a replacement for Llama2 for resource-constrained environment. Also, in the future I will perform the Fine-Tuning of this model for Chat and for a specific domain in Portuguese and Spanish 🤗.


## <font color="yellow">Prompt format</font>
The model training process is similar to the regular Llama2 model with a chat prompt format like this:
```
<|im_start|>user\n{question}<|im_end|>\n<|im_start|>assistant\n{answer}<|im_end|>\n
```

## <font color="yellow">Instructions for use</font>
```
User Input: Give me a sky blue color.
LLM response: #6092ff
```



## <font color="yellow">Model usage</font>
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer
from transformers import pipeline

def print_color_space(hex_color):
    def hex_to_rgb(hex_color):
        hex_color = hex_color.lstrip('#')
        return tuple(int(hex_color[i:i+2], 16) for i in (0, 2, 4))
    r, g, b = hex_to_rgb(hex_color)
    print(f'{hex_color}: \033[48;2;{r};{g};{b}m           \033[0m')

tokenizer = AutoTokenizer.from_pretrained(model_id_colorist_final)
pipe = pipeline(
    "text-generation",
    model=model_id_colorist_final,
    torch_dtype=torch.float16,
    device_map="auto",
)

from time import perf_counter
start_time = perf_counter()

prompt = formatted_prompt('give me a pure brown color')
sequences = pipe(
    prompt,
    do_sample=True,
    temperature=0.1,
    top_p=0.9,
    num_return_sequences=1,
    eos_token_id=tokenizer.eos_token_id,
    max_new_tokens=12
)
for seq in sequences:
    print(f"Result: {seq['generated_text']}")

output_time = perf_counter() - start_time
print(f"Time taken for inference: {round(output_time,2)} seconds")
```


### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0002
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- training_steps: 200
- mixed_precision_training: Native AMP

### Training results



### Framework versions

- PEFT 0.9.0
- Transformers 4.38.2
- Pytorch 2.1.0+cu121
- Datasets 2.18.0
- Tokenizers 0.15.2