Model Card for Model ID
This model is a adapter for databricks/dolly-v2-3b, finetuned on hivaze/emphatical_daily_dialogues. Main goal of this model is to train model to create emphatical dialogues, which are controlled by instructions.
Model Details
Model Description
Prompt template: "{intro}\n\n### Instruction:\n{instruction}\n\n### Response:\n{response}\n"
Example intro: "You are a kind and empathetic interlocutor. You are talking to a person. Below is an instruction that describes a task. Write a response that appropriately completes the request"
Example instruction: "You try to chit-chat. Complete a phrase, acting like an interlocutor."
Training params:
train_args = TrainingArguments(
per_device_train_batch_size=8, # can be 4 with llama
per_device_eval_batch_size=8, # can be 4 with llama
gradient_accumulation_steps=4,
warmup_steps=20,
# max_steps=200,
optim="adamw_torch",
learning_rate=4e-5, # many possible values here from 1e-5 to 2e-4
# save_strategy="steps",
fp16=True,
# bf16=True, # a100 required
num_train_epochs=2,
evaluation_strategy="steps",
eval_steps=50,
save_strategy="steps",
save_steps=400,
logging_strategy="steps",
logging_steps=10,
logging_dir=f"{local_output_dir}/runs",
report_to="tensorboard",
output_dir=local_output_dir
)
LoRA config:
config = LoraConfig(
r=16, # can be 8 with llama
lora_alpha=32, # can be 16 with llama
# target_modules=["q_proj", "v_proj"],
target_modules=['query_key_value'],
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM"
)
- Developed by: hivaze
- Model type: LoRA adapter for GPTNeoXForCausalLM
- Language(s) (NLP): Primarly english
- Finetuned from model [optional]: databricks/dolly-v2-3b
- Git repository: https://github.com/hivaze/friendly_chatbot_task
Tensorboard
