This model, ["Mistral-7B-Customer-Support-v1"], is a fine-tuned version of the mistralai/Mistral-7B-Instruct-v0.2, specifically tailored for the Custumer Support domain. It is optimized to answer questions and assist users with various support transactions. It has been trained using hybrid synthetic data generated using our NLP/NLG technology and our automated Data Labeling (DAL) tools.
The goal of this model is to show that a generic verticalized model makes customization for a final use case much easier. For example, if you are "ACME Company", you can create your own customized model by using this fine-tuned model and a doing an additional fine-tuning using a small amount of your own data. An overview of this approach can be found at: From General-Purpose LLMs to Verticalized Enterprise Models
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("bitext-llm/Mistral-7B-Customer-Support-v1")
tokenizer = AutoTokenizer.from_pretrained("bitext-llm/Mistral-7B-Customer-Support-v1")
inputs = tokenizer("<s>[INST] I want to change to the standard account [/INST] ", return_tensors="pt")
outputs = model.generate(inputs['input_ids'], max_length=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
The model employs the MistralForCausalLM architecture with a LlamaTokenizer. It maintains the configuration of the base Mistral model but has been fine-tuned to better understand and generate responses related to customer service.
The model was fine-tuned using the bitext/Bitext-customer-support-llm-chatbot-training-dataset, which is designed for question and answer interactions in the customer service sector. This dataset includes instructions and responses across a variety of customer service topics, ensuring that the model can handle a wide range of inquiries related to this field. The dataset covers 27 intents assigned to 10 categories such as cancel_order, place_order, change_order, and check_invoice. Each intent has around 1000 examples, illustrating a training process aimed at understanding and generating accurate responses for customer service interactions.
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 1.6865 | 0.01 | 1 | 2.0557 |
| 0.6351 | 0.25 | 32 | 0.8355 |
| 0.5724 | 0.5 | 64 | 0.7859 |
| 0.5249 | 0.75 | 96 | 0.7711 |
| 0.516 | 1.0 | 128 | 0.7667 |
The dataset used for fine-tuning can train Large Language Models for both Fine Tuning and Domain Adaptation. It includes:
The dataset covers the following categories and intents:
The dataset includes various entities such as:
The dataset contains tags for various linguistic phenomena:
This model should be used responsibly, considering ethical implications of automated customer service. It is important to ensure that the model's advice complements human expertise and adheres to relevant customer service guidelines.
This model was developed by Bitext and trained on infrastructure provided by Bitext.
This model, "Mistral-7B-Customer-Support-v1," is licensed under the Apache License 2.0 by Bitext Innovations International, Inc. This open-source license allows for free use, modification, and distribution of the model but requires that proper credit be given to Bitext.
You may view the full license text at Apache License 2.0.
This licensing ensures the model can be used widely and freely while respecting the intellectual contributions of Bitext. For more detailed information or specific legal questions about using this license, please refer to the official license documentation linked above.
axolotl version: 0.4.0
base_model: mistralai/Mistral-7B-Instruct-v0.2
model_type: MistralForCausalLM
tokenizer_type: LlamaTokenizer
is_mistral_derived_model: true
hub_model_id: malmarjeh/Mistral-7B-instruct-v0.2
load_in_8bit: false
load_in_4bit: true
strict: false
datasets:
- path: bitext/Bitext-customer-support-llm-chatbot-training-dataset
type:
system_prompt: 'You are an expert in customer support.'
field_instruction: instruction
field_output: response
format: '[INST] {instruction} [/INST]'
no_input_format: '[INST] {instruction} [/INST]'
#datasets:
# - path: json
# type: alpaca_w_system.load_open_orca
#data_files: file.zip
dataset_prepared_path:
val_set_size: 0.05
output_dir: ./qlora-out
adapter: qlora
lora_model_dir:
sequence_len: 1024
sample_packing: true
pad_to_sequence_len: true
eval_sample_packing: False
lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:
lora_target_modules:
- gate_proj
- down_proj
- up_proj
- q_proj
- v_proj
- k_proj
- o_proj
wandb_project: axolotl
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:
gradient_accumulation_steps: 4
micro_batch_size: 8
num_epochs: 1
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.0002
train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false
gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
loss_watchdog_threshold: 5.0
loss_watchdog_patience: 3
warmup_steps: 10
evals_per_epoch: 4
eval_table_size:
eval_max_new_tokens: 128
saves_per_epoch: 1
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
bos_token: '<s>'
eos_token: '</s>'
unk_token: '<unk>'