README.md

metadata

license: mit
base_model:
  - microsoft/phi-1_5

Model Card for Model ID

Summary

This is a supervised fine-tuned model for text completion based on Phi 1.5. It has been finetuned on a filtered version of the The Complete Works of William Shakespeare, which can be found and downloaded from here: https://www.gutenberg.org/ebooks/100.

Model Description

• Developed by: Course Organizers
• Finetuned from model: microsoft/phi-1_5

Training Details

This model has been trained using the TLR library and SFTTrainer class from Huggingface.

Training Data

The Complete Works of William Shakespeare, which can be found and downloaded from here: https://www.gutenberg.org/ebooks/100

Training Hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• per_device_train_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 1
• optimizer: Adam with betas=(0.9, 0.95)
• lr_scheduler_type: linear
• weight_decay: 0.1
• num_epochs: 1

Framework Versions

• accelerate==0.26.1
• datasets==2.16.1
• transformers==4.45.2
• trl==0.11.2

Compute Infrastructure and Hardware

Slurm cluster with 8 x H100 Nvidia GPUs.