README.md

metadata

language:
  - en
license: mit
library_name: transformers
tags:
  - nlp
  - phi
  - phi-2
  - instruct
base_model:
  - microsoft/phi-2
datasets:
  - Open-Orca/SlimOrca
  - prince-canuma/TinyOrca
model-index:
  - name: Damysus-2.7B-Chat
    results:
      - task:
          type: text-generation
        metrics:
          - name: Average
            type: Average
            value: 60.49
            verified: true
        source:
          name: Open LLM Leaderboard
          url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard
      - task:
          type: text-generation
        dataset:
          name: ARC (25-shot)
          type: ai2_arc
        metrics:
          - name: Accuracy Norm
            type: acc_norm
            value: 59.81
            verified: true
        source:
          name: Open LLM Leaderboard
          url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard
      - task:
          type: text-generation
        dataset:
          name: Hellaswag (10-shot)
          type: Hellaswag
        metrics:
          - name: Accuracy Norm
            type: acc
            value: 74.52
            verified: true
        source:
          name: Open LLM Leaderboard
          url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard
      - task:
          type: text-generation
        dataset:
          name: MMLU (5-shot)
          type: MMLU
        metrics:
          - name: Accuracy
            type: acc
            value: 56.33
            verified: true
        source:
          name: Open LLM Leaderboard
          url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard
      - task:
          type: text-generation
        dataset:
          name: Truthful QA
          type: Truthful_QA
        metrics:
          - name: Multi-true
            type: mc2
            value: 46.74
            verified: true
        source:
          name: Open LLM Leaderboard
          url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard
      - task:
          type: text-generation
        dataset:
          name: Winogrande (5-shot)
          type: Winogrande
        metrics:
          - name: Accuracy
            type: acc
            value: 75.06
            verified: true
        source:
          name: Open LLM Leaderboard
          url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard
      - task:
          type: text-generation
        dataset:
          name: GSM8K (5-shot)
          type: GSM8K
        metrics:
          - name: Accuracy
            type: acc
            value: 50.64
            verified: true
        source:
          name: Open LLM Leaderboard
          url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard

Model Summary

This model is a instruction-tuned version of Phi-2, a Transformer model with 2.7 billion parameters from Microsoft. The model has undergone further training to better follow specific user instructions, enhancing its ability to perform tasks as directed and improve its interaction with users. This additional training helps the model to understand context better, generate more accurate and relevant responses, and adapt to a wide range of language-based tasks such as:

• Questions and Answers,
• Data Extraction,
• Structured Outputs (i.e., JSON outputs),
• And providing explanations,

Model Description

This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.

• Developed by: Prince Canuma
• Model type: Transformer
• License: MIT
• Finetuned from model: microsoft/phi-2

Uses

You can use this model to build local/cloud RAG applications. It can serve as the:

• Answer synthesizer,
• Summarizer,
• Or query rewriter model.

Limitations

This model inherits some of the base model's limitations, such as:

• Generate Inaccurate Code and Facts: The model may produce incorrect code snippets and statements. Users should treat these outputs as suggestions or starting points, not as definitive or accurate solutions.
• Limited Scope for code: Majority of Phi-2 training data is based in Python and use common packages such as "typing, math, random, collections, datetime, itertools". If the model generates Python scripts that utilize other packages or scripts in other languages, we strongly recommend users manually verify all API uses.
• Language Limitations: The model is primarily designed to understand standard English. Informal English, slang, or any other languages might pose challenges to its comprehension, leading to potential misinterpretations or errors in response.

How to Get Started with the Model

Use the code below to get started with the model.

from transformers import pipeline, Conversation

chatbot = pipeline("conversational", model="prince-canuma/Damysus-2.7B-Chat")
conversation = Conversation("I'm looking for a movie - what's your favourite one?")
output = chatbot(conversation)

print(output)

Or you can instatiate the model and tokenizer directly

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("prince-canuma/Damysus-2.7B-Chat")
model = AutoModelForCausalLM.from_pretrained("prince-canuma/Damysus-2.7B-Chat")

inputs = tokenizer.apply_chat_template(
    [
        {"content":"You are an helpful AI assistant","role":"system"},
        {"content":"I'm looking for a movie - what's your favourite one?","role":"user"},
    ], add_generation_prompt=True, return_tensors="pt",
).to("cuda")

outputs = model.generate(inputs, do_sample=False, max_new_tokens=256)

input_length = inputs.shape[1]
print(tokenizer.batch_decode(outputs[:, input_length:], skip_special_tokens=True)[0])

Output:

My favorite movie is "The Shawshank Redemption."

It's a powerful and inspiring story about hope, friendship, and redemption.
The performances by Tim Robbins and Morgan Freeman are exceptional,
and the film's themes and messages are timeless.

I highly recommend it to anyone who enjoys a well-crafted and emotionally engaging story.

Structured Output

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("prince-canuma/Damysus-2.7B-Chat")
model = AutoModelForCausalLM.from_pretrained("prince-canuma/Damysus-2.7B-Chat")

inputs = tokenizer.apply_chat_template(
    [
        {"content":"You are a Robot that ONLY outputs JSON. Use this structure: {'entities': [{'type':..., 'name':...}]}.","role":"system"},
        {"content":""""Extract the entities of type 'technology' and 'file_type' in JSON format from the following passage: AI is a transformative
                      force in document processing employing technologies such as 'Machine Learning (ML), Natural Language Processing (NLP) and
                      Optical Character Recognition (OCR) to understand, interpret, and summarize text. These technologies enhance accuracy,
                      increase efficiency, and allow you and your company to process high volumes of data in short amount of time.
                      For instance, you can easily extract key points and summarize a large PDF document (i.e., 500 pages) in just a few seconds.""",
        "role":"user"},
    ], add_generation_prompt=True, return_tensors="pt",
).to("cuda")

outputs = model.generate(inputs, do_sample=False, max_new_tokens=256)

input_length = inputs.shape[1]
print(tokenizer.batch_decode(outputs[:, input_length:], skip_special_tokens=True)[0])

Output:

{
  "entities": [
    {
      "type": "technology",
      "name": "Machine Learning (ML)"
    },
    {
      "type": "technology",
      "name": "Natural Language Processing (NLP)"
    },
    {
      "type": "technology",
      "name": "Optical Character Recognition (OCR)"
    },
    {
      "type": "file_type",
      "name": "PDF"
    },
  ]
}

Training Details

Training Data

I used SlimOrca dataset, a new curated subset of our OpenOrca data. In the course of this study, the SlimOrca dataset was used, representing a meticulously curated subset derived from the broader OpenOrca dataset. This release provides an efficient means of reaching performance on-par with using larger slices of the OpenOrca, while only including ~500k GPT-4 completions.

Subsequently, two distinct subsets were crafted, comprising 102,000 and 1,000 samples, denoted as:

• prince-canuma/SmallOrca
• prince-canuma/TinyOrca

Although experimentation was conducted with both datasets, optimal results were achieved through fine-tuning on a modest set of 200 samples. Notably, the investigation revealed that augmenting the training data beyond this threshold predominantly enhanced the model's proficiency in generating Chain-of-Thought responses. However, it is imperative to note that the preference for Chain-of-Thought responses may not be universally applicable. Particularly in scenarios like the RAG setup, succinct answers to prompts are often favored, especially for straightforward queries.

Training Procedure

Preprocessing

1. Convert dataset to chatML format
2. Remove all samples with more than 2048 tokens (Phi-2 context size)
3. Mask instructions (System and User) at training time.

LoRA Config

• lora_alpha: 128,
• lora_dropout: 0.05,
• r: 256,
• bias: "none",
• target_modules: "all-linear",
• task_type: "CAUSAL_LM",

Training Hyperparameters

• Training regime: bf16 mixed precision,
• max_steps: 100,
• per_device_train_batch_size: 2,
• gradient_accumulation_steps: 2,
• optim: "adamw_torch_fused",
• learning_rate: 2e-4,
• max_grad_norm: 0.3,
• warmup_ratio: 0.03,
• lr_scheduler_type: "constant",

Trainer

• max_seq_length: 1744,
• data_collator: DataCollatorForCompletionOnlyLM

Evaluation

We evaluate models on 7 key benchmarks using the Eleuther AI Language Model Evaluation Harness , a unified framework to test generative language models on a large number of different evaluation tasks.

• AI2 Reasoning Challenge (25-shot) - a set of grade-school science questions.
• HellaSwag (10-shot) - a test of commonsense inference, which is easy for humans (~95%) but challenging for SOTA models.
• MMLU (5-shot) - a test to measure a text model's multitask accuracy. The test covers 57 tasks including elementary mathematics, US history, computer science, law, and more.
• TruthfulQA (0-shot) - a test to measure a model's propensity to reproduce falsehoods commonly found online. Note: TruthfulQA is technically a 6-shot task in the Harness because each example is prepended with 6 Q/A pairs, even in the 0-shot setting.
• Winogrande (5-shot) - an adversarial and difficult Winograd benchmark at scale, for commonsense reasoning.
• GSM8k (5-shot) - diverse grade school math word problems to measure a model's ability to solve multi-step mathematical reasoning problems. For all these evaluations, a higher score is a better score. We chose these benchmarks as they test a variety of reasoning and general knowledge across a wide variety of fields in 0-shot and few-shot settings.

Read more here.

Results

Model	AVG	ARC	Hellaswag	MMLU	Truthful QA	Winogrande	GSM8K
NousResearch/Nous-Puffin-70B	64.91	67.41	87.37	69.77	46.77	83.9	34.27
TheBloke/Llama-2-70B-fp16	64.52	67.32	87.33	69.83	44.92	83.74	33.97
NousResearch/Yarn-Mistral-7B-64k	59.63	59.9	82.51	62.96	41.86	77.27	33.28
Qwen1.5-4B-Chat	46.79	43.26	69.73	55.55	44.79	64.96	2.43
Microsoft/phi-2	61.33	61.09	75.11	58.11	44.47	74.35	54.81
Damysus-2.7B-Chat (Ours)	60.49	59.81	74.52	56.33	46.74	75.06	50.64

Technical Specifications

Compute Infrastructure

• Modal Labs

Hardware

• OS: Linux
• GPU: A10G

Libraries

• TRL
• Transformers
• PEFT
• Datasets
• Accelerate
• torch
• Wandb
• Bitsandbytes
• Plotly

Future work

I plan to explore the following tuning setups:

• Function calling
• DPO

Citation

BibTeX:

@misc{Damysus-2.7B-Chat,
      title={Damysus-2.7B-Chat} , 
      author={Prince Canuma},
      year={2024},
}

@misc{SlimOrca,
  title = {SlimOrca: An Open Dataset of GPT-4 Augmented FLAN Reasoning Traces, with Verification},
  author = {Wing Lian and Guan Wang and Bleys Goodson and Eugene Pentland and Austin Cook and Chanvichet Vong and "Teknium"},
  year = {2023},
  publisher = {HuggingFace},
  url = {https://https://huggingface.co/Open-Orca/SlimOrca}
}

@misc{open-llm-leaderboard,
  author = {Edward Beeching and Clémentine Fourrier and Nathan Habib and Sheon Han and Nathan Lambert and Nazneen Rajani and Omar Sanseviero and Lewis Tunstall and Thomas Wolf},
  title = {Open LLM Leaderboard},
  year = {2023},
  publisher = {Hugging Face},
  howpublished = "\url{https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard}"
}