README.md

metadata

language:
  - ko
  - en
license: llama2
library_name: transformers
model-index:
  - name: mnsim-dpo-peftmerged-2-eos
    results:
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: AI2 Reasoning Challenge (25-Shot)
          type: ai2_arc
          config: ARC-Challenge
          split: test
          args:
            num_few_shot: 25
        metrics:
          - type: acc_norm
            value: 55.63
            name: normalized accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=msy127/mnsim-dpo-peftmerged-2-eos
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: HellaSwag (10-Shot)
          type: hellaswag
          split: validation
          args:
            num_few_shot: 10
        metrics:
          - type: acc_norm
            value: 77.82
            name: normalized accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=msy127/mnsim-dpo-peftmerged-2-eos
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: MMLU (5-Shot)
          type: cais/mmlu
          config: all
          split: test
          args:
            num_few_shot: 5
        metrics:
          - type: acc
            value: 51.25
            name: accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=msy127/mnsim-dpo-peftmerged-2-eos
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: TruthfulQA (0-shot)
          type: truthful_qa
          config: multiple_choice
          split: validation
          args:
            num_few_shot: 0
        metrics:
          - type: mc2
            value: 46.37
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=msy127/mnsim-dpo-peftmerged-2-eos
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: Winogrande (5-shot)
          type: winogrande
          config: winogrande_xl
          split: validation
          args:
            num_few_shot: 5
        metrics:
          - type: acc
            value: 76.24
            name: accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=msy127/mnsim-dpo-peftmerged-2-eos
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: GSM8k (5-shot)
          type: gsm8k
          config: main
          split: test
          args:
            num_few_shot: 5
        metrics:
          - type: acc
            value: 16.91
            name: accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=msy127/mnsim-dpo-peftmerged-2-eos
          name: Open LLM Leaderboard

---

license: llama2 language: - ko - en library_name: transformers base_model: mncai/llama2-13b-dpo-v7 pipeline_tag: text-generation

mnsim-dpo-peftmerged-2-eos

Our Team

Research & Engineering	Product Management
David Sohn	David Sohn

Model Details

Base Model

mncai/llama2-13b-dpo-v7

Trained On

• OS: Ubuntu 22.04
• GPU: A100 40GB 1ea
• transformers: v4.35.2

Instruction format

It follows Custom format.

E.g.

text = """\
<s>
<|user|>
건강한 식습관을 만들기 위해서는 어떻게 하는것이 좋을까요?
<|assistant|>
"""

Implementation Code

This model contains the chat_template instruction format.
You can use the code below.

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="msy127/mnsim-dpo-peftmerged-2-eos")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("msy127/mnsim-dpo-peftmerged-2-eos")
model = AutoModelForCausalLM.from_pretrained("msy127/mnsim-dpo-peftmerged-2-eos")

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	54.04
AI2 Reasoning Challenge (25-Shot)	55.63
HellaSwag (10-Shot)	77.82
MMLU (5-Shot)	51.25
TruthfulQA (0-shot)	46.37
Winogrande (5-shot)	76.24
GSM8k (5-shot)	16.91