README.md

---
language:
- en
license: llama3
tags:
- fireplace
- valiant
- valiant-labs
- llama
- llama-3
- llama-3-instruct
- llama-3-instruct-70b
- 70b
- function-calling
- conversational
- chat
- instruct
pipeline_tag: text-generation
model_type: llama
model-index:
- name: Llama3-70B-Fireplace
  results:
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: IFEval (0-Shot)
      type: HuggingFaceH4/ifeval
      args:
        num_few_shot: 0
    metrics:
    - type: inst_level_strict_acc and prompt_level_strict_acc
      value: 77.74
      name: strict accuracy
    source:
      url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=ValiantLabs/Llama3-70B-Fireplace
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: BBH (3-Shot)
      type: BBH
      args:
        num_few_shot: 3
    metrics:
    - type: acc_norm
      value: 49.56
      name: normalized accuracy
    source:
      url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=ValiantLabs/Llama3-70B-Fireplace
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: MATH Lvl 5 (4-Shot)
      type: hendrycks/competition_math
      args:
        num_few_shot: 4
    metrics:
    - type: exact_match
      value: 19.64
      name: exact match
    source:
      url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=ValiantLabs/Llama3-70B-Fireplace
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: GPQA (0-shot)
      type: Idavidrein/gpqa
      args:
        num_few_shot: 0
    metrics:
    - type: acc_norm
      value: 13.98
      name: acc_norm
    source:
      url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=ValiantLabs/Llama3-70B-Fireplace
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: MuSR (0-shot)
      type: TAUR-Lab/MuSR
      args:
        num_few_shot: 0
    metrics:
    - type: acc_norm
      value: 16.77
      name: acc_norm
    source:
      url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=ValiantLabs/Llama3-70B-Fireplace
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: MMLU-PRO (5-shot)
      type: TIGER-Lab/MMLU-Pro
      config: main
      split: test
      args:
        num_few_shot: 5
    metrics:
    - type: acc
      value: 43.25
      name: accuracy
    source:
      url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=ValiantLabs/Llama3-70B-Fireplace
      name: Open LLM Leaderboard
---


![image/jpeg](https://cdn-uploads.huggingface.co/production/uploads/64f267a8a4f79a118e0fcc89/qg49GOlx8zogDOrMTnb89.jpeg)


Fireplace is a function-calling model for Llama 3 70b Instruct.
  - combines function-calling abilities with a high-performance, versatile chat model
  - system message function-calling utilizing the Llama 3 Instruct format

This version of Fireplace focuses on combining chat-instruct and system-message function-calling only.

**We've just released Fireplace 2 for Llama 3.1 8b**, which includes inline function calls as one of several technical skills (including JSON, SQL, and more!)
[Try it today!](https://huggingface.co/ValiantLabs/Llama3.1-8B-Fireplace2)

## Version

This is the **2024-05-09** release of Fireplace for Llama 3 70b.

We've also released Fireplace 2 for Llama 3.1 8b and we're working on more Fireplace releases to come :)

## Prompting Guide
Fireplace uses the [Llama 3 Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct) prompt format:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>{{ system_prompt }}<|eot_id|><|start_header_id|>user<|end_header_id|>{{ user_msg_1 }}<|eot_id|><|start_header_id|>assistant<|end_header_id|>{{ model_answer_1 }}<|eot_id|>

Example input for function calling:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n You are Fireplace, an expert code assistant with access to the following functions. Use them if required - { "name": "calculate_tip", "description": "Calculate the tip amount for a bill", "parameters": { "type": "object", "properties": { "bill_amount": { "type": "number", "description": "The total amount of the bill" }, "tip_percentage": { "type": "number", "description": "The percentage of tip to be given" } }, "required": [ "bill_amount", "tip_percentage" ] } } { "name": "check_website_availability", "description": "Check the availability of a website", "parameters": { "type": "object", "properties": { "url": { "type": "string", "description": "The URL of the website" } }, "required": [ "url" ] } } <|eot_id|><|start_header_id|>user<|end_header_id|>\n\nHi, I need help with calculating a tip. My bill is $100 and I want to leave a 30% tip. <|eot_id|><|start_header_id|>assistant<|end_header_id|>

For assistant handling of function responses, deliver them in a new user message:

<|start_header_id|>user<|end_header_id|>\n\n FUNCTION RESPONSE: {"status": "success", "message": "Email has been sent successfully"} <|eot_id|>

## WARNING: text-generation-webui

When using Llama 3 Instruct models (including Fireplace) with [text-generation-webui](https://github.com/oobabooga/text-generation-webui/tree/main) note that a current bug in webui can result in incorrect reading of the model's ending tokens, causing unfinished outputs and incorrect structure.

For a [temporary workaround](https://github.com/oobabooga/text-generation-webui/issues/5885) if you encounter this issue, edit Fireplace's tokenizer_config file as indicated:

from "eos_token": "<|end_of_text|>",

to "eos_token": "<|eot_id|>",

## The Model
Fireplace is built on top of Llama 3 70b Instruct, the highest performance open-source model currently available.

This version of Fireplace uses the [glaiveai/glaive-function-calling-v2](https://huggingface.co/datasets/glaiveai/glaive-function-calling-v2) dataset converted to Llama 3 Instruct format.

# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_ValiantLabs__Llama3-70B-Fireplace)

|      Metric       |Value|
|-------------------|----:|
|Avg.               |36.82|
|IFEval (0-Shot)    |77.74|
|BBH (3-Shot)       |49.56|
|MATH Lvl 5 (4-Shot)|19.64|
|GPQA (0-shot)      |13.98|
|MuSR (0-shot)      |16.77|
|MMLU-PRO (5-shot)  |43.25|



![image/jpeg](https://cdn-uploads.huggingface.co/production/uploads/63444f2687964b331809eb55/VCJ8Fmefd8cdVhXSSxJiD.jpeg)


Fireplace is created by [Valiant Labs.](http://valiantlabs.ca/)

[Check out our HuggingFace page for Shining Valiant 2 and our other models!](https://huggingface.co/ValiantLabs)

We care about open source.
For everyone to use.

We encourage others to finetune further from our models.