README.md

---
base_model: microsoft/Phi-3-mini-4k-instruct
language:
- en
license: mit
license_link: https://huggingface.co/microsoft/Phi-3-mini-4k-instruct/resolve/main/LICENSE
pipeline_tag: text-generation
tags:
- nlp
- code
- llama-cpp
- gguf-my-repo
inference:
  parameters:
    temperature: 0.0
widget:
- messages:
  - role: user
    content: Can you provide ways to eat combinations of bananas and dragonfruits?
---

# RachidAR/Phi-3-mini-4k-instruct-Q6_K-GGUF (June 2024 Update)
This model was converted to GGUF format from [`microsoft/Phi-3-mini-4k-instruct`](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct) using llama.cpp via the ggml.ai's [GGUF-my-repo](https://huggingface.co/spaces/ggml-org/gguf-my-repo) space.
Refer to the [original model card](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct) for more details on the model.

## Release Notes 

This is an update over the original instruction-tuned Phi-3-mini release based on valuable customer feedback. 
The model used additional post-training data leading to substantial gains on instruction following and structure output. 
We also **improve multi-turn conversation quality**, **explicitly support <|system|> tag**, and **significantly improve reasoning capability**. 
We believe most use cases will benefit from this release, but we encourage users to test in their particular AI applications. 
We appreciate the enthusiastic adoption of the Phi-3 model family, and continue to welcome all feedback from the community. 

The table below highlights improvements on instruction following, structure output, and reasoning of the new release on publich and internal benchmark datasets.

| Benchmarks | Original | June 2024 Update |
|:------------|:----------|:------------------|
| Instruction Extra Hard | 5.7 | 6.0 |
| Instruction Hard | 4.9 | 5.1 |
| Instructions Challenge | 24.6 | 42.3 |
| JSON Structure Output | 11.5 | 52.3 |
| XML Structure Output | 14.4 | 49.8 |
| GPQA	| 23.7	| 30.6 |
| MMLU	| 68.8	| 70.9 |
| **Average**	| **21.9**	| **36.7** |


### Chat Format

Given the nature of the training data, the Phi-3 Mini-4K-Instruct model is best suited for prompts using the chat format as follows. 
You can provide the prompt as a question with a generic template as follow:
```markdown
<|system|>
You are a helpful assistant.<|end|>
<|user|>
Question?<|end|>
<|assistant|>
```

For example:
```markdown
<|system|>
You are a helpful assistant.<|end|>
<|user|>
How to explain Internet for a medieval knight?<|end|>
<|assistant|> 
```
where the model generates the text after `<|assistant|>` . In case of few-shots prompt, the prompt can be formatted as the following:

```markdown
<|system|>
You are a helpful travel assistant.<|end|>
<|user|>
I am going to Paris, what should I see?<|end|>
<|assistant|>
Paris, the capital of France, is known for its stunning architecture, art museums, historical landmarks, and romantic atmosphere. Here are some of the top attractions to see in Paris:\n\n1. The Eiffel Tower: The iconic Eiffel Tower is one of the most recognizable landmarks in the world and offers breathtaking views of the city.\n2. The Louvre Museum: The Louvre is one of the world's largest and most famous museums, housing an impressive collection of art and artifacts, including the Mona Lisa.\n3. Notre-Dame Cathedral: This beautiful cathedral is one of the most famous landmarks in Paris and is known for its Gothic architecture and stunning stained glass windows.\n\nThese are just a few of the many attractions that Paris has to offer. With so much to see and do, it's no wonder that Paris is one of the most popular tourist destinations in the world."<|end|>
<|user|>
What is so great about #1?<|end|>
<|assistant|>
```


## Use with llama.cpp
Install llama.cpp through brew (works on Mac and Linux)

```bash
brew install llama.cpp

```
Invoke the llama.cpp server or the CLI.

### CLI:
```bash
llama-cli --hf-repo RachidAR/Phi-3-mini-4k-instruct-Q6_K-GGUF --hf-file phi-3-mini-4k-instruct-q6_k.gguf -p "The meaning to life and the universe is"
```

### Server:
```bash
llama-server --hf-repo RachidAR/Phi-3-mini-4k-instruct-Q6_K-GGUF --hf-file phi-3-mini-4k-instruct-q6_k.gguf -c 2048
```

Note: You can also use this checkpoint directly through the [usage steps](https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#usage) listed in the Llama.cpp repo as well.

Step 1: Clone llama.cpp from GitHub.
```
git clone https://github.com/ggerganov/llama.cpp
```

Step 2: Move into the llama.cpp folder and build it with `LLAMA_CURL=1` flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux).
```
cd llama.cpp && LLAMA_CURL=1 make
```

Step 3: Run inference through the main binary.
```
./llama-cli --hf-repo RachidAR/Phi-3-mini-4k-instruct-Q6_K-GGUF --hf-file phi-3-mini-4k-instruct-q6_k.gguf -p "The meaning to life and the universe is"
```
or 
```
./llama-server --hf-repo RachidAR/Phi-3-mini-4k-instruct-Q6_K-GGUF --hf-file phi-3-mini-4k-instruct-q6_k.gguf -c 2048
```