Triangle104/FuseChat-Qwen-2.5-7B-Instruct-Q8_0-GGUF

This model was converted to GGUF format from FuseAI/FuseChat-Qwen-2.5-7B-Instruct using llama.cpp via the ggml.ai's GGUF-my-repo space. Refer to the original model card for more details on the model.

---

transferring capabilities from source LLMs to a target LLM. First, during dataset construction, we sample N responses from each of the source LLMs and annotate these responses using an external reward model. Second, in the supervised fine-tuning (SFT) stage, we fine-tune the target model using the best responses, which not only enhances the target model's capabilities but also helps mitigate the distributional gap between the source and target models. Finally, in the direct preference optimization (DPO) stage, we optimize the target model by using the best and worst responses from the source models as preference pairs, further enhancing the target model's performance. The complete pipeline will be detailed in the following paragraph.

    Dataset






    


    Prompt Selection

Our datasets were designed to enhance model's instruction following, general conversation, mathematics, coding, and Chinese-language capabilities. We selected data from open-source community datasets, applying targeted filtering and preprocessing. Key datasets and filtering criteria included:

Instruction Following & General Conversation: Sourced from UltraFeedback, Magpie-Pro-DPO-100K-v0.1, and HelpSteer2, excluding code and math data. Mathematics: Selected from OpenMathInstruct-2, with nearly 60,000 unique samples. Coding: Curated from leetcode and self-oss-instruct-sc2-exec-filter-50k, retaining prompts with test cases. Chinese Language: Integrated alpaca_gpt4_zh and Magpie-Qwen2-Pro-200K-Chinese, filtering out code and math prompts to retain approximately 10,000 high-quality samples.

    Response Sampling

For each dataset's prompts, we synthesized responses mainly from four different series of source models, specifically Gemma-2-27b-It, Mistral-Large-Instruct-2407, Qwen-2.5-72B-Instruct, and Llama-3.1-70B-Instruct.

Instruction Following & General Conversation: We sampled each prompt five times from all the source models. Mathematics: We retained the responses generated by Llama-3.1-405B-Instruct from the original dataset (OpenMathInstruct-2) and additionally sampled responses using Qwen-2.5-Math-72B-Instruct. Coding: We sampled each prompt eight times for all source models. Chinese Language: We included single response sampled exclusively from Qwen-2.5-72B-Instruct.

---

Use with llama.cpp

Install llama.cpp through brew (works on Mac and Linux)

brew install llama.cpp

Invoke the llama.cpp server or the CLI.

CLI:

llama-cli --hf-repo Triangle104/FuseChat-Qwen-2.5-7B-Instruct-Q8_0-GGUF --hf-file fusechat-qwen-2.5-7b-instruct-q8_0.gguf -p "The meaning to life and the universe is"

Server:

llama-server --hf-repo Triangle104/FuseChat-Qwen-2.5-7B-Instruct-Q8_0-GGUF --hf-file fusechat-qwen-2.5-7b-instruct-q8_0.gguf -c 2048

Note: You can also use this checkpoint directly through the usage steps listed in the Llama.cpp repo as well.

Step 1: Clone llama.cpp from GitHub.

git clone https://github.com/ggerganov/llama.cpp

Step 2: Move into the llama.cpp folder and build it with LLAMA_CURL=1 flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux).

cd llama.cpp && LLAMA_CURL=1 make

Step 3: Run inference through the main binary.

./llama-cli --hf-repo Triangle104/FuseChat-Qwen-2.5-7B-Instruct-Q8_0-GGUF --hf-file fusechat-qwen-2.5-7b-instruct-q8_0.gguf -p "The meaning to life and the universe is"

or

./llama-server --hf-repo Triangle104/FuseChat-Qwen-2.5-7B-Instruct-Q8_0-GGUF --hf-file fusechat-qwen-2.5-7b-instruct-q8_0.gguf -c 2048