|
--- |
|
license: apache-2.0 |
|
datasets: |
|
- amphora/QwQ-LongCoT-130K |
|
language: |
|
- en |
|
base_model: |
|
- prithivMLmods/QwQ-LCoT-7B-Instruct |
|
pipeline_tag: text-generation |
|
library_name: transformers |
|
tags: |
|
- QwQ |
|
- Adapter |
|
- safetensors |
|
- Qwen2.5 |
|
- text-generation-inference |
|
--- |
|
<pre align="center"> |
|
________ ________ _____ ___. |
|
\_____ \ __ _ __\_____ \ / | | \_ |__ |
|
/ / \ \ \ \/ \/ / / / \ \ ______ / | |_ | __ \ |
|
/ \_/. \ \ / / \_/. \ /_____/ / ^ / | \_\ \ |
|
\_____\ \_/ \/\_/ \_____\ \_/ \____ | |___ / |
|
\__> \__> |__| \/ |
|
</pre> |
|
|
|
The **QwQ-4B-Instruct** is a lightweight and efficient fine-tuned language model for instruction-following tasks and reasoning. It is based on a quantized version of the **Qwen2.5-7B** model, optimized for inference speed and reduced memory consumption, while retaining robust capabilities for complex tasks. |
|
|
|
With its robust natural language processing capabilities, **QwQ-4B-Instruct** excels in generating step-by-step solutions, creative content, and logical analyses. Its architecture integrates advanced understanding of both structured and unstructured data, ensuring precise text generation aligned with user inputs. |
|
|
|
- Significantly **more knowledge** and has greatly improved capabilities in **coding** and **mathematics**, thanks to our specialized expert models in these domains. |
|
- Significant improvements in **instruction following**, **generating long texts** (over 8K tokens), **understanding structured data** (e.g, tables), and **generating structured outputs** especially JSON. **More resilient to the diversity of system prompts**, enhancing role-play implementation and condition-setting for chatbots. |
|
- **Long-context Support** up to 128K tokens and can generate up to 8K tokens. |
|
- **Multilingual support** for over 29 languages, including Chinese, English, French, Spanish, Portuguese, German, Italian, Russian, Japanese, Korean, Vietnamese, Thai, Arabic, and more. |
|
|
|
# **Demo Start** |
|
|
|
Here provides a code snippet with `apply_chat_template` to show you how to load the tokenizer and model and how to generate contents. |
|
|
|
```python |
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
|
model_name = "prithivMLmods/QwQ-4B-Instruct" |
|
|
|
model = AutoModelForCausalLM.from_pretrained( |
|
model_name, |
|
torch_dtype="auto", |
|
device_map="auto" |
|
) |
|
tokenizer = AutoTokenizer.from_pretrained(model_name) |
|
|
|
prompt = "Give me a short introduction to large language model." |
|
messages = [ |
|
{"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, |
|
{"role": "user", "content": prompt} |
|
] |
|
text = tokenizer.apply_chat_template( |
|
messages, |
|
tokenize=False, |
|
add_generation_prompt=True |
|
) |
|
model_inputs = tokenizer([text], return_tensors="pt").to(model.device) |
|
|
|
generated_ids = model.generate( |
|
**model_inputs, |
|
max_new_tokens=512 |
|
) |
|
generated_ids = [ |
|
output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids) |
|
] |
|
|
|
response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0] |
|
``` |
|
# **Run with Ollama [Ollama Run]** |
|
|
|
Ollama makes running machine learning models simple and efficient. Follow these steps to set up and run your GGUF models quickly. |
|
|
|
## Quick Start: Step-by-Step Guide |
|
|
|
| Step | Description | Command / Instructions | |
|
|------|-------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------| |
|
| 1 | **Install Ollama 🦙** | Download Ollama from [https://ollama.com/download](https://ollama.com/download) and install it on your system. | |
|
| 2 | **Create Your Model File** | - Create a file named after your model, e.g., `metallama`. | |
|
| | | - Add the following line to specify the base model: | |
|
| | | ```bash | |
|
| | | FROM Llama-3.2-1B.F16.gguf | |
|
| | | ``` | |
|
| | | - Ensure the base model file is in the same directory. | |
|
| 3 | **Create and Patch the Model** | Run the following commands to create and verify your model: | |
|
| | | ```bash | |
|
| | | ollama create metallama -f ./metallama | |
|
| | | ollama list | |
|
| | | ``` | |
|
| 4 | **Run the Model** | Use the following command to start your model: | |
|
| | | ```bash | |
|
| | | ollama run metallama | |
|
| | | ``` | |
|
| 5 | **Interact with the Model** | Once the model is running, interact with it: | |
|
| | | ```plaintext | |
|
| | | >>> Tell me about Space X. | |
|
| | | Space X, the private aerospace company founded by Elon Musk, is revolutionizing space exploration... | |
|
| | | ``` | |
|
|
|
## Conclusion |
|
With Ollama, running and interacting with models is seamless. Start experimenting today! |
