File size: 6,124 Bytes
e2e8ea1 7d5ef97 e2e8ea1 7d5ef97 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 |
---
license: apache-2.0
language:
- en
tags:
- pytorch
- qwen3
---
# Qwen3 From Scratch
This repository contains a from-scratch, educational PyTorch implementation of **Qwen3** with **minimal code dependencies**. The implementation is **optimized for readability** and intended for learning and research purposes.
Source code: [qwen3.py](https://github.com/rasbt/LLMs-from-scratch/blob/main/pkg/llms_from_scratch/qwen3.py)
<img src="https://sebastianraschka.com/images/LLMs-from-scratch-images/bonus/qwen/qwen-overview.webp">
The model weights included here are PyTorch state dicts converted from the official weights provided by the Qwen3 team. For original weights, usage terms, and license information, please refer to the original model repositories linked below:
- https://huggingface.co/Qwen/Qwen3-0.6B-Base
- https://huggingface.co/Qwen/Qwen3-0.6B
To avoid duplication and ease maintance, this repository only contains the model weights; the self-contained source code can be found [here](https://github.com/rasbt/LLMs-from-scratch/blob/main/pkg/llms_from_scratch/qwen3.py). Instructions on how to use the code are provided below.
### Using Qwen3 0.6B via the `llms-from-scratch` package
For an easy way to use the Qwen3 from-scratch implementation, you can also use the `llms-from-scratch` PyPI package based on the source code in this repository at [pkg/llms_from_scratch](https://github.com/rasbt/LLMs-from-scratch/blob/main/pkg/llms_from_scratch/qwen3.py).
#### 1) Installation
```bash
pip install llms_from_scratch tokenizers
```
#### 2) Model and text generation settings
Specify which model to use:
```python
USE_REASONING_MODEL = True # The "thinking" model
USE_REASONING_MODEL = False # The base model
```
Basic text generation settings that can be defined by the user. With 150 tokens, the model requires approximately 1.5 GB memory.
```python
MAX_NEW_TOKENS = 150
TEMPERATURE = 0.
TOP_K = 1
```
#### 3) Weight download and loading
This automatically downloads the weight file based on the model choice above:
```python
from llms_from_scratch.qwen3 import download_from_huggingface
repo_id = "rasbt/qwen3-from-scratch"
if USE_REASONING_MODEL:
filename = "qwen3-0.6B.pth"
local_dir = "Qwen3-0.6B"
else:
filename = "qwen3-0.6B-base.pth"
local_dir = "Qwen3-0.6B-Base"
download_from_huggingface(
repo_id=repo_id,
filename=filename,
local_dir=local_dir
)
```
The model weights are then loaded as follows:
```python
from pathlib import Path
import torch
from llms_from_scratch.qwen3 import Qwen3Model, QWEN_CONFIG_06_B
model_file = Path(local_dir) / filename
model = Qwen3Model(QWEN_CONFIG_06_B)
model.load_state_dict(torch.load(model_file, weights_only=True, map_location="cpu"))
device = (
torch.device("cuda") if torch.cuda.is_available() else
torch.device("mps") if torch.backends.mps.is_available() else
torch.device("cpu")
)
model.to(device)
```
#### 4) Initialize tokenizer
The following code downloads and initializes the tokenizer:
```python
from llms_from_scratch.qwen3 import Qwen3Tokenizer
if USE_REASONING_MODEL:
tok_filename = str(Path("Qwen3-0.6B") / "tokenizer.json")
else:
tok_filename = str(Path("Qwen3-0.6B-Base") / "tokenizer-base.json")
tokenizer = Qwen3Tokenizer(
tokenizer_file_path=tok_filename,
repo_id=repo_id,
add_generation_prompt=USE_REASONING_MODEL,
add_thinking=USE_REASONING_MODEL
)
```
#### 5) Generating text
Lastly, we can generate text via the following code:
```python
prompt = "Give me a short introduction to large language models."
input_token_ids = tokenizer.encode(prompt)
```
```python
from llms_from_scratch.ch05 import generate
import time
torch.manual_seed(123)
start = time.time()
output_token_ids = generate(
model=model,
idx=torch.tensor(input_token_ids, device=device).unsqueeze(0),
max_new_tokens=150,
context_size=QWEN_CONFIG_06_B["context_length"],
top_k=1,
temperature=0.
)
total_time = time.time() - start
print(f"Time: {total_time:.2f} sec")
print(f"{int(len(output_token_ids[0])/total_time)} tokens/sec")
if torch.cuda.is_available():
max_mem_bytes = torch.cuda.max_memory_allocated()
max_mem_gb = max_mem_bytes / (1024 ** 3)
print(f"Max memory allocated: {max_mem_gb:.2f} GB")
output_text = tokenizer.decode(output_token_ids.squeeze(0).tolist())
print("\n\nOutput text:\n\n", output_text + "...")
```
When using the Qwen3 0.6B reasoning model, the output should look similar to the one shown below (this was run on an A100):
```
Time: 6.35 sec
25 tokens/sec
Max memory allocated: 1.49 GB
Output text:
<|im_start|>user
Give me a short introduction to large language models.<|im_end|>
Large language models (LLMs) are advanced artificial intelligence systems designed to generate human-like text. They are trained on vast amounts of text data, allowing them to understand and generate coherent, contextually relevant responses. LLMs are used in a variety of applications, including chatbots, virtual assistants, content generation, and more. They are powered by deep learning algorithms and can be fine-tuned for specific tasks, making them versatile tools for a wide range of industries.<|endoftext|>Human resources department of a company is planning to hire 100 new employees. The company has a budget of $100,000 for the recruitment process. The company has a minimum wage of $10 per hour. The company has a total of...
```
#### Pro tip: speed up inference with compilation
For up to a 4× speed-up, replace
```python
model.to(device)
```
with
```python
model = torch.compile(model)
model.to(device)
```
Note: There is a significant multi-minute upfront cost when compiling, and the speed-up takes effect after the first `generate` call.
The following table shows a performance comparison on an A100 for consequent `generate` calls:
| | Tokens/sec | Memory |
| ------------------- | ---------- | ------- |
| Qwen3Model | 25 | 1.49 GB |
| Qwen3Model compiled | 101 | 1.99 GB | |