Qwen2.5 0.5B
494M parameters · 32k context · Apache License 2.0 · released 2024-09-16
chat
In short
Qwen2.5 0.5B is an open model from Alibaba Qwen with 494M parameters. At Q8_0, Qwen2.5 0.5B needs about 1 GB of memory at 8k context and 2 GB at 32k, counting the weights, the KV cache and runtime buffers. The cheapest hardware in the US that runs Qwen2.5 0.5B fully in memory at 32k context is the NVIDIA Tesla P40, $345 at Amazon, at about 230 tok/s (estimated).
Updated . Speeds are estimates; how they are worked out.
What to run it on in the US
Fully in memory at 32k context, priced at retailers today.
Cheapest
NVIDIA Tesla P40230tok/sest.Q8_0
$345
Best value
NVIDIA GeForce RTX 3080 10GB505tok/sest.Q8_0
$469
Fastest
NVIDIA GeForce RTX 50901190tok/sest.Q8_0
$4,921
Mac
Mac mini M4 16GB84tok/sest.Q8_0
$749
All hardware that runs it
Everything we price, best value first. Two cards split the model by layers.
| Hardware | Quant | Fits | Decode | Price | tok/s per 1,000 |
|---|---|---|---|---|---|
| NVIDIA Tesla P40 | Q8_0 | fits | 230 tok/s | $345 | 665.8 |
| Intel Arc B580 | Q8_0 | fits | 244 tok/s | $346 | 705.8 |
| NVIDIA GeForce RTX 3060 12GB | Q8_0 | fits | 239 tok/s | $360 | 663.9 |
| Intel Arc A770 16GB | Q8_0 | fits | 300 tok/s | $400 | 749.8 |
| NVIDIA GeForce RTX 3080 10GB | Q8_0 | fits | 505 tok/s | $469 | 1075.1 |
| AMD Radeon RX 7600 XT | Q8_0 | fits | 170 tok/s | $480 | 353.3 |
| AMD Radeon RX 9060 XT 16GB | Q8_0 | fits | 189 tok/s | $520 | 362.5 |
| AMD Radeon RX 7800 XT | Q8_0 | fits | 368 tok/s | $550 | 668.4 |
| NVIDIA GeForce RTX 5060 Ti 16GB | Q8_0 | fits | 298 tok/s | $618 | 481.4 |
| Intel Arc Pro B60 | Q8_0 | fits | 244 tok/s | $650 | 375.7 |
| NVIDIA GeForce RTX 4070 Super | Q8_0 | fits | 335 tok/s | $699 | 478.7 |
| Mac mini M4 16GB | Q8_0 | fits | 84 tok/s | $749 | 111.5 |
| NVIDIA GeForce RTX 5070 | Q8_0 | fits | 446 tok/s | $750 | 594.9 |
| NVIDIA GeForce RTX 4060 Ti 16GB | Q8_0 | fits | 191 tok/s | $770 | 248.3 |
| AMD Radeon RX 7900 XT | Q8_0 | fits | 471 tok/s | $800 | 589.0 |
| AMD Radeon RX 9070 XT | Q8_0 | fits | 377 tok/s | $815 | 462.6 |
| NVIDIA GeForce RTX 4070 Ti Super | Q8_0 | fits | 446 tok/s | $984 | 453.7 |
| NVIDIA GeForce RTX 5070 Ti | Q8_0 | fits | 595 tok/s | $995 | 598.0 |
| NVIDIA GeForce RTX 4080 Super | Q8_0 | fits | 489 tok/s | $1,199 | 407.6 |
| Mac mini M4 32GB | Q8_0 | fits | 84 tok/s | $1,420 | 58.8 |
| AMD Radeon RX 7900 XTX | Q8_0 | fits | 566 tok/s | $1,533 | 368.8 |
| NVIDIA GeForce RTX 3090 | Q8_0 | fits | 622 tok/s | $1,600 | 388.4 |
| NVIDIA GeForce RTX 5080 | Q8_0 | fits | 637 tok/s | $1,600 | 398.4 |
| AMD Radeon AI PRO R9700 | Q8_0 | fits | 377 tok/s | $1,700 | 221.8 |
| NVIDIA GeForce RTX 3090 Ti | Q8_0 | fits | 669 tok/s | $2,060 | 324.9 |
| NVIDIA GeForce RTX 4090 | Q8_0 | fits | 669 tok/s | $2,073 | 322.8 |
| Mac mini M4 Pro 48GB | Q8_0 | fits | 190 tok/s | $2,149 | 88.4 |
| Mac mini M4 Pro 64GB | Q8_0 | fits | 190 tok/s | $2,900 | 65.5 |
| Mac Studio M4 Max 64GB | Q8_0 | fits | 380 tok/s | $3,400 | 111.8 |
| Framework Desktop Ryzen AI Max+ 395 128GB | Q8_0 | fits | 151 tok/s | $3,449 | 43.7 |
| GMKtec EVO-X2 Ryzen AI Max+ 395 128GB | Q8_0 | fits | 151 tok/s | $3,650 | 41.3 |
| Mac Studio M3 Ultra 96GB | Q8_0 | fits | 570 tok/s | $4,399 | 129.6 |
| NVIDIA GeForce RTX 5090 | Q8_0 | fits | 1190 tok/s | $4,921 | 241.8 |
| Mac Studio M4 Max 128GB | Q8_0 | fits | 380 tok/s | $4,999 | 76.0 |
| ASUS Ascent GX10 128GB | Q8_0 | fits | 181 tok/s | $6,100 | 29.7 |
| NVIDIA DGX Spark 128GB | Q8_0 | fits | 181 tok/s | $7,399 | 24.5 |
| NVIDIA RTX PRO 6000 Blackwell | Q8_0 | fits | 1190 tok/s | $7,997 | 148.8 |
| Mac Studio M3 Ultra 256GB | Q8_0 | fits | 570 tok/s | $8,499 | 67.1 |
| Mac Studio M3 Ultra 512GB | Q8_0 | fits | 570 tok/s | $23,000 | 24.8 |
| Framework Desktop Ryzen AI Max+ 395 64GB | Q8_0 | fits | 151 tok/s | – | – |
Memory needed
Weights, KV cache and runtime buffers at each context length (KV cache at fp16).
| Quant | File | @8k | @32k |
|---|---|---|---|
| Q8_0gguf | 0.5 GB | 1.3 GB | 1.8 GB |
Run it
Copy a command for the runtime you use.
- Q8_0
- Ollama
ollama run hf.co/bartowski/Qwen2.5-0.5B-Instruct-GGUF:Q8_0 - LM Studio
lms get bartowski/Qwen2.5-0.5B-Instruct-GGUF@q8_0 - llama.cpp
llama-server -hf bartowski/Qwen2.5-0.5B-Instruct-GGUF:Q8_0
- Ollama
Notes
The card says the full context is 32,768 tokens and generation is up to 8192 tokens. Needs transformers 4.37.0 or later, because older versions raise KeyError: 'qwen2'.
Questions
- How much VRAM does Qwen2.5 0.5B need?
- At Q8_0, Qwen2.5 0.5B needs about 1 GB of memory at 8k context and 2 GB at 32k, counting the weights, the KV cache and runtime buffers.
- What is the best GPU for running Qwen2.5 0.5B locally?
- The cheapest hardware in the US that runs Qwen2.5 0.5B fully in memory at 32k context is the NVIDIA Tesla P40, $345 at Amazon, at about 230 tok/s (estimated). The best value is the NVIDIA GeForce RTX 3080 10GB (about 505 tok/s for $469). The fastest we price is the NVIDIA GeForce RTX 5090, at about 1190 tok/s.
- Can I run Qwen2.5 0.5B on a 16 GB GPU?
- Yes: at Q8_0 it fits fully in 16 GB at 32k context. On the NVIDIA GeForce RTX 5080 it runs at about 637 tok/s (estimated).
- Can I run Qwen2.5 0.5B on a 24 GB GPU?
- Yes: at Q8_0 it fits fully in 24 GB at 32k context. On the NVIDIA GeForce RTX 3090 Ti it runs at about 669 tok/s (estimated).
- How do I run Qwen2.5 0.5B locally?
- With Ollama: ollama run hf.co/bartowski/Qwen2.5-0.5B-Instruct-GGUF:Q8_0 (the Q8_0 build). LM Studio and llama.cpp commands are on this page.
- What licence is Qwen2.5 0.5B released under?
- Apache License 2.0, which allows commercial use.