Qwen2.5 0.5B
494M parameters · 32k context · Apache License 2.0 · released 2024-09-16
chat
In short
The cheapest hardware in the US that runs Qwen2.5 0.5B fully in memory at 64k context is the NVIDIA Tesla P40, $345 at Amazon, at about 161 tok/s (estimated).
Updated . Speeds are estimates; how they are worked out.
What to run it on in the US
Fully in memory at 64k context, priced at retailers today.
Cheapest
NVIDIA Tesla P40161tok/sest.Q8_0
$345
Best value
NVIDIA GeForce RTX 3080 10GB353tok/sest.Q8_0
$469
Fastest
NVIDIA GeForce RTX 5090831tok/sest.Q8_0
$4,921
Mac
Mac mini M4 16GB58tok/sest.Q8_0
$749
All hardware that runs it
Everything we price, best value first. Two cards split the model by layers.
| Hardware | Quant | Fits | Decode | Price | tok/s per 1,000 |
|---|---|---|---|---|---|
| NVIDIA Tesla P40 | Q8_0 | fits | 161 tok/s | $345 | 465.2 |
| Intel Arc B580 | Q8_0 | fits | 171 tok/s | $346 | 493.1 |
| NVIDIA GeForce RTX 3060 12GB | Q8_0 | fits | 167 tok/s | $360 | 463.9 |
| Intel Arc A770 16GB | Q8_0 | fits | 210 tok/s | $400 | 523.8 |
| NVIDIA GeForce RTX 3080 10GB | Q8_0 | fits | 353 tok/s | $469 | 751.3 |
| AMD Radeon RX 7600 XT | Q8_0 | fits | 119 tok/s | $480 | 246.9 |
| AMD Radeon RX 9060 XT 16GB | Q8_0 | fits | 132 tok/s | $520 | 253.3 |
| AMD Radeon RX 7800 XT | Q8_0 | fits | 257 tok/s | $550 | 466.9 |
| NVIDIA GeForce RTX 5060 Ti 16GB | Q8_0 | fits | 208 tok/s | $618 | 336.3 |
| Intel Arc Pro B60 | Q8_0 | fits | 171 tok/s | $650 | 262.5 |
| NVIDIA GeForce RTX 4070 Super | Q8_0 | fits | 234 tok/s | $699 | 334.4 |
| Mac mini M4 16GB | Q8_0 | fits | 58 tok/s | $749 | 78.0 |
| NVIDIA GeForce RTX 5070 | Q8_0 | fits | 312 tok/s | $750 | 415.7 |
| NVIDIA GeForce RTX 4060 Ti 16GB | Q8_0 | fits | 134 tok/s | $770 | 173.5 |
| AMD Radeon RX 7900 XT | Q8_0 | fits | 329 tok/s | $800 | 411.5 |
| AMD Radeon RX 9070 XT | Q8_0 | fits | 263 tok/s | $815 | 323.2 |
| NVIDIA GeForce RTX 4070 Ti Super | Q8_0 | fits | 312 tok/s | $984 | 317.0 |
| NVIDIA GeForce RTX 5070 Ti | Q8_0 | fits | 416 tok/s | $995 | 417.8 |
| NVIDIA GeForce RTX 4080 Super | Q8_0 | fits | 342 tok/s | $1,199 | 284.8 |
| Mac mini M4 32GB | Q8_0 | fits | 58 tok/s | $1,420 | 41.1 |
| AMD Radeon RX 7900 XTX | Q8_0 | fits | 395 tok/s | $1,533 | 257.7 |
| NVIDIA GeForce RTX 3090 | Q8_0 | fits | 434 tok/s | $1,600 | 271.4 |
| NVIDIA GeForce RTX 5080 | Q8_0 | fits | 445 tok/s | $1,600 | 278.4 |
| AMD Radeon AI PRO R9700 | Q8_0 | fits | 263 tok/s | $1,700 | 154.9 |
| NVIDIA GeForce RTX 3090 Ti | Q8_0 | fits | 468 tok/s | $2,060 | 227.0 |
| NVIDIA GeForce RTX 4090 | Q8_0 | fits | 468 tok/s | $2,073 | 225.6 |
| Mac mini M4 Pro 48GB | Q8_0 | fits | 133 tok/s | $2,149 | 61.8 |
| Mac mini M4 Pro 64GB | Q8_0 | fits | 133 tok/s | $2,900 | 45.8 |
| Mac Studio M4 Max 64GB | Q8_0 | fits | 266 tok/s | $3,400 | 78.1 |
| Framework Desktop Ryzen AI Max+ 395 128GB | Q8_0 | fits | 105 tok/s | $3,449 | 30.6 |
| GMKtec EVO-X2 Ryzen AI Max+ 395 128GB | Q8_0 | fits | 105 tok/s | $3,650 | 28.9 |
| Mac Studio M3 Ultra 96GB | Q8_0 | fits | 398 tok/s | $4,399 | 90.6 |
| NVIDIA GeForce RTX 5090 | Q8_0 | fits | 831 tok/s | $4,921 | 168.9 |
| Mac Studio M4 Max 128GB | Q8_0 | fits | 266 tok/s | $4,999 | 53.1 |
| ASUS Ascent GX10 128GB | Q8_0 | fits | 127 tok/s | $6,100 | 20.8 |
| NVIDIA DGX Spark 128GB | Q8_0 | fits | 127 tok/s | $7,399 | 17.1 |
| NVIDIA RTX PRO 6000 Blackwell | Q8_0 | fits | 831 tok/s | $7,997 | 104.0 |
| Mac Studio M3 Ultra 256GB | Q8_0 | fits | 398 tok/s | $8,499 | 46.9 |
| Mac Studio M3 Ultra 512GB | Q8_0 | fits | 398 tok/s | $23,000 | 17.3 |
| Framework Desktop Ryzen AI Max+ 395 64GB | Q8_0 | fits | 105 tok/s | – | – |
Memory needed
Weights, KV cache and runtime buffers at each context length (KV cache at fp16).
| Quant | File | @8k | @32k |
|---|---|---|---|
| Q8_0gguf | 0.5 GB | 1.3 GB | 1.8 GB |
Run it
Copy a command for the runtime you use.
- Q8_0
- Ollama
ollama run hf.co/bartowski/Qwen2.5-0.5B-Instruct-GGUF:Q8_0 - LM Studio
lms get bartowski/Qwen2.5-0.5B-Instruct-GGUF@q8_0 - llama.cpp
llama-server -hf bartowski/Qwen2.5-0.5B-Instruct-GGUF:Q8_0
- Ollama
Notes
The card says the full context is 32,768 tokens and generation is up to 8192 tokens. Needs transformers 4.37.0 or later, because older versions raise KeyError: 'qwen2'.
Questions
- What is the best GPU for running Qwen2.5 0.5B locally?
- The cheapest hardware in the US that runs Qwen2.5 0.5B fully in memory at 64k context is the NVIDIA Tesla P40, $345 at Amazon, at about 161 tok/s (estimated). The best value is the NVIDIA GeForce RTX 3080 10GB (about 353 tok/s for $469). The fastest we price is the NVIDIA GeForce RTX 5090, at about 831 tok/s.
- Can I run Qwen2.5 0.5B on a 16 GB GPU?
- Yes: at Q8_0 it fits fully in 16 GB at 64k context. On the NVIDIA GeForce RTX 5080 it runs at about 445 tok/s (estimated).
- Can I run Qwen2.5 0.5B on a 24 GB GPU?
- Yes: at Q8_0 it fits fully in 24 GB at 64k context. On the NVIDIA GeForce RTX 3090 Ti it runs at about 468 tok/s (estimated).
- How do I run Qwen2.5 0.5B locally?
- With Ollama: ollama run hf.co/bartowski/Qwen2.5-0.5B-Instruct-GGUF:Q8_0 (the Q8_0 build). LM Studio and llama.cpp commands are on this page.
- What licence is Qwen2.5 0.5B released under?
- Apache License 2.0, which allows commercial use.