Qwen2.5 0.5B
494M parameters · 32k context · Apache License 2.0 · released 2024-09-16
chat
In short
The cheapest hardware in the US that runs Qwen2.5 0.5B fully in memory at 128k context is the NVIDIA Tesla P40, $345 at Amazon, at about 100 tok/s (estimated).
Updated . Speeds are estimates; how they are worked out.
What to run it on in the US
Fully in memory at 128k context, priced at retailers today.
Cheapest
NVIDIA Tesla P40100tok/sest.Q8_0
$345
Best value
NVIDIA GeForce RTX 3080 10GB220tok/sest.Q8_0
$469
Fastest
NVIDIA GeForce RTX 5090519tok/sest.Q8_0
$4,921
Mac
Mac mini M4 16GB36tok/sest.Q8_0
$749
All hardware that runs it
Everything we price, best value first. Two cards split the model by layers.
| Hardware | Quant | Fits | Decode | Price | tok/s per 1,000 |
|---|---|---|---|---|---|
| NVIDIA Tesla P40 | Q8_0 | fits | 100 tok/s | $345 | 290.4 |
| Intel Arc B580 | Q8_0 | fits | 107 tok/s | $346 | 307.8 |
| NVIDIA GeForce RTX 3060 12GB | Q8_0 | fits | 104 tok/s | $360 | 289.5 |
| Intel Arc A770 16GB | Q8_0 | fits | 131 tok/s | $400 | 326.8 |
| NVIDIA GeForce RTX 3080 10GB | Q8_0 | fits | 220 tok/s | $469 | 468.8 |
| AMD Radeon RX 7600 XT | Q8_0 | fits | 74 tok/s | $480 | 154.2 |
| AMD Radeon RX 9060 XT 16GB | Q8_0 | fits | 82 tok/s | $520 | 158.1 |
| AMD Radeon RX 7800 XT | Q8_0 | fits | 160 tok/s | $550 | 291.3 |
| NVIDIA GeForce RTX 5060 Ti 16GB | Q8_0 | fits | 130 tok/s | $618 | 209.9 |
| Intel Arc Pro B60 | Q8_0 | fits | 107 tok/s | $650 | 163.8 |
| NVIDIA GeForce RTX 4070 Super | Q8_0 | fits | 146 tok/s | $699 | 208.7 |
| Mac mini M4 16GB | Q8_0 | fits | 36 tok/s | $749 | 48.6 |
| NVIDIA GeForce RTX 5070 | Q8_0 | fits | 195 tok/s | $750 | 259.3 |
| NVIDIA GeForce RTX 4060 Ti 16GB | Q8_0 | fits | 83 tok/s | $770 | 108.3 |
| AMD Radeon RX 7900 XT | Q8_0 | fits | 205 tok/s | $800 | 256.8 |
| AMD Radeon RX 9070 XT | Q8_0 | fits | 164 tok/s | $815 | 201.7 |
| NVIDIA GeForce RTX 4070 Ti Super | Q8_0 | fits | 195 tok/s | $984 | 197.8 |
| NVIDIA GeForce RTX 5070 Ti | Q8_0 | fits | 259 tok/s | $995 | 260.7 |
| NVIDIA GeForce RTX 4080 Super | Q8_0 | fits | 213 tok/s | $1,199 | 177.7 |
| Mac mini M4 32GB | Q8_0 | fits | 36 tok/s | $1,420 | 25.6 |
| AMD Radeon RX 7900 XTX | Q8_0 | fits | 247 tok/s | $1,533 | 160.8 |
| NVIDIA GeForce RTX 3090 | Q8_0 | fits | 271 tok/s | $1,600 | 169.4 |
| NVIDIA GeForce RTX 5080 | Q8_0 | fits | 278 tok/s | $1,600 | 173.7 |
| AMD Radeon AI PRO R9700 | Q8_0 | fits | 164 tok/s | $1,700 | 96.7 |
| NVIDIA GeForce RTX 3090 Ti | Q8_0 | fits | 292 tok/s | $2,060 | 141.7 |
| NVIDIA GeForce RTX 4090 | Q8_0 | fits | 292 tok/s | $2,073 | 140.7 |
| Mac mini M4 Pro 48GB | Q8_0 | fits | 83 tok/s | $2,149 | 38.6 |
| Mac mini M4 Pro 64GB | Q8_0 | fits | 83 tok/s | $2,900 | 28.6 |
| Mac Studio M4 Max 64GB | Q8_0 | fits | 166 tok/s | $3,400 | 48.7 |
| Framework Desktop Ryzen AI Max+ 395 128GB | Q8_0 | fits | 66 tok/s | $3,449 | 19.0 |
| GMKtec EVO-X2 Ryzen AI Max+ 395 128GB | Q8_0 | fits | 66 tok/s | $3,650 | 18.0 |
| Mac Studio M3 Ultra 96GB | Q8_0 | fits | 249 tok/s | $4,399 | 56.5 |
| NVIDIA GeForce RTX 5090 | Q8_0 | fits | 519 tok/s | $4,921 | 105.4 |
| Mac Studio M4 Max 128GB | Q8_0 | fits | 166 tok/s | $4,999 | 33.1 |
| ASUS Ascent GX10 128GB | Q8_0 | fits | 79 tok/s | $6,100 | 13.0 |
| NVIDIA DGX Spark 128GB | Q8_0 | fits | 79 tok/s | $7,399 | 10.7 |
| NVIDIA RTX PRO 6000 Blackwell | Q8_0 | fits | 519 tok/s | $7,997 | 64.9 |
| Mac Studio M3 Ultra 256GB | Q8_0 | fits | 249 tok/s | $8,499 | 29.3 |
| Mac Studio M3 Ultra 512GB | Q8_0 | fits | 249 tok/s | $23,000 | 10.8 |
| Framework Desktop Ryzen AI Max+ 395 64GB | Q8_0 | fits | 66 tok/s | – | – |
Memory needed
Weights, KV cache and runtime buffers at each context length (KV cache at fp16).
| Quant | File | @8k | @32k |
|---|---|---|---|
| Q8_0gguf | 0.5 GB | 1.3 GB | 1.8 GB |
Run it
Copy a command for the runtime you use.
- Q8_0
- Ollama
ollama run hf.co/bartowski/Qwen2.5-0.5B-Instruct-GGUF:Q8_0 - LM Studio
lms get bartowski/Qwen2.5-0.5B-Instruct-GGUF@q8_0 - llama.cpp
llama-server -hf bartowski/Qwen2.5-0.5B-Instruct-GGUF:Q8_0
- Ollama
Notes
The card says the full context is 32,768 tokens and generation is up to 8192 tokens. Needs transformers 4.37.0 or later, because older versions raise KeyError: 'qwen2'.
Questions
- What is the best GPU for running Qwen2.5 0.5B locally?
- The cheapest hardware in the US that runs Qwen2.5 0.5B fully in memory at 128k context is the NVIDIA Tesla P40, $345 at Amazon, at about 100 tok/s (estimated). The best value is the NVIDIA GeForce RTX 3080 10GB (about 220 tok/s for $469). The fastest we price is the NVIDIA GeForce RTX 5090, at about 519 tok/s.
- Can I run Qwen2.5 0.5B on a 16 GB GPU?
- Yes: at Q8_0 it fits fully in 16 GB at 128k context. On the NVIDIA GeForce RTX 5080 it runs at about 278 tok/s (estimated).
- Can I run Qwen2.5 0.5B on a 24 GB GPU?
- Yes: at Q8_0 it fits fully in 24 GB at 128k context. On the NVIDIA GeForce RTX 3090 Ti it runs at about 292 tok/s (estimated).
- How do I run Qwen2.5 0.5B locally?
- With Ollama: ollama run hf.co/bartowski/Qwen2.5-0.5B-Instruct-GGUF:Q8_0 (the Q8_0 build). LM Studio and llama.cpp commands are on this page.
- What licence is Qwen2.5 0.5B released under?
- Apache License 2.0, which allows commercial use.