Qwen2.5 0.5B
494M parameters · 32k context · Apache License 2.0 · released 2024-09-16
chat
In short
Qwen2.5 0.5B is an open model from Alibaba Qwen with 494M parameters. At Q8_0, Qwen2.5 0.5B needs about 1 GB of memory at 8k context and 1 GB at 8k, counting the weights, the KV cache and runtime buffers. The cheapest hardware in the US that runs Qwen2.5 0.5B fully in memory at 8k context is the NVIDIA Tesla P40, $345 at Amazon, at about 340 tok/s (estimated).
Updated . Speeds are estimates; how they are worked out.
What to run it on in the US
Fully in memory at 8k context, priced at retailers today.
Cheapest
NVIDIA Tesla P40340tok/sest.Q8_0
$345
Best value
NVIDIA GeForce RTX 3080 10GB746tok/sest.Q8_0
$469
Fastest
NVIDIA GeForce RTX 50901759tok/sest.Q8_0
$4,921
Mac
Mac mini M4 16GB124tok/sest.Q8_0
$749
All hardware that runs it
Everything we price, best value first. Two cards split the model by layers.
| Hardware | Quant | Fits | Decode | Price | tok/s per 1,000 |
|---|---|---|---|---|---|
| NVIDIA Tesla P40 | Q8_0 | fits | 340 tok/s | $345 | 984.3 |
| Intel Arc B580 | Q8_0 | fits | 361 tok/s | $346 | 1043.1 |
| NVIDIA GeForce RTX 3060 12GB | Q8_0 | fits | 353 tok/s | $360 | 981.4 |
| Intel Arc A770 16GB | Q8_0 | fits | 443 tok/s | $400 | 1108.0 |
| NVIDIA GeForce RTX 3080 10GB | Q8_0 | fits | 746 tok/s | $469 | 1589.3 |
| AMD Radeon RX 7600 XT | Q8_0 | fits | 251 tok/s | $480 | 522.3 |
| AMD Radeon RX 9060 XT 16GB | Q8_0 | fits | 279 tok/s | $520 | 535.8 |
| AMD Radeon RX 7800 XT | Q8_0 | fits | 543 tok/s | $550 | 987.8 |
| NVIDIA GeForce RTX 5060 Ti 16GB | Q8_0 | fits | 440 tok/s | $618 | 711.5 |
| Intel Arc Pro B60 | Q8_0 | fits | 361 tok/s | $650 | 555.2 |
| NVIDIA GeForce RTX 4070 Super | Q8_0 | fits | 495 tok/s | $699 | 707.4 |
| Mac mini M4 16GB | Q8_0 | fits | 124 tok/s | $749 | 164.9 |
| NVIDIA GeForce RTX 5070 | Q8_0 | fits | 660 tok/s | $750 | 879.3 |
| NVIDIA GeForce RTX 4060 Ti 16GB | Q8_0 | fits | 283 tok/s | $770 | 367.1 |
| AMD Radeon RX 7900 XT | Q8_0 | fits | 697 tok/s | $800 | 870.6 |
| AMD Radeon RX 9070 XT | Q8_0 | fits | 557 tok/s | $815 | 683.7 |
| NVIDIA GeForce RTX 4070 Ti Super | Q8_0 | fits | 660 tok/s | $984 | 670.5 |
| NVIDIA GeForce RTX 5070 Ti | Q8_0 | fits | 879 tok/s | $995 | 883.8 |
| NVIDIA GeForce RTX 4080 Super | Q8_0 | fits | 722 tok/s | $1,199 | 602.4 |
| Mac mini M4 32GB | Q8_0 | fits | 124 tok/s | $1,420 | 87.0 |
| AMD Radeon RX 7900 XTX | Q8_0 | fits | 836 tok/s | $1,533 | 545.1 |
| NVIDIA GeForce RTX 3090 | Q8_0 | fits | 919 tok/s | $1,600 | 574.1 |
| NVIDIA GeForce RTX 5080 | Q8_0 | fits | 942 tok/s | $1,600 | 588.9 |
| AMD Radeon AI PRO R9700 | Q8_0 | fits | 557 tok/s | $1,700 | 327.8 |
| NVIDIA GeForce RTX 3090 Ti | Q8_0 | fits | 989 tok/s | $2,060 | 480.2 |
| NVIDIA GeForce RTX 4090 | Q8_0 | fits | 989 tok/s | $2,073 | 477.1 |
| Mac mini M4 Pro 48GB | Q8_0 | fits | 281 tok/s | $2,149 | 130.7 |
| Mac mini M4 Pro 64GB | Q8_0 | fits | 281 tok/s | $2,900 | 96.9 |
| Mac Studio M4 Max 64GB | Q8_0 | fits | 562 tok/s | $3,400 | 165.2 |
| Framework Desktop Ryzen AI Max+ 395 128GB | Q8_0 | fits | 223 tok/s | $3,449 | 64.6 |
| GMKtec EVO-X2 Ryzen AI Max+ 395 128GB | Q8_0 | fits | 223 tok/s | $3,650 | 61.1 |
| Mac Studio M3 Ultra 96GB | Q8_0 | fits | 843 tok/s | $4,399 | 191.6 |
| NVIDIA GeForce RTX 5090 | Q8_0 | fits | 1759 tok/s | $4,921 | 357.4 |
| Mac Studio M4 Max 128GB | Q8_0 | fits | 562 tok/s | $4,999 | 112.4 |
| ASUS Ascent GX10 128GB | Q8_0 | fits | 268 tok/s | $6,100 | 43.9 |
| NVIDIA DGX Spark 128GB | Q8_0 | fits | 268 tok/s | $7,399 | 36.2 |
| NVIDIA RTX PRO 6000 Blackwell | Q8_0 | fits | 1759 tok/s | $7,997 | 219.9 |
| Mac Studio M3 Ultra 256GB | Q8_0 | fits | 843 tok/s | $8,499 | 99.2 |
| Mac Studio M3 Ultra 512GB | Q8_0 | fits | 843 tok/s | $23,000 | 36.6 |
| Framework Desktop Ryzen AI Max+ 395 64GB | Q8_0 | fits | 223 tok/s | – | – |
Memory needed
Weights, KV cache and runtime buffers at each context length (KV cache at fp16).
| Quant | File | @8k | @32k |
|---|---|---|---|
| Q8_0gguf | 0.5 GB | 1.3 GB | 1.8 GB |
Run it
Copy a command for the runtime you use.
- Q8_0
- Ollama
ollama run hf.co/bartowski/Qwen2.5-0.5B-Instruct-GGUF:Q8_0 - LM Studio
lms get bartowski/Qwen2.5-0.5B-Instruct-GGUF@q8_0 - llama.cpp
llama-server -hf bartowski/Qwen2.5-0.5B-Instruct-GGUF:Q8_0
- Ollama
Notes
The card says the full context is 32,768 tokens and generation is up to 8192 tokens. Needs transformers 4.37.0 or later, because older versions raise KeyError: 'qwen2'.
Questions
- How much VRAM does Qwen2.5 0.5B need?
- At Q8_0, Qwen2.5 0.5B needs about 1 GB of memory at 8k context and 1 GB at 8k, counting the weights, the KV cache and runtime buffers.
- What is the best GPU for running Qwen2.5 0.5B locally?
- The cheapest hardware in the US that runs Qwen2.5 0.5B fully in memory at 8k context is the NVIDIA Tesla P40, $345 at Amazon, at about 340 tok/s (estimated). The best value is the NVIDIA GeForce RTX 3080 10GB (about 746 tok/s for $469). The fastest we price is the NVIDIA GeForce RTX 5090, at about 1759 tok/s.
- Can I run Qwen2.5 0.5B on a 16 GB GPU?
- Yes: at Q8_0 it fits fully in 16 GB at 8k context. On the NVIDIA GeForce RTX 5080 it runs at about 942 tok/s (estimated).
- Can I run Qwen2.5 0.5B on a 24 GB GPU?
- Yes: at Q8_0 it fits fully in 24 GB at 8k context. On the NVIDIA GeForce RTX 3090 Ti it runs at about 989 tok/s (estimated).
- How do I run Qwen2.5 0.5B locally?
- With Ollama: ollama run hf.co/bartowski/Qwen2.5-0.5B-Instruct-GGUF:Q8_0 (the Q8_0 build). LM Studio and llama.cpp commands are on this page.
- What licence is Qwen2.5 0.5B released under?
- Apache License 2.0, which allows commercial use.