Qwen2.5 0.5B
494M parameters · 32k context · Apache License 2.0 · released 2024-09-16
chat
In short
Qwen2.5 0.5B is an open model from Alibaba Qwen with 494M parameters. At Q8_0, Qwen2.5 0.5B needs about 1 GB of memory at 8k context and 2 GB at 32k, counting the weights, the KV cache and runtime buffers. The cheapest hardware in the UK that runs Qwen2.5 0.5B fully in memory at 32k context is the NVIDIA Tesla P40, £261 at eBay, at about 230 tok/s (estimated).
Updated . Speeds are estimates; how they are worked out.
What to run it on in the UK
Fully in memory at 32k context, priced at retailers today.
Cheapest
NVIDIA Tesla P40230tok/sest.Q8_0
£261
Best value
NVIDIA GeForce RTX 3080 10GB505tok/sest.Q8_0
£466
Fastest
NVIDIA GeForce RTX 50901190tok/sest.Q8_0
£5,000
Mac
Mac mini M4 16GB84tok/sest.Q8_0
£765
All hardware that runs it
Everything we price, best value first. Two cards split the model by layers.
| Hardware | Quant | Fits | Decode | Price | tok/s per 1,000 |
|---|---|---|---|---|---|
| NVIDIA Tesla P40 | Q8_0 | fits | 230 tok/s | £261 | 881.1 |
| AMD Radeon RX 7600 XT | Q8_0 | fits | 170 tok/s | £348 | 486.8 |
| Intel Arc B580 | Q8_0 | fits | 244 tok/s | £349 | 700.6 |
| NVIDIA GeForce RTX 3060 12GB | Q8_0 | fits | 239 tok/s | £390 | 612.8 |
| NVIDIA GeForce RTX 3080 10GB | Q8_0 | fits | 505 tok/s | £466 | 1083.5 |
| AMD Radeon RX 9060 XT 16GB | Q8_0 | fits | 189 tok/s | £499 | 377.9 |
| NVIDIA GeForce RTX 5060 Ti 16GB | Q8_0 | fits | 298 tok/s | £550 | 540.9 |
| AMD Radeon RX 7800 XT | Q8_0 | fits | 368 tok/s | £568 | 647.3 |
| Intel Arc A770 16GB | Q8_0 | fits | 300 tok/s | £588 | 510.0 |
| AMD Radeon RX 9070 XT | Q8_0 | fits | 377 tok/s | £670 | 562.9 |
| NVIDIA GeForce RTX 5070 | Q8_0 | fits | 446 tok/s | £695 | 642.0 |
| Intel Arc Pro B60 | Q8_0 | fits | 244 tok/s | £742 | 329.1 |
| Mac mini M4 16GB | Q8_0 | fits | 84 tok/s | £765 | 109.2 |
| AMD Radeon RX 7900 XT | Q8_0 | fits | 471 tok/s | £789 | 597.2 |
| NVIDIA GeForce RTX 4070 Super | Q8_0 | fits | 335 tok/s | £874 | 383.1 |
| NVIDIA GeForce RTX 4070 Ti Super | Q8_0 | fits | 446 tok/s | £950 | 469.7 |
| NVIDIA GeForce RTX 5070 Ti | Q8_0 | fits | 595 tok/s | £975 | 610.3 |
| NVIDIA GeForce RTX 4060 Ti 16GB | Q8_0 | fits | 191 tok/s | £977 | 195.7 |
| AMD Radeon RX 7900 XTX | Q8_0 | fits | 566 tok/s | £999 | 566.1 |
| NVIDIA GeForce RTX 5080 | Q8_0 | fits | 637 tok/s | £1,199 | 531.6 |
| Mac mini M4 32GB | Q8_0 | fits | 84 tok/s | £1,200 | 69.6 |
| NVIDIA GeForce RTX 4080 Super | Q8_0 | fits | 489 tok/s | £1,429 | 341.9 |
| AMD Radeon AI PRO R9700 | Q8_0 | fits | 377 tok/s | £1,674 | 225.2 |
| NVIDIA GeForce RTX 3090 | Q8_0 | fits | 622 tok/s | £1,707 | 364.1 |
| NVIDIA GeForce RTX 3090 Ti | Q8_0 | fits | 669 tok/s | £2,199 | 304.4 |
| Mac mini M4 Pro 48GB | Q8_0 | fits | 190 tok/s | £3,000 | 63.3 |
| GMKtec EVO-X2 Ryzen AI Max+ 395 128GB | Q8_0 | fits | 151 tok/s | £3,320 | 45.4 |
| Mac Studio M4 Max 64GB | Q8_0 | fits | 380 tok/s | £3,399 | 111.8 |
| NVIDIA GeForce RTX 5090 | Q8_0 | fits | 1190 tok/s | £5,000 | 238.0 |
| Mac Studio M3 Ultra 96GB | Q8_0 | fits | 570 tok/s | £5,499 | 103.7 |
| NVIDIA GeForce RTX 4090 | Q8_0 | fits | 669 tok/s | £6,015 | 111.3 |
| Mac Studio M3 Ultra 256GB | Q8_0 | fits | 570 tok/s | £8,499 | 67.1 |
| Mac Studio M3 Ultra 512GB | Q8_0 | fits | 570 tok/s | £17,495 | 32.6 |
| NVIDIA RTX PRO 6000 Blackwell | Q8_0 | fits | 1190 tok/s | £29,504 | 40.3 |
| Mac Studio M4 Max 128GB | Q8_0 | fits | 380 tok/s | – | – |
| Mac mini M4 Pro 64GB | Q8_0 | fits | 190 tok/s | – | – |
| Framework Desktop Ryzen AI Max+ 395 128GB | Q8_0 | fits | 151 tok/s | – | – |
| NVIDIA DGX Spark 128GB | Q8_0 | fits | 181 tok/s | – | – |
| ASUS Ascent GX10 128GB | Q8_0 | fits | 181 tok/s | – | – |
| Framework Desktop Ryzen AI Max+ 395 64GB | Q8_0 | fits | 151 tok/s | – | – |
Memory needed
Weights, KV cache and runtime buffers at each context length (KV cache at fp16).
| Quant | File | @8k | @32k |
|---|---|---|---|
| Q8_0gguf | 0.5 GB | 1.3 GB | 1.8 GB |
Run it
Copy a command for the runtime you use.
- Q8_0
- Ollama
ollama run hf.co/bartowski/Qwen2.5-0.5B-Instruct-GGUF:Q8_0 - LM Studio
lms get bartowski/Qwen2.5-0.5B-Instruct-GGUF@q8_0 - llama.cpp
llama-server -hf bartowski/Qwen2.5-0.5B-Instruct-GGUF:Q8_0
- Ollama
Notes
The card says the full context is 32,768 tokens and generation is up to 8192 tokens. Needs transformers 4.37.0 or later, because older versions raise KeyError: 'qwen2'.
Questions
- How much VRAM does Qwen2.5 0.5B need?
- At Q8_0, Qwen2.5 0.5B needs about 1 GB of memory at 8k context and 2 GB at 32k, counting the weights, the KV cache and runtime buffers.
- What is the best GPU for running Qwen2.5 0.5B locally?
- The cheapest hardware in the UK that runs Qwen2.5 0.5B fully in memory at 32k context is the NVIDIA Tesla P40, £261 at eBay, at about 230 tok/s (estimated). The best value is the NVIDIA GeForce RTX 3080 10GB (about 505 tok/s for £466). The fastest we price is the NVIDIA GeForce RTX 5090, at about 1190 tok/s.
- Can I run Qwen2.5 0.5B on a 16 GB GPU?
- Yes: at Q8_0 it fits fully in 16 GB at 32k context. On the NVIDIA GeForce RTX 5080 it runs at about 637 tok/s (estimated).
- Can I run Qwen2.5 0.5B on a 24 GB GPU?
- Yes: at Q8_0 it fits fully in 24 GB at 32k context. On the NVIDIA GeForce RTX 3090 Ti it runs at about 669 tok/s (estimated).
- How do I run Qwen2.5 0.5B locally?
- With Ollama: ollama run hf.co/bartowski/Qwen2.5-0.5B-Instruct-GGUF:Q8_0 (the Q8_0 build). LM Studio and llama.cpp commands are on this page.
- What licence is Qwen2.5 0.5B released under?
- Apache License 2.0, which allows commercial use.