Magistral Small 2506
23.6B parameters · 40k context · Apache 2.0 · released 2025-06-04
reasoningchat
In short
The cheapest hardware in Germany that runs Magistral Small 2506 fully in memory at 64k context is the NVIDIA GeForce RTX 5090, 6.028 € at Amazon, at about 37 tok/s (estimated).
Updated . Speeds are estimates; how they are worked out.
What to run it on in Germany
Fully in memory at 64k context, priced at retailers today.
Cheapest
NVIDIA GeForce RTX 509037tok/sest.Q6_K
6.028 €
Best value
NVIDIA GeForce RTX 509037tok/sest.Q6_K
6.028 €
Fastest
NVIDIA GeForce RTX 509037tok/sest.Q6_K
6.028 €
All hardware that runs it
Everything we price, best value first. Two cards split the model by layers.
| Hardware | Quant | Fits | Decode | Price | tok/s per 1,000 |
|---|---|---|---|---|---|
| AMD Radeon AI PRO R9700 | Q6_K | fits | 12 tok/s | 1.867 € | 6.3 |
| Mac mini M4 Pro 48GB | Q8_0 | fits | 5 tok/s | 2.699 € | 1.9 |
| Mac mini M4 Pro 64GB | Q8_0 | fits | 5 tok/s | 2.799 € | 1.8 |
| GMKtec EVO-X2 Ryzen AI Max+ 395 128GB | Q8_0 | fits | 4 tok/s | 3.700 € | 1.1 |
| Mac Studio M4 Max 64GB | Q8_0 | fits | 10 tok/s | 4.183 € | 2.4 |
| Mac Studio M4 Max 128GB | Q8_0 | fits | 10 tok/s | 5.800 € | 1.7 |
| Mac Studio M3 Ultra 96GB | Q8_0 | fits | 15 tok/s | 5.999 € | 2.5 |
| NVIDIA GeForce RTX 5090 | Q6_K | fits | 37 tok/s | 6.028 € | 6.1 |
| NVIDIA DGX Spark 128GB | Q8_0 | fits | 5 tok/s | 8.313 € | 0.6 |
| ASUS Ascent GX10 128GB | Q8_0 | fits | 5 tok/s | 8.623 € | 0.5 |
| Mac Studio M3 Ultra 256GB | Q8_0 | fits | 15 tok/s | 15.555 € | 1.0 |
| Mac Studio M3 Ultra 512GB | Q8_0 | fits | 15 tok/s | 18.995 € | 0.8 |
| NVIDIA RTX PRO 6000 Blackwell | Q8_0 | fits | 31 tok/s | 19.076 € | 1.6 |
| Framework Desktop Ryzen AI Max+ 395 128GB | Q8_0 | fits | 4 tok/s | – | – |
| Framework Desktop Ryzen AI Max+ 395 64GB | Q8_0 | fits | 4 tok/s | – | – |
| NVIDIA GeForce RTX 3060 12GB | Q4_K_M | offload | 3 tok/s | 309 € | 10.4 |
| Intel Arc B580 | Q4_K_M | offload | 3 tok/s | 360 € | 8.9 |
| NVIDIA Tesla P40 | Q4_K_M | offload | 8 tok/s | 369 € | 20.6 |
| NVIDIA GeForce RTX 3080 10GB | Q4_K_M | offload | 3 tok/s | 500 € | 6.2 |
| AMD Radeon RX 9060 XT 16GB | Q4_K_M | offload | 4 tok/s | 550 € | 6.7 |
| Intel Arc A770 16GB | Q4_K_M | offload | 4 tok/s | 646 € | 6.5 |
| NVIDIA GeForce RTX 4060 Ti 16GB | Q4_K_M | offload | 4 tok/s | 650 € | 5.8 |
| NVIDIA GeForce RTX 4070 Super | Q4_K_M | offload | 3 tok/s | 707 € | 4.8 |
| NVIDIA GeForce RTX 5060 Ti 16GB | Q4_K_M | offload | 4 tok/s | 739 € | 5.7 |
| AMD Radeon RX 9070 XT | Q4_K_M | offload | 4 tok/s | 778 € | 5.7 |
| NVIDIA GeForce RTX 5070 | Q4_K_M | offload | 4 tok/s | 798 € | 4.4 |
| AMD Radeon RX 7600 XT | Q4_K_M | offload | 4 tok/s | 813 € | 4.4 |
| AMD Radeon RX 7900 XT | Q4_K_M | offload | 7 tok/s | 855 € | 7.8 |
| AMD Radeon RX 7800 XT | Q4_K_M | offload | 4 tok/s | 1.002 € | 4.3 |
| Intel Arc Pro B60 | Q4_K_M | offload | 8 tok/s | 1.074 € | 7.2 |
| NVIDIA GeForce RTX 4080 Super | Q4_K_M | offload | 5 tok/s | 1.100 € | 4.3 |
| NVIDIA GeForce RTX 5070 Ti | Q4_K_M | offload | 5 tok/s | 1.149 € | 4.2 |
| NVIDIA GeForce RTX 4070 Ti Super | Q4_K_M | offload | 5 tok/s | 1.247 € | 3.7 |
| NVIDIA GeForce RTX 5080 | Q4_K_M | offload | 5 tok/s | 1.500 € | 3.3 |
| AMD Radeon RX 7900 XTX | Q4_K_M | offload | 14 tok/s | 1.555 € | 9.0 |
| NVIDIA GeForce RTX 3090 | Q4_K_M | offload | 16 tok/s | 1.850 € | 8.8 |
| NVIDIA GeForce RTX 3090 Ti | Q4_K_M | offload | 17 tok/s | 2.500 € | 6.8 |
| NVIDIA GeForce RTX 4090 | Q4_K_M | offload | 17 tok/s | 3.955 € | 4.3 |
2 other cards and machines we price cannot hold it at this context.
Memory needed
Weights, KV cache and runtime buffers at each context length (KV cache at fp16).
| Quant | File | @8k | @32k |
|---|---|---|---|
| Q4_K_Mgguf | 13.3 GB | 15.3 GB | 19.2 GB |
| Q5_K_Mgguf | 15.6 GB | 17.6 GB | 21.5 GB |
| Q6_Kgguf | 18.0 GB | 20.0 GB | 23.9 GB |
| Q8_0gguf | 23.3 GB | 25.3 GB | 29.2 GB |
Run it
Copy a command for the runtime you use.
- Q4_K_M
- Ollama
ollama run hf.co/unsloth/Magistral-Small-2506-GGUF:Q4_K_M - LM Studio
lms get unsloth/Magistral-Small-2506-GGUF@q4_k_m - llama.cpp
llama-server -hf unsloth/Magistral-Small-2506-GGUF:Q4_K_M
- Ollama
- Q5_K_M
- Ollama
ollama run hf.co/unsloth/Magistral-Small-2506-GGUF:Q5_K_M - LM Studio
lms get unsloth/Magistral-Small-2506-GGUF@q5_k_m - llama.cpp
llama-server -hf unsloth/Magistral-Small-2506-GGUF:Q5_K_M
- Ollama
- Q6_K
- Ollama
ollama run hf.co/unsloth/Magistral-Small-2506-GGUF:Q6_K - LM Studio
lms get unsloth/Magistral-Small-2506-GGUF@q6_k - llama.cpp
llama-server -hf unsloth/Magistral-Small-2506-GGUF:Q6_K
- Ollama
- Q8_0
- Ollama
ollama run hf.co/unsloth/Magistral-Small-2506-GGUF:Q8_0 - LM Studio
lms get unsloth/Magistral-Small-2506-GGUF@q8_0 - llama.cpp
llama-server -hf unsloth/Magistral-Small-2506-GGUF:Q8_0
- Ollama
Notes
Performance may degrade past 40k tokens, so the card recommends setting max model length to 40k. Needs a recent vLLM with mistral_common >= 1.6.0.
Questions
- What is the best GPU for running Magistral Small 2506 locally?
- The cheapest hardware in Germany that runs Magistral Small 2506 fully in memory at 64k context is the NVIDIA GeForce RTX 5090, 6.028 € at Amazon, at about 37 tok/s (estimated).
- Can I run Magistral Small 2506 on a 16 GB GPU?
- Only partly: at 64k context it does not fit in 16 GB, so part of it runs from system RAM and it slows to about 4 tok/s on the AMD Radeon RX 9060 XT 16GB (estimated).
- Can I run Magistral Small 2506 on a 24 GB GPU?
- Only partly: at 64k context it does not fit in 24 GB, so part of it runs from system RAM and it slows to about 8 tok/s on the NVIDIA Tesla P40 (estimated).
- How do I run Magistral Small 2506 locally?
- With Ollama: ollama run hf.co/unsloth/Magistral-Small-2506-GGUF:Q4_K_M (the Q4_K_M build). LM Studio and llama.cpp commands are on this page.
- What licence is Magistral Small 2506 released under?
- Apache 2.0, which allows commercial use.