Magistral Small 2507
23.6B parameters · 40k context · Apache 2.0 · released 2025-07-18
reasoningchatagentic
In short
The cheapest hardware in the US that runs Magistral Small 2507 fully in memory at 64k context is the NVIDIA GeForce RTX 5090, $4,921 at eBay, at about 40 tok/s (estimated).
Updated . Speeds are estimates; how they are worked out.
What to run it on in the US
Fully in memory at 64k context, priced at retailers today.
Cheapest
NVIDIA GeForce RTX 509040tok/sest.Q5_K_M
$4,921
Best value
NVIDIA GeForce RTX 509040tok/sest.Q5_K_M
$4,921
Fastest
NVIDIA GeForce RTX 509040tok/sest.Q5_K_M
$4,921
All hardware that runs it
Everything we price, best value first. Two cards split the model by layers.
| Hardware | Quant | Fits | Decode | Price | tok/s per 1,000 |
|---|---|---|---|---|---|
| AMD Radeon AI PRO R9700 | Q5_K_M | fits | 13 tok/s | $1,700 | 7.5 |
| Mac mini M4 Pro 48GB | Q8_0 | fits | 5 tok/s | $2,149 | 2.3 |
| Mac mini M4 Pro 64GB | Q8_0 | fits | 5 tok/s | $2,900 | 1.7 |
| Mac Studio M4 Max 64GB | Q8_0 | fits | 10 tok/s | $3,400 | 2.9 |
| Framework Desktop Ryzen AI Max+ 395 128GB | Q8_0 | fits | 4 tok/s | $3,449 | 1.1 |
| GMKtec EVO-X2 Ryzen AI Max+ 395 128GB | Q8_0 | fits | 4 tok/s | $3,650 | 1.1 |
| Mac Studio M3 Ultra 96GB | Q8_0 | fits | 15 tok/s | $4,399 | 3.4 |
| NVIDIA GeForce RTX 5090 | Q5_K_M | fits | 40 tok/s | $4,921 | 8.2 |
| Mac Studio M4 Max 128GB | Q8_0 | fits | 10 tok/s | $4,999 | 2.0 |
| ASUS Ascent GX10 128GB | Q8_0 | fits | 5 tok/s | $6,100 | 0.8 |
| NVIDIA DGX Spark 128GB | Q8_0 | fits | 5 tok/s | $7,399 | 0.6 |
| NVIDIA RTX PRO 6000 Blackwell | Q8_0 | fits | 31 tok/s | $7,997 | 3.9 |
| Mac Studio M3 Ultra 256GB | Q8_0 | fits | 15 tok/s | $8,499 | 1.8 |
| Mac Studio M3 Ultra 512GB | Q8_0 | fits | 15 tok/s | $23,000 | 0.6 |
| Framework Desktop Ryzen AI Max+ 395 64GB | Q8_0 | fits | 4 tok/s | – | – |
| NVIDIA Tesla P40 | Q4_K_M | offload | 8 tok/s | $345 | 22.0 |
| Intel Arc B580 | Q4_K_M | offload | 3 tok/s | $346 | 9.2 |
| NVIDIA GeForce RTX 3060 12GB | Q4_K_M | offload | 3 tok/s | $360 | 8.9 |
| Intel Arc A770 16GB | Q4_K_M | offload | 4 tok/s | $400 | 10.5 |
| NVIDIA GeForce RTX 3080 10GB | Q4_K_M | offload | 3 tok/s | $469 | 6.6 |
| AMD Radeon RX 7600 XT | Q4_K_M | offload | 4 tok/s | $480 | 7.5 |
| AMD Radeon RX 9060 XT 16GB | Q4_K_M | offload | 4 tok/s | $520 | 7.1 |
| AMD Radeon RX 7800 XT | Q4_K_M | offload | 4 tok/s | $550 | 7.8 |
| NVIDIA GeForce RTX 5060 Ti 16GB | Q4_K_M | offload | 4 tok/s | $618 | 6.8 |
| Intel Arc Pro B60 | Q4_K_M | offload | 8 tok/s | $650 | 11.8 |
| NVIDIA GeForce RTX 4070 Super | Q4_K_M | offload | 3 tok/s | $699 | 4.9 |
| NVIDIA GeForce RTX 5070 | Q4_K_M | offload | 4 tok/s | $750 | 4.7 |
| NVIDIA GeForce RTX 4060 Ti 16GB | Q4_K_M | offload | 4 tok/s | $770 | 4.9 |
| AMD Radeon RX 7900 XT | Q4_K_M | offload | 7 tok/s | $800 | 8.4 |
| AMD Radeon RX 9070 XT | Q4_K_M | offload | 4 tok/s | $815 | 5.4 |
| NVIDIA GeForce RTX 4070 Ti Super | Q4_K_M | offload | 5 tok/s | $984 | 4.7 |
| NVIDIA GeForce RTX 5070 Ti | Q4_K_M | offload | 5 tok/s | $995 | 4.8 |
| NVIDIA GeForce RTX 4080 Super | Q4_K_M | offload | 5 tok/s | $1,199 | 3.9 |
| AMD Radeon RX 7900 XTX | Q4_K_M | offload | 14 tok/s | $1,533 | 9.1 |
| NVIDIA GeForce RTX 3090 | Q4_K_M | offload | 16 tok/s | $1,600 | 10.1 |
| NVIDIA GeForce RTX 5080 | Q4_K_M | offload | 5 tok/s | $1,600 | 3.1 |
| NVIDIA GeForce RTX 3090 Ti | Q4_K_M | offload | 17 tok/s | $2,060 | 8.3 |
| NVIDIA GeForce RTX 4090 | Q4_K_M | offload | 17 tok/s | $2,073 | 8.2 |
2 other cards and machines we price cannot hold it at this context.
Memory needed
Weights, KV cache and runtime buffers at each context length (KV cache at fp16).
| Quant | File | @8k | @32k |
|---|---|---|---|
| Q4_K_Mgguf | 13.3 GB | 15.3 GB | 19.2 GB |
| Q5_K_Mgguf | 15.6 GB | 17.6 GB | 21.5 GB |
| Q8_0gguf | 23.3 GB | 25.3 GB | 29.2 GB |
Run it
Copy a command for the runtime you use.
- Q4_K_M
- Ollama
ollama run hf.co/mistralai/Magistral-Small-2507-GGUF:Q4_K_M - LM Studio
lms get mistralai/Magistral-Small-2507-GGUF@q4_k_m - llama.cpp
llama-server -hf mistralai/Magistral-Small-2507-GGUF:Q4_K_M
- Ollama
- Q5_K_M
- Ollama
ollama run hf.co/mistralai/Magistral-Small-2507-GGUF:Q5_K_M - LM Studio
lms get mistralai/Magistral-Small-2507-GGUF@q5_k_m - llama.cpp
llama-server -hf mistralai/Magistral-Small-2507-GGUF:Q5_K_M
- Ollama
- Q8_0
- Ollama
ollama run hf.co/mistralai/Magistral-Small-2507-GGUF:Q8_0 - LM Studio
lms get mistralai/Magistral-Small-2507-GGUF@q8_0 - llama.cpp
llama-server -hf mistralai/Magistral-Small-2507-GGUF:Q8_0
- Ollama
Notes
128k context window, but performance may degrade past 40k, so a 40k max model length is recommended. Needs mistral_common >= 1.8.2 and the latest vLLM.
Questions
- What is the best GPU for running Magistral Small 2507 locally?
- The cheapest hardware in the US that runs Magistral Small 2507 fully in memory at 64k context is the NVIDIA GeForce RTX 5090, $4,921 at eBay, at about 40 tok/s (estimated).
- Can I run Magistral Small 2507 on a 16 GB GPU?
- Only partly: at 64k context it does not fit in 16 GB, so part of it runs from system RAM and it slows to about 4 tok/s on the Intel Arc A770 16GB (estimated).
- Can I run Magistral Small 2507 on a 24 GB GPU?
- Only partly: at 64k context it does not fit in 24 GB, so part of it runs from system RAM and it slows to about 8 tok/s on the NVIDIA Tesla P40 (estimated).
- How do I run Magistral Small 2507 locally?
- With Ollama: ollama run hf.co/mistralai/Magistral-Small-2507-GGUF:Q4_K_M (the Q4_K_M build). LM Studio and llama.cpp commands are on this page.
- What licence is Magistral Small 2507 released under?
- Apache 2.0, which allows commercial use.