Can I run Mellum2.1 12B A2.5B Thinking on a 16 GB GPU?

Yes: at Q8_0 it fits fully in 16 GB at 32k context. On the NVIDIA GeForce RTX 5080 it runs at about 113 tok/s (estimated).

Can I run Mellum2.1 12B A2.5B Thinking on a 24 GB GPU?

Yes: at Q8_0 it fits fully in 24 GB at 32k context. On the NVIDIA GeForce RTX 3090 Ti it runs at about 118 tok/s (estimated).

How do I run Mellum2.1 12B A2.5B Thinking locally?

With Ollama: ollama run hf.co/bartowski/Mellum2.1-12B-A2.5B-Thinking-GGUF:Q6_K (the Q6_K build). LM Studio and llama.cpp commands are on this page.

What licence is Mellum2.1 12B A2.5B Thinking released under?

Apache 2.0, which allows commercial use.

Mellum2.1 12B A2.5B Thinking

12.1B total, 2.5B active · 128k context · Apache 2.0 · released 2026-09-20

chatcodingagenticreasoning

In short

Mellum2.1 12B A2.5B Thinking is an open model from JetBrains with 12.1B parameters (2.5B active per token). At Q6_K, Mellum2.1 12B A2.5B Thinking needs about 12 GB of memory at 8k context and 12 GB at 32k, counting the weights, the KV cache and runtime buffers. The cheapest hardware in Canada that runs Mellum2.1 12B A2.5B Thinking fully in memory at 32k context is the Intel Arc A770 16GB, $655 at Amazon, at about 49 tok/s (estimated).

Updated October 15, 2026. Speeds are estimates; how they are worked out.

What to run it on in Canada

Fully in memory at 32k context, priced at retailers today.

Cheapest
Intel Arc A770 16GB
49tok/sest.Q8_0
$655
Best value
Intel Arc A770 16GB
49tok/sest.Q8_0
$655
Fastest
NVIDIA RTX PRO 6000 Blackwell
210tok/sest.Q8_0
$3,133

All hardware that runs it

Everything we price, best value first. Two cards split the model by layers.

Hardware	Quant	Fits	Decode	Price	tok/s per 1,000
Intel Arc A770 16GB	Q8_0	fits	49 tok/s	$655	74.0
AMD Radeon RX 9060 XT 16GB	Q8_0	fits	33 tok/s	$800	41.6
AMD Radeon RX 9070 XT	Q8_0	fits	67 tok/s	$1,050	63.4
Intel Arc Pro B60	Q8_0	fits	40 tok/s	$1,060	37.3
NVIDIA GeForce RTX 5060 Ti 16GB	Q8_0	fits	53 tok/s	$1,120	46.9
AMD Radeon RX 7800 XT	Q8_0	fits	65 tok/s	$1,397	46.5
AMD Radeon RX 7600 XT	Q8_0	fits	30 tok/s	$1,440	20.8
NVIDIA GeForce RTX 5070 Ti	Q8_0	fits	105 tok/s	$1,690	62.1
NVIDIA GeForce RTX 4070 Ti Super	Q8_0	fits	79 tok/s	$1,790	44.0
NVIDIA GeForce RTX 4080 Super	Q8_0	fits	86 tok/s	$2,000	43.2
AMD Radeon RX 7900 XT	Q8_0	fits	83 tok/s	$2,083	39.9
NVIDIA GeForce RTX 5080	Q8_0	fits	113 tok/s	$2,370	47.5
NVIDIA GeForce RTX 3090	Q8_0	fits	110 tok/s	$2,800	39.2
AMD Radeon AI PRO R9700	Q8_0	fits	67 tok/s	$2,800	23.8
NVIDIA RTX PRO 6000 Blackwell	Q8_0	fits	210 tok/s	$3,133	67.1
NVIDIA GeForce RTX 3090 Ti	Q8_0	fits	118 tok/s	$3,999	29.6
GMKtec EVO-X2 Ryzen AI Max+ 395 128GB	Q8_0	fits	27 tok/s	$5,099	5.2
NVIDIA GeForce RTX 4090	Q8_0	fits	118 tok/s	$7,107	16.6
ASUS Ascent GX10 128GB	Q8_0	fits	32 tok/s	$8,299	3.9
NVIDIA GeForce RTX 5090	Q8_0	fits	210 tok/s	$10,300	20.4
Mac Studio M4 Max 128GB	Q8_0	fits	67 tok/s	–	–
Mac mini M4 Pro 64GB	Q8_0	fits	34 tok/s	–	–
AMD Radeon RX 7900 XTX	Q8_0	fits	100 tok/s	–	–
Framework Desktop Ryzen AI Max+ 395 128GB	Q8_0	fits	27 tok/s	–	–
Mac Studio M3 Ultra 512GB	Q8_0	fits	101 tok/s	–	–
NVIDIA DGX Spark 128GB	Q8_0	fits	32 tok/s	–	–
Mac Studio M3 Ultra 256GB	Q8_0	fits	101 tok/s	–	–
Mac mini M4 Pro 48GB	Q8_0	fits	34 tok/s	–	–
NVIDIA GeForce RTX 4060 Ti 16GB	Q8_0	fits	34 tok/s	–	–
Mac Studio M4 Max 64GB	Q8_0	fits	67 tok/s	–	–
Mac Studio M3 Ultra 96GB	Q8_0	fits	101 tok/s	–	–
Framework Desktop Ryzen AI Max+ 395 64GB	Q8_0	fits	27 tok/s	–	–
Mac mini M4 32GB	Q8_0	fits	15 tok/s	–	–
NVIDIA Tesla P40	Q8_0	fits	41 tok/s	–	–
NVIDIA GeForce RTX 3060 12GB	Q6_K	offload	40 tok/s	$497	79.7
Intel Arc B580	Q6_K	offload	34 tok/s	$560	61.4
NVIDIA GeForce RTX 3080 10GB	Q6_K	offload	35 tok/s	$769	45.1
NVIDIA GeForce RTX 5070	Q6_K	offload	63 tok/s	$1,181	53.3
NVIDIA GeForce RTX 4070 Super	Q6_K	offload	51 tok/s	$1,400	36.6

1 other card or machine we price cannot hold it at this context.

Memory needed

Weights, KV cache and runtime buffers at each context length (KV cache at fp16).

Quant	File	@8k	@32k	@64k	@128k
Q6_Kgguf	10.7 GB	11.6 GB	12.0 GB	12.7 GB	13.9 GB
Q8_0gguf	12.0 GB	12.9 GB	13.4 GB	14.0 GB	15.3 GB

Run it

Copy a command for the runtime you use.

Q6_K
- Ollamaollama run hf.co/bartowski/Mellum2.1-12B-A2.5B-Thinking-GGUF:Q6_K
- LM Studiolms get bartowski/Mellum2.1-12B-A2.5B-Thinking-GGUF@q6_k
- llama.cppllama-server -hf bartowski/Mellum2.1-12B-A2.5B-Thinking-GGUF:Q6_K
Q8_0
- Ollamaollama run hf.co/bartowski/Mellum2.1-12B-A2.5B-Thinking-GGUF:Q8_0
- LM Studiolms get bartowski/Mellum2.1-12B-A2.5B-Thinking-GGUF@q8_0
- llama.cppllama-server -hf bartowski/Mellum2.1-12B-A2.5B-Thinking-GGUF:Q8_0

Notes

Thinking model; serve with vLLM using the qwen3 reasoning parser (and hermes tool-call parser for tools). GGUF builds and the MTP head are not yet released.

Questions

How much VRAM does Mellum2.1 12B A2.5B Thinking need?: At Q6_K, Mellum2.1 12B A2.5B Thinking needs about 12 GB of memory at 8k context and 12 GB at 32k, counting the weights, the KV cache and runtime buffers.
What is the best GPU for running Mellum2.1 12B A2.5B Thinking locally?: The cheapest hardware in Canada that runs Mellum2.1 12B A2.5B Thinking fully in memory at 32k context is the Intel Arc A770 16GB, $655 at Amazon, at about 49 tok/s (estimated). The fastest we price is the NVIDIA RTX PRO 6000 Blackwell, at about 210 tok/s.
Can I run Mellum2.1 12B A2.5B Thinking on a 16 GB GPU?: Yes: at Q8_0 it fits fully in 16 GB at 32k context. On the NVIDIA GeForce RTX 5080 it runs at about 113 tok/s (estimated).
Can I run Mellum2.1 12B A2.5B Thinking on a 24 GB GPU?: Yes: at Q8_0 it fits fully in 24 GB at 32k context. On the NVIDIA GeForce RTX 3090 Ti it runs at about 118 tok/s (estimated).
How do I run Mellum2.1 12B A2.5B Thinking locally?: With Ollama: ollama run hf.co/bartowski/Mellum2.1-12B-A2.5B-Thinking-GGUF:Q6_K (the Q6_K build). LM Studio and llama.cpp commands are on this page.
What licence is Mellum2.1 12B A2.5B Thinking released under?: Apache 2.0, which allows commercial use.