Qwen2.5 0.5B

• Cheapest

  NVIDIA GeForce RTX 3060 12GB

  239tok/sest.Q8_0

  309 €

• Best value

  NVIDIA GeForce RTX 3080 10GB

  505tok/sest.Q8_0

  500 €

• Fastest

  NVIDIA GeForce RTX 5090

  1190tok/sest.Q8_0

  6.028 €

• Mac

  Mac mini M4 16GB

  84tok/sest.Q8_0

  800 €

• Q8_0
  • Ollamaollama run hf.co/bartowski/Qwen2.5-0.5B-Instruct-GGUF:Q8_0
  • LM Studiolms get bartowski/Qwen2.5-0.5B-Instruct-GGUF@q8_0
  • llama.cppllama-server -hf bartowski/Qwen2.5-0.5B-Instruct-GGUF:Q8_0

The card says the full context is 32,768 tokens and generation is up to 8192 tokens. Needs transformers 4.37.0 or later, because older versions raise KeyError: 'qwen2'.

How much VRAM does Qwen2.5 0.5B need?

At Q8_0, Qwen2.5 0.5B needs about 1 GB of memory at 8k context and 2 GB at 32k, counting the weights, the KV cache and runtime buffers.

What is the best GPU for running Qwen2.5 0.5B locally?

The cheapest hardware in Germany that runs Qwen2.5 0.5B fully in memory at 32k context is the NVIDIA GeForce RTX 3060 12GB, 309 € at Amazon, at about 239 tok/s (estimated). The best value is the NVIDIA GeForce RTX 3080 10GB (about 505 tok/s for 500 €). The fastest we price is the NVIDIA GeForce RTX 5090, at about 1190 tok/s.

Can I run Qwen2.5 0.5B on a 16 GB GPU?

Yes: at Q8_0 it fits fully in 16 GB at 32k context. On the NVIDIA GeForce RTX 5080 it runs at about 637 tok/s (estimated).

Can I run Qwen2.5 0.5B on a 24 GB GPU?

Yes: at Q8_0 it fits fully in 24 GB at 32k context. On the NVIDIA GeForce RTX 3090 Ti it runs at about 669 tok/s (estimated).

How do I run Qwen2.5 0.5B locally?

With Ollama: ollama run hf.co/bartowski/Qwen2.5-0.5B-Instruct-GGUF:Q8_0 (the Q8_0 build). LM Studio and llama.cpp commands are on this page.

What licence is Qwen2.5 0.5B released under?

Apache License 2.0, which allows commercial use.