Qwen2.5 0.5B

• Cheapest

  NVIDIA GeForce RTX 3060 12GB

  353tok/sest.Q8_0

  309 €

• Best value

  NVIDIA GeForce RTX 3080 10GB

  746tok/sest.Q8_0

  500 €

• Fastest

  NVIDIA GeForce RTX 5090

  1759tok/sest.Q8_0

  6.028 €

• Mac

  Mac mini M4 16GB

  124tok/sest.Q8_0

  800 €

• Q8_0
  • Ollamaollama run hf.co/bartowski/Qwen2.5-0.5B-Instruct-GGUF:Q8_0
  • LM Studiolms get bartowski/Qwen2.5-0.5B-Instruct-GGUF@q8_0
  • llama.cppllama-server -hf bartowski/Qwen2.5-0.5B-Instruct-GGUF:Q8_0

The card says the full context is 32,768 tokens and generation is up to 8192 tokens. Needs transformers 4.37.0 or later, because older versions raise KeyError: 'qwen2'.

How much VRAM does Qwen2.5 0.5B need?

At Q8_0, Qwen2.5 0.5B needs about 1 GB of memory at 8k context and 1 GB at 8k, counting the weights, the KV cache and runtime buffers.

What is the best GPU for running Qwen2.5 0.5B locally?

The cheapest hardware in Germany that runs Qwen2.5 0.5B fully in memory at 8k context is the NVIDIA GeForce RTX 3060 12GB, 309 € at Amazon, at about 353 tok/s (estimated). The best value is the NVIDIA GeForce RTX 3080 10GB (about 746 tok/s for 500 €). The fastest we price is the NVIDIA GeForce RTX 5090, at about 1759 tok/s.

Can I run Qwen2.5 0.5B on a 16 GB GPU?

Yes: at Q8_0 it fits fully in 16 GB at 8k context. On the NVIDIA GeForce RTX 5080 it runs at about 942 tok/s (estimated).

Can I run Qwen2.5 0.5B on a 24 GB GPU?

Yes: at Q8_0 it fits fully in 24 GB at 8k context. On the NVIDIA GeForce RTX 3090 Ti it runs at about 989 tok/s (estimated).

How do I run Qwen2.5 0.5B locally?

With Ollama: ollama run hf.co/bartowski/Qwen2.5-0.5B-Instruct-GGUF:Q8_0 (the Q8_0 build). LM Studio and llama.cpp commands are on this page.

What licence is Qwen2.5 0.5B released under?

Apache License 2.0, which allows commercial use.