Can I run Gemma 4 26B A4B?

Gemma 4 26B A4B is Google's mixture-of-experts entry, and the trick behind it is the naming: it holds 25.2B parameters total but only routes 3.8B of them per token. That means it generates at the speed of a small model while needing memory for the whole thing. At a 4-bit quant it lands around 15.3 GB, and you want at least 24 GB of RAM to load it comfortably. That rules out a 12 GB card like the RTX 3060, where it simply does not fit, and points you at a 24 GB GPU or an Apple Silicon machine with generous unified memory.

In daily use the MoE design pays off. On an RTX 4090 it runs at roughly 372 tokens per second, which is well past the point of streaming faster than you can read, and an M-series Max still turns in about 151 tok/s. CPU-only on DDR5 drops to around 22 tok/s, usable for batch work but not interactive chat. The context window is a generous 256K, but treat that as a ceiling: at 128K of context the full memory footprint climbs to about 43.4 GB, so on a 24 GB setup you keep working context modest unless you drop to the 10.6 GB q2 build.

Against Mistral Small 3.1 24B, a similarly-sized dense vision model, Gemma 4 generally feels quicker per token thanks to the sparse routing, though dense models of that class can hold an edge on the hardest single-pass reasoning. Its standout trait is that speed-to-capability ratio plus genuine vision and coding support in one package. The license is Apache 2.0, so you can use it commercially and in production without provider-specific restrictions, which is rare at this size and a real point in its favor. Pull it with ollama as gemma4:26b.