Can I run Granite 4.0 H Small?

Granite 4.0 H Small is IBM's mixture-of-experts model: 32B parameters on paper, but only about 9B active per token. That split is the whole story. It runs fast for its size because each token touches a fraction of the weights, but you still have to hold the full model in memory, so plan around the 32 GB minimum rather than the active count. At a 4-bit quant it lands around 19.4 GB, which rules out a 12 GB card like an RTX 3060 entirely and points you instead at a 24 GB GPU or an Apple Silicon Mac with plenty of unified memory. It is aimed at chat and coding work, not casual experimenting on a laptop.

On an RTX 4090 it moves at roughly 157 tokens per second, which is well past reading speed and makes it pleasant for interactive coding sessions. On an M-series Max you are looking at around 64 tok/s, still comfortable, and pure CPU on DDR5 drops to about 9 tok/s, usable for batch jobs but not live chat. The 128K context is genuine, but it is expensive: fill it and total memory climbs to roughly 50.7 GB, so on a 24 GB card you cannot actually run near the ceiling. Keep working context modest unless you have a 64 GB Mac to spare.

Against Qwen 3 32B, a dense model of similar total size, Granite generally feels lighter on hardware thanks to the MoE design, though Qwen 3 tends to have the edge on heavier reasoning since it activates all its parameters every token. If you want something smaller and simpler, the dense Granite 3.3 8B is the easier fit on modest GPUs. Granite 4.0 H Small's standout trait is being a properly fast 32B-class model you can actually self-host, and it ships under Apache 2.0, so you are free to use it commercially with no license worries.