Can I run Granite 4.0 H Small?
Granite 4.0 H Small by IBM needs around 32 GB of RAM at the recommended 4-bit quantization (19.4 GB download). Your hardware is checked below — instantly, nothing leaves your browser. Expect roughly ~64 tok/s on a Apple M-series Max.
Reading your hardware signals…
Real-world notes
Granite 4.0 H Small is IBM's mixture-of-experts model: 32B parameters on paper, but only about 9B active per token. That split is the whole story. It runs fast for its size because each token touches a fraction of the weights, but you still have to hold the full model in memory, so plan around the 32 GB minimum rather than the active count. At a 4-bit quant it lands around 19.4 GB, which rules out a 12 GB card like an RTX 3060 entirely and points you instead at a 24 GB GPU or an Apple Silicon Mac with plenty of unified memory. It is aimed at chat and coding work, not casual experimenting on a laptop.
On an RTX 4090 it moves at roughly 157 tokens per second, which is well past reading speed and makes it pleasant for interactive coding sessions. On an M-series Max you are looking at around 64 tok/s, still comfortable, and pure CPU on DDR5 drops to about 9 tok/s, usable for batch jobs but not live chat. The 128K context is genuine, but it is expensive: fill it and total memory climbs to roughly 50.7 GB, so on a 24 GB card you cannot actually run near the ceiling. Keep working context modest unless you have a 64 GB Mac to spare.
Against Qwen 3 32B, a dense model of similar total size, Granite generally feels lighter on hardware thanks to the MoE design, though Qwen 3 tends to have the edge on heavier reasoning since it activates all its parameters every token. If you want something smaller and simpler, the dense Granite 3.3 8B is the easier fit on modest GPUs. Granite 4.0 H Small's standout trait is being a properly fast 32B-class model you can actually self-host, and it ships under Apache 2.0, so you are free to use it commercially with no license worries.
Specifications
Size by quantization
| Quantization | Bits/weight | Download | Min RAM | Quality |
|---|---|---|---|---|
| Q2_K | 3.35 | 13.4 GB | 24 GB | Noticeable loss |
| Q4_K_MRecommended | 4.85 | 19.4 GB | 32 GB | Recommended |
| Q5_K_M | 5.65 | 22.6 GB | 32 GB | High |
| Q8_0 | 8.5 | 34.0 GB | 48 GB | Near-original |
| F16 | 16 | 64.0 GB | 96 GB | Original |
Sizes are estimates from parameter count × bits per weight; real GGUF builds vary slightly. · Data updated: 2026-06-11 · How we calculate these numbers →
Memory needed by context length
| Context | KV cache (est.) | Total memory (Q4) |
|---|---|---|
| 4K tokens | ~1.0 GB | ~20.4 GB |
| 8K tokens | ~2.0 GB | ~21.4 GB |
| 32K tokens | ~7.8 GB | ~27.2 GB |
| 128K tokens | ~31.3 GB | ~50.7 GB |
The KV cache grows with context length — a model that fits at 4K can run out of memory at 32K. Estimates assume an FP16 cache with grouped-query attention; actual usage varies by runtime.
Estimated speed by hardware
| Hardware | Bandwidth | ~Speed |
|---|---|---|
| NVIDIA RTX 3060 12GB | 360 GB/s | Won't fit in VRAM |
| NVIDIA RTX 4090 24GB | 1008 GB/s | ~157 tok/s |
| Apple M-series (base) | 100 GB/s | ~16 tok/s |
| Apple M-series Pro | 270 GB/s | ~42 tok/s |
| Apple M-series Max | 410 GB/s | ~64 tok/s |
| CPU only (dual-channel DDR5) | 60 GB/s | ~9 tok/s |
Token generation is memory-bandwidth bound: tok/s ≈ bandwidth × 0.85 ÷ model size at Q4. Real-world numbers vary by runtime and context length.
Run it locally
The easiest path is Ollama — one command and you're chatting:
ollama run granite4:32b-a9b-h