# Bonsai 27B Bonsai 27B is an extremely low-bit compression of [[Qwen3.6-27B]] from PrismML, released on 14 July 2026 under Apache 2.0. The 1-bit variant is 3.9 GB. That is a 27B-class model, with reasoning and vision intact, running on an iPhone. For context: the full-precision model is about 54 GB, and a conventional 4-bit build is 18 GB. ## The two variants | | Weights | Effective bits | Size | Benchmark retention | |---|---|---|---|---| | **Ternary** | {−1, 0, +1} + FP16 group scaling | 1.71 | 5.9 GB | 95% of baseline (80.5) | | **1-bit** | {−1, +1} + FP16 group scaling | 1.125 | 3.9 GB | 90% of baseline (76.1) | Retention is measured across 15 benchmarks in thinking mode. Math and coding degrade very little. **Tool calling is where it hurts**: 74.0 ternary and 66.0 for 1-bit, against an 80.0 baseline. If you plan to use this in an agent loop, that is the number that decides whether it works, not the headline average. Speed: up to 163 tok/s (1-bit) and 134 tok/s (ternary) on an RTX 5090; 87 and 58 tok/s on an M5 Max. ## Why "1-bit" needs an asterisk A weight is still a bit. The fractional bit counts come from **group-wise scaling**: a group of weights (128 here) shares one FP16 scale factor, so the per-weight cost averages out to something between 1 and 2 once you amortize the scale. The ternary variant is not binary at all. Its values are {−1, 0, +1}, which is a *trit*, not a bit. Five trits pack into a byte at 1.6 bits each, close enough to the theoretical optimum that squeezing further costs more in unpacking time than it saves in memory. Worth knowing: the current release appears to use llama.cpp's Q2_0 (2 bits per weight plus a scale per group), which wastes one of four available bit patterns on ternary weights. PrismML's fork fixes the group-size mismatch but not the wasted pattern. There is real headroom left. ## Why it matters **The compression is trained in, not bolted on.** This is the distinction that matters and the one most easily missed. Post-training quantization to 2 bits, of the kind you get from an off-the-shelf UD_Q2 build, lands near the 1-bit numbers here while losing more where it counts. A 5% drop in tool-calling reads as small on a table and shows up constantly in real use. Bonsai's low-bit representation runs end to end, with no higher-precision layers anywhere, including a 4-bit vision tower. **On-device stops meaning small.** A 12 GB iPhone has roughly 6 GB usable for a model once you account for KV cache and activations. Until now that ceiling meant models in the single-digit billions. A 27B-class model with a 262K context and multimodal input now fits under it. That is a step change in what "local" means, not an increment. **Compression is becoming its own layer of the stack.** PrismML is a Caltech spinout backed by Khosla, Cerberus, Google, and Samsung, and its entire business is compressing networks without losing reasoning. The pattern to watch: labs train frontier models, specialists compress them, and the gap between "what runs in a datacenter" and "what runs in a pocket" is being attacked by a different set of people than the ones training the models. ## Caveats - **Tool calling degrades most.** Fine for chat, summarization, and reasoning. Test it hard before putting it in an agent - Tooling had not caught up at release. Reports of the GGUF and MLX builds failing in [[LM Studio]], pending llama.cpp and MLX engine updates - Running the 27B on iOS via the Locally AI app requires an iPhone 17 Pro or Pro Max - Benchmarks are PrismML's own, comparing against their own baseline. The relative claim (retains 90-95% of the source model) is the meaningful one, and it is the kind of claim that is easy to verify independently. Worth checking before relying on it - Speedups from low-bit formats are not automatic. Unpacking ternary encodings costs cycles, and several people report no gain over a straight Q4 quantization of the same base model ## References - Announcement — https://prismml.com/news/bonsai-27b - Models — https://huggingface.co/prism-ml/models - Hacker News discussion (706 points) — https://news.ycombinator.com/item?id=48910545 ## Related - [[Qwen3.6-27B]] - [[Qwen]] - [[AI Quantization]] - [[AI Open Weight Models]] - [[Running AI Models Locally]] - [[Large Language Models (LLMs)]] - [[AI Tool Use]] - [[llama.cpp]] - [[MLX]] - [[LM Studio]] - [[Ollama]] - [[AI Foundation Models]] - [[Kimi K3]]