Ternary Bonsai: a 27B reasoning model in 8 GB of RAM, on your CPU

v1.5.0 adds ternary-bonsai-27b — prism-ml's Ternary-Bonsai-2-27B, a 27-billion-parameter thinking model whose weights are stored at an average of 1.72 bits each. The entire model is a 7.2 GB file that runs in about 8 GB of resident memory, which means a 27B-class reasoning model — with a visible reasoning_content channel and a reasoning_effort dial — now fits a 16 GB CPU host. No GPU, no quality-destroying self-quantization: this is the format the model was trained for, served with a CPU kernel we wrote for it.

The numbers in this post are measured: 8.9 tok/s decode on an Apple M4 at short context at launch (10.8 tok/s as of v1.6.0), 5.2 tok/s on an AWS Graviton c9g.8xlarge installed from the public channel, and token-for-token identical output to the upstream reference implementation on greedy decoding.

Most quantized LLMs squeeze 16-bit weights down to 4 or 5 bits after training. Ternary models go much further: every weight in the big matrices is one of just three values — −1, 0, or +1 — scaled by one shared fp16 multiplier per block of 128 weights. Packed, that lands at the advertised ~1.72 bits per weight across the model, which is how 27B parameters fit in 7.2 GB.

You can't get away with that by rounding an ordinary model — the information loss would be catastrophic. Two things make it work here:

• It's trained ternary. The released weights were optimized in this format. (The upstream card reports 98.2% of the FP16 model's scores across their thinking benchmarks — their measurement, not ours.)
• A Hadamard transform at runtime. Before every ternary matrix multiply, the activations pass through a fast Walsh–Hadamard transform (a 1024-point "mixing" rotation, plus per-channel sign flips). The rotation spreads each value's information across many channels, so outlier activations — the thing that normally kills extreme quantization — get smeared out before they ever meet the ternary weights. The transform is mandatory: the weights only make sense in the rotated space.

So serving this model isn't "load gguf, go" — the engine has to apply the rotation at the right seven points of the forward pass, invert it at the embedding, and decode the 2-bit weight stream. That's the port.

The upstream release ships a llama.cpp fork whose Metal (Apple GPU) lane is quick — ~36 tok/s on an M4 — but whose CPU lane decodes at ~1.3 tok/s on that same machine. For a format whose whole promise is "big model, small cheap host", that left most of the opportunity on the table: at 7.2 GB of weights, a laptop-class memory system should be able to stream the model many times per second.

Our first attempt — unpacking the ternary weights into our usual int8 decode packs — worked but needed 26 GB resident, defeating the point of a 1.72-bit model. The shipped kernel instead decodes the 2-bit weight stream inline, straight out of the memory-mapped file, multiplying against int8-quantized activations with exact integer math. Nothing is unpacked ahead of time:

• 8.9 tok/s decode on an Apple M4 at short (~100-token) context at launch — 10.8 tok/s as of v1.6.0,
• 5.2 tok/s on an AWS Graviton c9g.8xlarge, measured on a fresh install from the published channel during the v1.5.0 release gate,
• ~8 GB resident, model load in under 10 seconds.

That's roughly 7× the upstream CPU lane on identical hardware — and it changes what the model is for. At 1.3 tok/s a 27B thinker is a curiosity; at 5–11 tok/s it's a usable private reasoning tier on a 16 GB box.

Aggressive quantization earns extra skepticism, so the port gate was token-exact parity: we captured greedy (temperature 0) completions from the upstream fork's own server on the same gguf file, then required our engine to reproduce the exact token sequence — not "similar text", the same tokens. The shipped kernel passes 4/4 parity fixtures token-for-token, plus 8/8 chat-template goldens (including thinking-off and reasoning_effort variants), and the full multi-model regression gate ran with zero failures before release.

In other words: whatever you think of 1.72-bit weights, our serving of them is bit-faithful to the reference implementation. The quantization trade-off is the model's published one, not something our kernel adds.

On a 16 GB (or larger) Linux host:

curl -fsSL https://inference-server.searchblox.com/install | sudo bash
sudo ./fetch-models.sh ternary-27b     # 7.2 GB download
# already installed? just fetch the group and restart the service

Then ask it something — and note enable_thinking, because thinking defaults off for this model:

curl -s http://127.0.0.1:8081/v1/chat/completions \
  -H "Authorization: Bearer $SEARCHAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"ternary-bonsai-27b",
       "messages":[{"role":"user","content":"Why is the sky blue?"}],
       "enable_thinking":true, "max_tokens":768}'

The response carries the model's deliberation in reasoning_content and the answer in content, like our other thinking models. You can pass reasoning_effort (low / medium) to trim deliberation — but budget max_tokens generously either way: even at low effort this model routinely spends ~360 tokens thinking before it answers. A too-small max_tokens gets you an empty content with the budget consumed by reasoning.

• Decode slows as context grows. The short-context figures above are the best case; across a 414-token generation the M4 averaged ~5.3 tok/s. Long documents plus long reasoning chains will sit at the lower end. (This is a known optimization target, not a format limit.)
• Text-only in this release. Upstream publishes a vision projector for the model and we mirror it alongside, but our loader doesn't support it yet — image requests return a clean error. The catalog lists the model as text-only.
• It's a thinker, not a chatbot. Per-answer latency is dominated by reasoning tokens; at 5–9 tok/s that's tens of seconds per considered answer. Use it where you'd use our other reasoning tiers: analysis, review queues, agent planning — with the fast small models handling interactive traffic on the same server.
• Never self-requantize other models to this format. The ternary types only work for checkpoints trained in them; converting an arbitrary gguf produces silent garbage. Use the published file.

Model: prism-ml Ternary-Bonsai-2-27B (Apache 2.0), served as ternary-bonsai-27b from the ternary-27b fetch group. Measurements: M4 numbers on Apple M4 (release build); Graviton number from the v1.5.0 fleet gate on c9g.8xlarge installed from the public channel; upstream CPU / Metal comparison numbers measured with the upstream fork on the same M4 and gguf file.