Blog
Engineering notes and field reports from the team building the Inference Server — private AI on the hardware you already own, with every claim tied to a published measurement.
measured head-to-head+48–62% decode 2B: 14/16 vs 10/16
Same size, quarter the bytes: Qwen 3.5 vs Ternary Bonsai at 2B, 4B and 9B
October 2026 · 5 min read
Both families, same machine, same session, page methodology: the ternary
models decode 48–62% faster than their same-size Qwen 3.5 siblings —
the 9B bonsai outruns the 4B Qwen — and on a 16-question spot check the 2B
bonsai’s thinking beats plain q35-2b 14/16 to 10/16. Plus the two
measurement gotchas: short budgets truncate inside reasoning, and terse
extraction wants enable_thinking:false.
new in v1.70.52 GB thinking model 2B / 4B / 9B · images
A thinking model in half a gigabyte: Ternary-Bonsai-2 small (2B/4B/9B)
October 2026 · 5 min read
The ternary family now starts at 0.52 GB — and the 9B fits an 8 GB host in 2.42 GB with the full 262K context. Thinking ON by default, image questions via F16 vision projectors (v1.7.1), 423/424-token teacher-forced parity vs the reference fork, 62.5 tok/s decode for the 2B on an M4 (+48% vs q35-2b). The deploy-anywhere reasoning tier.
new in v1.7.0POST /v1/systemone accuracy tier all questions · one prefill
Clef-Flash: the accuracy tier for local decisions, every question scored in one prefill
October 2026 · 5 min read
Cloudflare’s 9B decision model joins strands-decider on the same API: a joint schema head scores every option of every question in a single prefill, text + images at native resolution (it routes the error-screenshot fixture to billing at 0.9636). Q8 serves within 0.001 of f16; BFCL 98.8 upstream. Same request body — switch tiers by model id.
images on /v1/systemonetyped questions · zero decode calibrated probabilities
Show, don't tell: answering questions about images with a calibrated decision model
October 2026 · 6 min read
Attach a screenshot, a scanned form, or a photo to a decision request and threshold the answer: in real measured runs, a payment-error screenshot routed to billing at 0.9449 (the model read "card declined by issuer" in the image — not just the pixels), and a claim form with a blank signature landed completeness at 0.5598 — squarely in the confirm band where a calibrated gate catches it. Three worked examples, unedited outputs, all local.
new in v1.6.0POST /v1/evaluate score · systemone · evaluatecalibrated probabilities
Jev-style models on your own box: /v1/score, /v1/systemone, and the new /v1/evaluate
October 2026 · 7 min read
Jev-style decisioning reads the answer off the logits instead of generating
text: one prefill, a restricted softmax over just the allowed answers, a
probability you can threshold. The server now ships the full local
stack — /v1/score for classify/route (~355 ms),
/v1/systemone for calibrated typed decisions (ECE 0.054), and new
in v1.6.0 /v1/evaluate for grading answers against evidence:
deterministic validators plus five judged rubrics, with a measured injection
fixture staying at p(support) 0.248. Worked curl examples for all three.
console Pipeline tabsystemone → chat confidence gates: act / confirm / escalate
Decision pipelines: chaining calibrated decisions into generation, worked end-to-end
October 2026 · 6 min read
Every agent flow is secretly "decide, then generate." The console now ships two runnable pipelines: route-then-generate (strands-decider picks the team and urgency in one request, a confidence gate applies ≥0.90 act / 0.50–0.90 confirm / <0.50 escalate, then a chat model drafts the reply) and a three-stage search query understanding chain (gibberish gate, language, intent — then spelling correction and synonym expansion). Real JSON, measured latency budgets (~1.8 s per decision, extra questions nearly free), honest "~6 s — use it async" framing.
new in v1.5.027B thinking · 7.2 GB file 10.8 tok/s M4 (v1.6.0) · 5.2 Gravitontoken-exact parity
Ternary Bonsai: a 27B reasoning model in 8 GB of RAM, on your CPU
October 2026 · 6 min read
A 27B thinking model whose weights are just −1, 0, +1 (1.72 bits each, trained that way, with a runtime Hadamard rotation) now serves natively from a 7.2 GB file in ~8 GB resident — on a 16 GB CPU host. Our fused kernel decodes the ternary stream inline: 10.8 tok/s on an M4 (as of v1.6.0; 8.9 at launch) vs 1.3 from the upstream fork's own CPU lane, token-for-token identical to the reference on greedy decoding. With the honest caveats: slower at long context, text-only this release.
new in v1.5.0POST /v1/systemone calibrated probabilitiesJevBench ties the GPU reference
A calibrated decision model, 100% local: strands-decider on /v1/systemone
October 2026 · 7 min read
Typed decisions — true/false, pick-one, ordered score — over text, JSON records, and screenshots, with probabilities honest enough to gate automation: act ≥0.9, confirm 0.5–0.9, escalate below. On the public JevBench set the official harness ran unchanged against our CPU server: 72.29%, identical to the reference GPU stack (calibration error 0.054 vs 0.052). Working curls for the payout check, multi-question requests, and vision.
new in v1.4.0Nemotron-Parse 2.0 image → MarkdownCPU · 8 GB
Document parsing on CPU: page images to structured Markdown with Nemotron-Parse 2.0
September 2026 · 5 min read
NVIDIA's sub-1B document-OCR model now serves natively: a page image in, structured Markdown out — text with layout classes and bounding boxes, from one gguf on an 8 GB host. Same OpenAI-compatible endpoint; a chart parsed into a clean table, byte-consistent on Arm and x86.
new in v1.4.0/v1/score one prefill · no decodedeterministic
Ten things to decide on-box: fast local classification with /v1/score
September 2026 · 6 min read
Route tickets, gate actions, flag sensitivity, pick a tool — a decision plus a probability distribution in a single prefill with no decode loop, fully deterministic, ~355 ms on a 4B CPU model (and ~84 ms on an exact repeat). Ten concrete local-decisioning use cases on the Qwen 3.5 and Spark families.
new in v1.4.0score → generate agent workflows4B on CPU
Decide, then generate: pairing /v1/score with a 4B model for fast agent workflows
September 2026 · 5 min read
Use the deterministic /v1/score decision to route or gate, then hand off to a 4B chat model (Spark-X2.5 or Qwen 3.5) for the actual generation — decide cheaply, generate only on the path you chose. A worked support-triage example, both calls on one private server.
releasenew in v1.3.0 2× team throughputanswers unchanged
v1.3.0: double your AI throughput — no new hardware, no new risks
September 2026 · 5 min read
A pure speed release: deep document analysis with Flash-Next finishes in half the time, one server handles a whole team at 2.2× the throughput with Spark, and chat follow-ups start near-instantly. Every answer is guaranteed identical to the previous version — so validated pipelines, reports, and extractions need zero re-testing. Which use cases benefit most, with measured examples.
125B MoE · 6B activeone 128 GB CPU host thinking · reasoning_contentnew in v1.2.0
One token per second, on purpose: 125B-class reasoning on a single CPU box
September 2026 · 6 min read
Qwen3.8-Flash-Next — a 125B mixture-of-experts thinking model with ~6B active parameters — now serves natively from one 128 GB-RAM CPU host, air-gapped, no GPU cluster. Now ~5.8 tok/s as of v1.6.0 (launched at ~1 tok/s — a 5.9× fused-kernel fix), it has grown from async workhorse toward reading-speed: overnight contract analysis, research queues, agent planning, second-opinion review — while the new Spark-X2.5 small models (26–55 tok/s) handle interactive traffic on the same server, routed by model id. Honest framing, verified token-parity, one 79 GB file.
new modelsApache 2.0 ~55 tok/s on CPUnew in v1.2.0
Spark-X2.5: fast, cheap, private inference — 55 tok/s on a CPU
September 2026 · 6 min read
XHToken's Apache-2.0 Spark-X2.5 models (4B and 1.7B) now serve natively: ~55 tok/s decode for the 1.7B and ~26 tok/s for the 4B on Apple-M4-class CPUs, ~177 tok/s near-llama.cpp-parity prefill, and token-exact greedy parity (45/45) on identical files. A per-request thinking toggle gives instant direct answers, the 1.7B fits a 4 GB host, and six fast-inference use cases show where this tier earns its keep — including as the executor half of a flash-next planner.
model routingQwen3.5 family same task, 3× fasterprompts buy a tier down
Right-size the model: one Qwen3.5 family, three speed tiers, one server
September 2026 · 5 min read
The same executive summary: 105.6 tok/s on 0.8B vs 32.5 on 4B — identical quality. Where the small model breaks (strict JSON schemas), an optimized prompt fixes it and buys the tier back. A measured routing table for classification, summaries, extraction, agents, and drafting — all sizes serving side-by-side from one CPU box, routed by one request field.
new section6 use cases measured on CPUcopy-paste walkthroughs
Six things enterprises actually do with a private AI server — measured, end to end
September 2026 · 4 min read
Announcing the use-cases section: document extraction (33 s), RAG that files the ticket itself (58 s), compliance drafts with clause citations (64 s), log triage (51 s), write-and-run code (66 s), and assistants that remember across sessions (2.6 s recall) — every walkthrough validated on a CPU-only box before publishing, caveats included.
memory APIsessions hybrid recallnew in v1.1.0
Give your local LLM a memory: sessions + long-term recall, built into the server
September 2026 · 6 min read
Add one "session" field and the server keeps the
conversation; add a user and durable facts follow that user
across sessions — extracted in the background by a small model,
recalled by hybrid vector + keyword search, all on your own hardware.
With measured numbers: 622 ms follow-up turns on CPU, session context
surviving restarts 8× faster, and a killed cluster node whose
conversation a peer continued with full history.
AI agentspi.dev runs on CPU · 4Bstep-by-step + measured
Build private AI agents with pi + the Inference Server
September 2026 · 7 min read
A 20-line provider file connects the open-source pi coding agent to your private endpoint — then real agent work happens on a CPU box with the 4B model: contract→JSON extraction (33 s), batch folder pipelines (43 s), log triage with grep-verified incidents (51 s), and code the agent writes and executes (66 s). Every command and timing in the guide is from a real run, including the honest small-model caveats.
image editing/v1/images/edits 20B editor · 24 GB VRAMbefore/after samples
Edit images with plain English on your own box
August 2026 · 5 min read
"Turn the scene into a warm golden sunset." Instruction-based image editing behind a private, OpenAI-compatible endpoint: a 20B Qwen-Image-Edit-class editor on the same $1.21/hour GPU box as our benchmark — with real before/after samples, per-edit timings (~3–4.5 min measured), and honest RAM/VRAM sizing.
GPU accelerationBACKEND=cuda AWS g5.2xlarge9 models measured
One flag, 4× the tokens: GPU acceleration measured on AWS
August 2026 · 6 min read
Append BACKEND=cuda to the install line and the same binary, models, and API serve from the GPU. On a $1.21/hour g5.2xlarge (NVIDIA A10G), the Qwen 3.8 27B reasoner goes from 6.5 to 26 tokens/sec — and all nine measured models gain 3–4×. Exact instance settings, the full comparison table, and honest VRAM sizing for a 24 GB card.
Windows nativeone PowerShell line no Docker · no WSLstep-by-step
Install private AI natively on Windows — no Docker, no WSL
August 2026 · 5 min read
A native Windows x64 build of the inference server installs with one elevated PowerShell line and runs at boot as a Scheduled Task — same engine, models, and OpenAI-compatible API as the Linux builds. The guide walks the Windows-specific steps honestly: SmartScreen on the unsigned binary, endpoint allow-listing, the firewall rule for LAN serving — plus exactly what the single-node, CPU-only edition does and doesn't support.
embeddingsreranking text + images, one vector spaceOpenAI & Jina compatible
Search that sees: multimodal embeddings and reranking on your own box
August 2026 · 6 min read
Two new retrieval models complete the private RAG stack: /v1/embeddings (OpenAI-compatible, Matryoshka dimensions 64–2048) and /v1/rerank (Jina/Cohere-compatible) — with text and images in one vector space, so a text query finds a screenshot. Embed, recall, rerank, and generate on one CPU box, with captured similarity and score numbers from a fresh install.
OpenAI-compatiblemigration two-line changeSDK · streaming · tools
Point your OpenAI code at your own box: a drop-in migration
August 2026 · 5 min read
Migrating off a per-token API is a base URL and key change — your OpenAI SDK, streaming, tools, vision, and JSON mode all keep working. Real captured output from the unmodified OpenAI Python SDK pointed at a local box, with before/after diffs for Python, Node, curl, and a zero-code env-var path.
clusteringno external LB smart routingops
Scale from one box to a fleet: model-aware, load-aware, cache-aware routing — no load balancer
August 2026 · 6 min read
Outgrow one box by adding nodes — same one-line install, one shared secret, no external load balancer and no coordinator. How the built-in routing (model-aware → load-aware → cache-aware) spreads load and lets nodes specialize behind one endpoint, with the exact join/verify commands and the measured 3-node scaling.
function callingagents Qwen 2BMCP-compatible
Function calling on a 2B model: build a local agent that actually runs tools
August 2026 · 6 min read
Every model the server ships does OpenAI-style function calling — even the 2B. A worked multi-step agent loop with real captured output, the exact API calls, the tool_choice trick that forces a small model to call a tool instead of guessing, and how MCP wires in (it's client-side, not an engine flag). All local, all free.
videosummarize + caption Qwen visionpublishing
Understand video with the local models — summarize, caption, and index clips on your box
August 2026 · 5 min read
The bundled vision models read video, not just stills — frames over time. Real captured output turning sample clips into step lists, captions, and summaries, with the exact API calls, so you can summarize, caption, and index video for publishing entirely on your own box — no upload, no per-minute fee.
speech-to-texttext-to-speech local & privatevoice cloning
Speech in, speech out: ASR and TTS with the local models
August 2026 · 5 min read
Every install ships speech-to-text and text-to-speech that run entirely on your box. Real captured transcripts, a synthesized sample to listen to, the exact API calls, the round-trip voice pattern, and file-based voice cloning — no cloud speech API, no per-minute fee, no audio leaving the network.
visiondocument extraction Qwen 2B / 4Bfinancial services
Read documents with the vision models that ship free — and get clean JSON out
August 2026 · 6 min read
The Qwen 2B and 4B vision models bundled with the server read forms, rate tables, pharmacy labels, and checks accurately — locally and free. A worked example with real captured output, the read-then-structure pattern that turns image content into clean JSON your code can use, and four synthetic sample images to download and try on your own documents.
prompt testingworked example local inferencetoken savings
Two prompts, one small model: a worked example of testing locally to cut tokens and latency
August 2026 · 6 min read
A hands-on before/after with real captured runs: the same 2B local model, two prompts. A sharper prompt turns rambling prose into a parseable answer — dropping a classification response from 120 output tokens to 1 and 2.7 s to 0.3 s, and recovering line items a vague prompt silently lost. Plus why the iterate-for-free, test-what-you-ship economics only work when the model runs locally.
prompt engineeringsmall models console workbenchbenchmarks
Optimize the prompt, shrink the model: a practical workflow for efficient AI at scale
August 2026 · 8 min read
When outputs disappoint, most teams reach for a bigger model — the most expensive possible fix. On our 380-prompt, 13-industry evaluation the quality gap between 4B-class and 27B-class models is a few points; the cost gap is an order of magnitude per served user. This article shows the compare → optimize → re-compare workflow using the console's built-in workbench: side-by-side model comparison (local or against a frontier API), a one-click Prompt Optimizer, and the prompt patterns that consistently let small models match big ones on your task.
RAGAWS EC2 cost analysisbenchmarks
Run private RAG on one EC2 box — and stop paying by the token
August 2026 · 10 min read
Retrieval-augmented generation re-sends the same tokens over and over — and per-token APIs bill you every time. We put the model inside the VPC on a fixed-price Graviton instance instead, benchmarked it with a 380-prompt, 13-industry test pack, and published the math: measured latency, a side-by-side monthly cost comparison with the assumptions on the table, and the one-config-line clustering story for scale-out.