Measured Performance

Architecture	Node class tested	Notes
Arm — AWS Graviton4	c9g · 32 cores / 64 GB · 3.3 GHz	Fastest CPU decode per node (highest memory bandwidth); one binary serves all Arm hosts
Intel Xeon	c8i · 32 vCPU / 64 GB · 3.9 GHz	Same x86_64 binary as AMD; strong prompt processing
AMD EPYC (Turin)	c8a · 32 cores / 64 GB · 2.6 GHz	Same x86_64 binary as Intel; current AMD generation
NVIDIA GPU	L4 (24 GB) measured · add-on also runs A100, A10/A10G, RTX 40-series, H100	One-flag install add-on; ~10× CPU speeds at ~96% of the hardware's practical ceiling

Two binaries cover every deployment: one for Arm, one for x86_64 (Intel and AMD). Clusters can mix architectures freely — see cluster scaling below.

Model	Node	Pack quality (PASS / 380)	Avg response	Median (p50)
Gemma 26B A4B mixture-of-experts	AMD c8a	329	9.8 s	7.1 s
q35-4b (flagship default)	Arm c9g	325	6.7 s	4.7 s
Qwen 3.8 27B (thinking)	Arm c9g	321	29.3 s	22.9 s
Gemma E4B	Intel c8i	320	9.4 s	6.8 s
Gemma 12B (media flagship)	Arm c9g	319	15.5 s	12.6 s
Gemma E2B	Intel c8i	317	14.2 s	11.1 s
q35-2b	Intel c8i	311	4.8 s	3.1 s
q35moe-35b	AMD c8a	25/25 sample (incl. vision)	~102 s	~97 s

Every model ran the same 380-prompt pack (13 industries, all APIs) on the release binaries, greedy decoding, timed as complete round trips; quality is the pack heuristic's PASS count out of 380 (each model additionally skips ~35–39 prompts whose modality it does not serve). Headlines: Gemma 26B tops the catalog on quality at 3× the 27B's speed and about a third of its memory; q35-4b keeps the zero-error speed crown with identical pass/fail counts on Arm (6.7 s avg), AMD (7.8 s), and Intel (8.2 s); Gemma E4B has closed to within ~1.15× of q35-4b's response times as a full-multimodal alternative.

Qwen 3.8 27B guidance: this release completes the full pack on both architectures for the first time, with zero errors. It ends the run near 58 GB resident — run it on a 64 GB node; the prefix-cache byte budget is tunable via SAI_PFX_MAX_MB if you need to trim its envelope. Other bounded envelopes measured on the same run: Gemma E4B 9.7 GB, Gemma 26B 18.7 GB, 35B-MoE 27.2 GB.

Model	CPU · Arm (Graviton)	CPU · Intel	CPU · AMD	GPU · NVIDIA L4
0.8B chat	114	71	68	—
2B chat + vision	68	40	50	138
4B chat + vision (recommended)	34	18	35	70
9B chat + vision	21	12	18	—
Gemma E2B	39	26	33	—
Gemma E4B	26	16	21	—
Gemma 12B media flagship	12	—	—	—
27B hybrid reasoning	6.5	—	—	—
35B mixture-of-experts	13–19	—	—	—

Single-stream warm decode (256-token answer, greedy), quantized models, 8-bit KV cache, native kernels; measured on 32-vCPU nodes (Arm Graviton, Intel c8i, AMD c8a). The August 2026 releases stream decode weights as interleaved 4–6-bit packs on Arm and VNNI-capable x86 (the installer default), lifting decode 25–38% on every architecture — measured 4B: AMD 28→35, Arm 25→34, Intel 14→18. Decode follows memory bandwidth: it is the #1 hardware lever. A GPU node serves the same API roughly 10× faster.

Gemma 4 low-RAM tier (SAI_G4_W4=only)	Decode (tok/s)	Memory
Gemma E2B	30	fits 8 GB hosts (~2.8 GB RSS)
Gemma E4B	16	fits 16 GB hosts (~4.4 GB RSS)
Gemma 12B (media flagship)	10.4	~7 GB RSS

The W4 profile streams the original 4-bit weights directly — instant weight load (no pack-build step) and pack-parity answer quality, at a fraction of the memory: the 12B media flagship runs in ~7 GB resident. Enable with SAI_G4_W4=only in the service environment; single-stream greedy decode, measured on the Apple-silicon (M4) build. Hosts with RAM to spare can add the interleaved decode packs (SAI_G4_Q4IL=1 — the default on Arm Linux installs) for ~25–34% faster decode at roughly the model's weight size in extra resident memory (12B: 13.9 tok/s at ~13 GB); low-RAM deployments on Arm should set SAI_G4_Q4IL=0 to keep the minimal footprint.

v1.2.0 thinking models	Decode (tok/s)	RAM class	Download
Spark X2.5 1.7B (spark-x25-1.7b)	~55	4 GB	~1.0 GB
Spark X2.5 4B (spark-x25-4b)	~26	8 GB	~2.4 GB
Qwen 3.8 Flash-Next 125B MoE, 6B active (qwen38-flash-next-q2)	~1	128 GB	~73 GB

The v1.2.0 additions are thinking models: every response carries its chain of thought as reasoning_content alongside the answer. Single-stream greedy decode measured on the Apple-silicon (M4-Max-class) Arm build. The Apache-2.0 Spark X2.5 pair brings reasoning to 4–8 GB edge hosts; Flash-Next serves frontier-scale 125B knowledge at 6B-active-parameter cost (UD-Q2_K_XL quantization) on a single 128 GB-class node — a batch/agent workhorse rather than an interactive chat tier.

This release batches prompt processing across the model's layers, making time-to-first-token 3.3–3.7× faster on long prompts — measured across models on all three CPU architectures. The built-in prefix cache then makes conversations cheap: follow-up turns re-process only the newest message (−60% multi-turn prefill), and a repeat question on the same document answers from cache in ~0.1 s.

Node	Prompt processing (tok/s)	8 concurrent users — aggregate (tok/s)
Arm (Graviton c9g)	307	140
Intel (c8i)	131	28
AMD (c8a, current gen)	221	44

Prompt processing measured on a 3,000-token document with a cold cache; a repeat question on the same document answers from the built-in prompt cache in ~0.1 s. Concurrency figures are aggregate output tokens/sec across 8 simultaneous clients issuing short chat requests (64-token answers) — a worst-case bursty pattern; sustained generation aggregates higher. Prompt processing sets time-to-first-token on long documents and scales with vCPU count. Under concurrency the batcher raises aggregate throughput while each user's share falls; max-inflight controls admission (validated stable to 64 concurrent on a 64 GB node).

Concurrent requests	3-node cluster (4B) — aggregate tok/s	vs single node	Per-node memory
16	43.8	1.49×	16 GB
32	43.2	1.52×	23 GB
64	47.7	1.65×	31 GB

Tested cluster: three 32-vCPU / 64 GB nodes — one Arm (Graviton4, c9g.8xlarge), one Intel Xeon (c8i.8xlarge), one AMD EPYC (c7a.8xlarge) — with the Graviton as the entry node. Any node accepts requests and load-balances across the fleet; scaling efficiency improves as concurrency rises and each node stays independently memory-bounded — so capacity grows with node count. Mixed-architecture clusters are supported (the measured fleet above combined all three CPU vendors); requests route by capacity and model residency — a request for any model reaches a node that has it loaded, so a mixed fleet can serve different models behind one endpoint — with the fastest node as the entry point. See the Deployment Guide for the two-command cluster join.

Before binaries publish, every release is byte-compared against the currently published build: all 380 prompt outputs, on fresh Arm (Graviton c9g), Intel (c8i), and AMD (c8a) nodes — 380/380 identical on all three, plus concurrency sweeps (1–8 parallel clients) with zero errors. In the model comparison above, q35-4b and the 27B produced identical pass/fail counts on every architecture they ran — the engine is behaviorally uniform across CPU vendors, so a mixed-architecture cluster gives every user the same answers.

Text throughput measured with greedy decoding on quantized production models (8-bit KV cache, fast prompt processing and fast attention kernels enabled — the installer defaults) on 32 vCPU / 64 GB cloud nodes; GPU figures from a single NVIDIA L4 at ~96% of the hardware's practical ceiling. Multimodal times are averages over a 380-prompt validation suite covering 13 industries, measured client-side as complete request round trips. Every release is gated on that suite plus a concurrency sweep on all three CPU architectures before binaries are published. First request after installing a model additionally pays a one-time disk load (seconds to ~1 minute depending on model size and storage).