Measured Performance
All numbers below are measured on real deployments of the shipping build — not projections. CPU results are from production-class cloud nodes (32 vCPU / 64 GB); GPU results from a single NVIDIA L4. Decode tokens/sec is the per-user response speed; 20+ tokens/sec is faster than reading speed.
per-user tokens/secfast prompt processing ~0.1 s cached follow-ups multimodal response times GPU ≈ 10× CPUcluster scaling
Architectures tested
| Architecture | Node class tested | Notes |
|---|---|---|
| Arm — AWS Graviton4 | c9g · 32 cores / 64 GB · 3.3 GHz | Fastest CPU decode per node (highest memory bandwidth); one binary serves all Arm hosts |
| Intel Xeon | c8i · 32 vCPU / 64 GB · 3.9 GHz | Same x86_64 binary as AMD; strong prompt processing |
| AMD EPYC (Turin) | c8a · 32 cores / 64 GB · 2.6 GHz | Same x86_64 binary as Intel; current AMD generation |
| NVIDIA GPU | L4 (24 GB) measured · add-on also runs A100, A10/A10G, RTX 40-series, H100 | One-flag install add-on; ~10× CPU speeds at ~96% of the hardware's practical ceiling |
Two binaries cover every deployment: one for Arm, one for x86_64 (Intel and AMD). Clusters can mix architectures freely — see cluster scaling below.
Text generation — per-user speed (tokens/sec)
| Model | CPU · Arm (Graviton) | CPU · Intel | CPU · AMD | GPU · NVIDIA L4 |
|---|---|---|---|---|
| 2B chat + vision | 51 | 42 | 41 | 138 |
| 4B chat + vision (recommended) | 24 | 20 | 29 | 70 |
| 35B mixture-of-experts | 13–19 | — | — | — |
Single-stream warm decode (256-token answer, greedy), quantized models, 8-bit KV cache; table figures from 32-vCPU nodes. Decode follows memory bandwidth, so node size matters: 4B measures ≈24 tok/s on a 32-vCPU Graviton (c9g.8xlarge) and ≈23 tok/s on a 16-vCPU (c9g.4xlarge). A GPU node serves the same API roughly 10× faster.
Prompt processing & multi-user throughput (2B, per node)
| Node | Prompt processing (tok/s) | 8 concurrent users — aggregate (tok/s) |
|---|---|---|
| Arm (Graviton c9g) | 307 | 88 |
| Intel (c8i) | 131 | 28 |
| AMD (c8a, current gen) | 221 | 44 |
Prompt processing measured on a 3,000-token document with a cold cache; a repeat question on the same document answers from the built-in prompt cache in ~0.1 s. Concurrency figures are aggregate output tokens/sec across 8 simultaneous clients issuing short chat requests (64-token answers) — a worst-case bursty pattern; sustained generation aggregates higher. Prompt processing sets time-to-first-token on long documents and scales with vCPU count. Under concurrency the batcher raises aggregate throughput while each user's share falls; max-inflight controls admission (validated stable to 64 concurrent on a 64 GB node).
Multimodal — full request response times
| Capability (CPU node) | 2B | 4B |
|---|---|---|
| Image question (photo, chart, form) | ~8 s | ~13 s |
| Video question (multi-frame) | ~17 s | ~28 s |
| Speech-to-text (short clip) | 2–4 s (dedicated model) | |
| Text-to-speech (sentence) | ~8 s (dedicated model) | |
| Long-document outline (1,000+ words) | ~10 s | ~23 s |
Averages across a 380-prompt validation suite on a CPU node — complete round trip including prompt processing and the full answer.
| Capability (GPU · L4) | Warm response |
|---|---|
| Image / video question | a few hundred ms + answer tokens |
| Speech-to-text (short clip) | ~200 ms |
| Prompt processing | sub-second |
| Image editing (20-step, 20B editor) | ~81 s |
Text-to-speech runs on CPU in this release (it is fast there). Image editing is GPU-recommended; CPU-only edits take minutes.
Cluster scaling — adding nodes for more users
| Concurrent requests | 3-node cluster (4B) — aggregate tok/s | vs single node | Per-node memory |
|---|---|---|---|
| 16 | 43.8 | 1.49× | 16 GB |
| 32 | 43.2 | 1.52× | 23 GB |
| 64 | 47.7 | 1.65× | 31 GB |
Tested cluster: three 32-vCPU / 64 GB nodes — one Arm (Graviton4, c9g.8xlarge), one Intel Xeon (c8i.8xlarge), one AMD EPYC (c7a.8xlarge) — with the Graviton as the entry node. Any node accepts requests and load-balances across the fleet; scaling efficiency improves as concurrency rises and each node stays independently memory-bounded — so capacity grows with node count. Mixed-architecture clusters are supported (the measured fleet above combined all three CPU vendors); requests route by capacity and model residency — a request for any model reaches a node that has it loaded, so a mixed fleet can serve different models behind one endpoint — with the fastest node as the entry point. See the Deployment Guide for the two-command cluster join.
Methodology
Text throughput measured with greedy decoding on quantized production models (8-bit KV cache, fast prompt processing and fast attention kernels enabled — the installer defaults) on 32 vCPU / 64 GB cloud nodes; GPU figures from a single NVIDIA L4 at ~96% of the hardware's practical ceiling. Multimodal times are averages over a 380-prompt validation suite covering 13 industries, measured client-side as complete request round trips. Every release is gated on that suite plus a concurrency sweep on all three CPU architectures before binaries are published. First request after installing a model additionally pays a one-time disk load (seconds to ~1 minute depending on model size and storage).