One flag, 4× the tokens: GPU acceleration measured on AWS

The CPU story is where this server starts: private AI on hardware you already own, no accelerator required. But some workloads want more tokens per second than any CPU can stream — long-form generation, deep-reasoning models, many concurrent users. For those, the same install takes a one-flag GPU add-on: append BACKEND=cuda to the install line, and the same binary, same models, and same OpenAI-compatible API serve from the GPU.

This post is the measured receipt. We benchmarked nine models through the server's public API on a stock AWS GPU instance — the exact launch settings are below so you can reproduce every number.

Qwen 3.8 27B is the deep-reasoning model in the catalog — thinking mode, per-request reasoning_effort levels, hybrid attention for long context. On our fastest CPU reference (a 32-vCPU Graviton4) it decodes at 6.5 tokens/sec: usable for batch work, below comfortable reading speed for interactive use.

On a $1.21/hour g5.2xlarge, the same model, same weights, same API serves at:

qwen38-27b (256-token greedy, warm)	tokens/sec
CPU — 32-vCPU Graviton4 (m9g.8xlarge)	6.5
GPU — g5.2xlarge (NVIDIA A10G 24 GB)	26.2 — 4.0×

That's the difference between "run it overnight" and "chat with it". The 17 GB of weights fit entirely in the A10G's 24 GB of VRAM (the GPU ran at 99% utilization, 16.9 GB resident), and repeated runs held 26.1–26.2 tokens/sec. At this VRAM size the 27B serves text; give it an L40S/A100-class card if you also want its vision input.

Instance	g5.2xlarge — NVIDIA A10G (24 GB VRAM), 8 vCPU, 32 GB RAM · $1.212/hr on-demand (us-east-1)
AMI	Deep Learning Base OSS Nvidia Driver GPU AMI (Ubuntu 24.04) — NVIDIA driver preinstalled, nothing else needed
OS requirement	Ubuntu 24.04 or newer. The server binary needs glibc ≥ 2.36 — the Ubuntu 22.04 variant of the same AMI fails with GLIBC_2.36 not found. We hit this so you don't have to.
Disk	200 GB gp3 (the AMI plus the full 9-model set comprises ~100 GB; smaller is fine for fewer models)
Security group	Inbound TCP 22 (SSH) and 8081 (the API) from your network

Then, on the fresh box:

# one missing runtime lib on DL AMIs (the installer warns if absent)
sudo apt-get install -y libomp5

# install with the GPU flag — downloads the CUDA adapter (~710 MB) + models
curl -fsSL https://inference-server.searchblox.com/install | \
  sudo API_KEY='your-key' BACKEND=cuda MODELS='4b 2b' bash

# optional: the bigger models, fetched into the models dir
curl -fsSL https://inference-server.searchblox.com/deploy/fetch-models.sh | \
  sudo bash -s -- --dest /var/lib/searchai/models \
  gemma-e2b gemma-e4b gemma-12b gemma-26b q38-27b 9b 0.8b

Verify the GPU is engaged — backend and accelerated tell you directly:

curl -s http://127.0.0.1:8081/health
# {"backend":"cuda","accelerated":true,
#  "device":{"vendor":"nvidia","name":"NVIDIA A10G","memory_total_mb":22589,...}}

The adapter ships native code for A100, A10/A10G, L4, RTX 40-series, and H100. If no usable GPU is found, BACKEND=cuda refuses to start rather than silently serving slow — remove the flag to run CPU-only.

Method: 256-token greedy completions through /v1/chat/completions (the server's public API, not a raw benchmark harness), warm model, single stream, three runs — the same protocol behind our published CPU numbers. CPU column: 32-vCPU Graviton4 (m9g.8xlarge, ~$1.24/hr) running the current release with the interleaved-quant decode packs.

Model	GPU — A10G (tok/s)	CPU — 32 vCPU (tok/s)	Speed-up
qwen38-27b (27B dense reasoner)	26.2	6.5	4.0×
gemma-4-26B-A4B (MoE)	108.7	—	—
gemma-4-12B	55.4	13.3	4.2×
gemma-4-E4B	104.2	—	—
gemma-4-E2B	164.6	—	—
q35-9b	77.0	20.8	3.7×
q35-4b (flagship default)	119.0	33.6	3.5×
q35-2b	220.1	67.5	3.3×
q35-0.8b	327.8	113.5	2.9×

"—" = no published fleet CPU figure for that model yet (the Gemma E-series and 26B have only dev-box CPU measurements; we don't mix methodologies in one table). All numbers are from the runs described above on 2026-08-31; expect the usual ±few-percent run-to-run variance.

Two patterns worth naming:

• The speed-up grows with model size — 2.9× at 0.8B, 4× at 27B. Decode is memory-bandwidth-bound, and the gap between GPU and CPU bandwidth widens as more weight bytes stream per token. The models that most need the help get the most.
• MoE loves the GPU. The 26B-A4B mixture routes ~4B active parameters per token, so it decodes at 108 tok/s — twice as fast as the dense 12B despite having twice the total parameters. High-quality answers at 4B streaming cost.

Resident set	Fits on A10G/L4 (24 GB)?
q35-0.8b + 2b + 4b + both Gemma E-models together	✅ ~8.4 GB — multi-model serving, no juggling
gemma-4-12B (+ a small model or two)	✅ ~7 GB
gemma-4-26B-A4B alone	✅ ~14 GB
qwen38-27b alone, text	✅ ~17 GB
qwen38-27b + vision encoder	❌ needs L40S/A100-class

Models load and unload through the API (/v1/models/load / unload) or the console's Models tab, so one 24 GB card can serve the whole catalog one large model at a time — or a fleet of small ones simultaneously.

The honest counterpoint: at 33.6 tok/s, the 4B flagship on CPU is already well past reading speed, and CPU boxes bring things GPUs don't — more RAM than VRAM (audio + vision + text resident together), clustering, and the option to run on the servers you already own at zero marginal cost. Use the GPU flag when you need large-model interactivity (12B–27B chat), long open-ended generation, or throughput headroom for many concurrent streams. A g5.2xlarge and a 32-vCPU CPU node cost within pennies of each other per hour — the choice is workload shape, not budget.

GPU acceleration here is not a different product: it's the same single binary, the same models, the same API key and endpoints, with one environment variable at install time. On a stock $1.21/hour AWS instance it turns a batch-speed 27B reasoner into a 26-tokens/sec interactive one, and lifts every model in the catalog 3–4× — measured, reproducible, and honest about what a 24 GB card can and can't hold.

curl -fsSL https://inference-server.searchblox.com/install | \
  sudo API_KEY='your-key' BACKEND=cuda bash

CPU numbers: current release on the EC2 fleet — see Measured Performance. Full install and sizing reference: Getting Started.