Spark-X2.5: fast, cheap, private inference — 55 tok/s on a CPU

Most of what production LLM traffic actually does — classify, route, extract, rewrite, answer over retrieved context — doesn't need a giant model. It needs a fast one: fast enough that latency disappears from the UX, cheap enough to run on hardware you already own, and private because the requests never leave your network.

As of v1.2.0, the SearchAI Inference Server natively serves Spark-X2.5 — XHToken's Apache-2.0 model family, in 4B and 1.7B sizes. They're dense sliding-window-attention models with a thinking mode, text-only, and built for exactly this tier: the 1.7B decodes at ~55 tok/s on a CPU in our measurements, and the 1.7B's footprint fits a 4 GB host. They serve side-by-side with everything else on the server — model ids spark-x25-4b and spark-x25-1.7b, quantized Q4_K_M — so routing to them is just the "model" field on the request.

Measured on Apple M4 Max-class hardware, CPU only:

Model	Decode	Fits in
spark-x25-1.7b	~55 tok/s	4 GB hosts (Pi-class-adjacent)
spark-x25-4b	~26 tok/s	8 GB hosts

Prompt processing on the 4B runs at ~177 tok/s — near llama.cpp-parity prefill — which matters more than decode speed for the pipeline workloads below, where the prompt (a document, a batch of snippets, a transcript) is usually much longer than the answer.

And a correctness number we care about as much as the speed ones: running greedy decoding against llama.cpp on identical model files, our native implementation is token-exact on 45/45 parity prompts. Same weights, same tokens out. Speed claims are easy; byte-for-byte agreement with the reference implementation is the part that lets you trust them.

Spark-X2.5 models can reason before answering. When thinking is on, the reasoning trace comes back in the standard reasoning_content field, separate from the answer — your application shows the answer and can log or display the reasoning as it chooses.

The toggle is per-request:

curl -s http://127.0.0.1:8081/v1/chat/completions \
  -H "Authorization: Bearer $SEARCHAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "spark-x25-1.7b",
    "enable_thinking": false,
    "messages": [{"role":"user","content":"Is this email spam? Answer yes or no: ..."}]
  }'

That off switch is the point for the fast tier. With thinking disabled, the model goes straight to the answer — in our tests, direct questions came back as 7-token responses: no preamble to wait through, no reasoning tokens to pay for. At ~55 tok/s that's an answer in a blink. Flip thinking on, per request, for the questions that deserve a moment's deliberation — same model, same endpoint.

Two real request/response pairs from our v1.2.0 release-gate runs (Apple M4 Max-class CPU), same question both times, so you can see the two shapes this family produces.

Thinking off — the instant direct answer. This is the shape your classification, routing, and direct-QA traffic gets:

# request
{ "model": "spark-x25-1.7b",
  "enable_thinking": false,
  "messages": [{"role":"user","content":"What is the capital of France? Answer in one short sentence."}] }

# response message
{ "role": "assistant",
  "content": "The capital of France is Paris." }

# 7 completion tokens · sub-second · ~55 tok/s

Thinking on — the visible reasoning channel. Same question to the 4B with thinking enabled. The model narrates its (short) route to the answer in reasoning_content, then answers:

# request
{ "model": "spark-x25-4b",
  "enable_thinking": true,
  "messages": [{"role":"user","content":"What is the capital of France? Answer in one short sentence."}] }

# response message
{ "role": "assistant",
  "reasoning_content": "We are asked: \"What is the capital of France? Answer in one short sentence.\" The capital of France is Paris. So the answer should be a one short sentence stating that. For example: \"The capital of France is Paris.\" That is one short sentence.",
  "content": "The capital of France is Paris." }

# 62 completion tokens total · ~2.4 s · ~26 tok/s

The toggle costs tokens — 62 where the direct path spent 7 — but it shows its work, separated from the answer. Use it for the requests whose answers you want auditable; leave it off for volume.

Both samples are unedited real outputs captured during our v1.2.0 release-gate testing.

1. Interactive chat and support widgets. When the model is embedded in a product surface, latency is the UX. At ~55 tok/s the 1.7B streams faster than people read, and thinking-off responses start essentially immediately — on a CPU box, with no per-token bill. Output shape: a 50–150 token conversational reply streaming at ~55 tok/s — ahead of reading speed from the first word.

2. High-volume classification, routing, and extraction pipelines. Tagging tickets, routing emails, pulling fields out of documents — many requests per second, none of them hard. The per-stream numbers above are the floor, not the ceiling: the engine batches concurrent requests to the same model, so parallel streams multiply aggregate throughput rather than queueing behind each other. Small, fast models are exactly where that batching pays most. Output shape: a 1–5 token label at 55 tok/s = roughly 50–100 ms per decision.

3. RAG answer synthesis. In a retrieval pipeline the model's job is to read the retrieved snippets and compose — the knowledge is in your index, not the weights. That's prefill-heavy work, and ~177 tok/s near-parity prompt processing on the 4B means the "read the context" phase is no longer the tax it used to be. Output shape: a 100–200 token grounded answer in 2–4 s on the 1.7B, after the context is read at prefill speed.

4. Edge and branch-office deployments. The 1.7B fits a 4 GB host — Pi-class-adjacent hardware. A capable, private language model on the kind of machine that sits in a branch office, a kiosk, or a rack corner, with no cloud dependency at all. Output shape: the same instant labels and short answers as above — just produced on hardware that costs less than a phone.

5. Agent executors, paired with a big planner. Our flash-next post covers the deep-reasoning end of this release; Spark-X2.5 is the companion pattern. Let the big model plan, and route the many small steps — format this, extract that, summarize the tool output — to spark-x25-1.7b. One server hosts both; the split is just the model id on each request. Output shape: many 10–50 token tool and formatting steps, each sub-second at ~55 tok/s.

6. Real-time query rewriting and autocomplete. Reformulating a search query, expanding an abbreviation, suggesting a completion — jobs where anything slower than "instant" isn't worth shipping. This is the tier that makes them shippable on CPU. Output shape: a 5–20 token rewrite in well under half a second — the same order as the 7-token sample above.

Text-only. No vision, no audio. If your workload needs to look at documents or images, that's the vision-capable models on the same server.

Small-model quality ceilings are real. A 1.7B — even a good one — will not carry multi-step reasoning, hard coding tasks, or long-form analysis. That's not what this tier is for. For deep reasoning, route those requests to flash-next or qwen38-27b — same server, same API, different "model" field. Right-sizing, not wishful thinking, is how a self-hosted fleet stays fast: our routing guide covers where each tier earns its keep.

Both models are in the console catalog — fetch groups are live in v1.2.0, so downloading is a click from the console (or an installer group), and they serve alongside your existing models from the same process.

curl -fsSL https://inference-server.searchblox.com/install | sudo bash
# then fetch spark-x25-1.7b / spark-x25-4b from the console catalog
# and route with: "model": "spark-x25-1.7b"

Numbers in this post: decode and prompt-processing rates measured on Apple M4 Max-class hardware (CPU); Q4_K_M quantization; token-exact greedy parity (45/45) verified against llama.cpp on identical model files. Spark-X2.5 is released by XHToken under Apache 2.0.