One token per second, on purpose: 125B-class reasoning on a single CPU box

Most of what we ship is about making capable small models fast on CPUs. This release goes the other direction: Qwen3.8-Flash-Next, a 125-billion-parameter mixture-of-experts reasoning model, served natively by the same CPU engine as of v1.2.0 — on a single 128 GB-RAM host, no GPU, no cluster. It decodes at about one token per second, and we're going to spend most of this post explaining why that is a number you can build on rather than one to apologize for.

The short version: this is not a chat model. It is an async reasoning workhorse — a model you hand the hard, high-stakes work and collect the answer from minutes later, running entirely inside your network, air-gapped if you need it, alongside the fast small models that handle your interactive traffic on the very same server.

Flash-Next is a sparse mixture-of-experts: the full model holds 125B parameters of knowledge, but each generated token routes through only ~6B active parameters. That sparsity is what makes a frontier-scale model tractable on a CPU at all — per-token compute is closer to a 6B model's, while the breadth of knowledge and reasoning behavior comes from the full 125B.

The architecture (Qwen's qwen4exp generation) is the most modern graph our engine runs: a gated-delta-net linear-attention hybrid interleaved with sparse attention, plus n-gram embeddings and hyper-connections between layers. We ported it natively — same single binary, same OpenAI-compatible API — and verified greedy outputs token-for-token against llama.cpp on identical model files before calling it done. The served quantization is UD-Q2_K_XL: one 79 GB file, model id qwen38-flash-next-q2.

On the CPU hosts we run (Apple M4 Max, Graviton4-class), Flash-Next decodes at roughly 1 token per second. We are not going to dress that up. Do not put this model behind a chat box; nobody wants to watch a paragraph arrive over five minutes.

What that speed is good for is everything where quality beats latency and a queue beats a cursor: overnight document analysis, multi-step reasoning jobs, agent plans, high-stakes drafts. A thousand tokens of carefully-reasoned output in ~20 minutes is a perfectly good contract review; it is a terrible chat turn. Submit jobs in the evening, read results in the morning — the box costs the same whether it's idle or thinking.

Two engine properties make the batch pattern more practical than the headline number suggests:

• Prompt processing is much faster than decode. Reading a long document into context costs far less per token than generating output, so "large input, focused output" jobs — summarize, review, flag, plan — are the sweet spot.
• Follow-ups don't re-read history. The server's session and prefix cache means a multi-turn job — "now check clause 12 again" — reuses the already-ingested context instead of paying for it twice.

Flash-Next is a thinking model: it reasons before it answers, and the API returns that reasoning in the DeepSeek-style reasoning_content field, separate from the final content. Your application logs the chain of reasoning for audit and shows users only the answer:

curl -s http://127.0.0.1:8081/v1/chat/completions \
  -H "Authorization: Bearer $SEARCHAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen38-flash-next-q2",
    "messages": [{"role":"user","content":"Compare the indemnification clauses in these two drafts and flag any asymmetry: ..."}]
  }'

# response message:
# { "role": "assistant",
#   "reasoning_content": "Let me work through both clauses...",  ← the thinking
#   "content": "The drafts are asymmetric in three respects: ..." }  ← the answer

Thinking is toggleable per request — add "enable_thinking": false to the request body when you want a direct answer without the reasoning pass. For the workloads this model is for, we'd leave it on: the reasoning is the point.

One real request/response pair from our v1.2.0 release-gate runs (Apple M4 Max-class CPU), so you can see the output character rather than take our word for it. We ask the simplest question we test with, precisely because it shows the model's temperament — it deliberates even when it doesn't need to:

# request
{ "model": "qwen38-flash-next-q2",
  "enable_thinking": true,
  "messages": [{"role":"user","content":"What is the capital of France? Answer in one short sentence."}] }

# response message
{ "role": "assistant",
  "reasoning_content": "The user asks for the capital of France in one short sentence. This is straightforward factual retrieval. The answer is Paris. I need to provide a concise response without unnecessary elaboration.",
  "content": "The capital of France is Paris." }

# 46 completion tokens · ~74 s · ~0.62–0.93 tok/s (load-dependent)

Note what's the same and what isn't. The API shape is identical to the fast models on this server — reasoning_content, then content, routed by model id. The economics are an order of magnitude different: this run spent about 74 seconds on 46 tokens, and the decode rate honestly ranges from ~0.62 to ~0.93 tok/s depending on what else the box is doing. Every token carries far more deliberation — which is exactly the trade the four scenarios below are built around.

The sample above is an unedited real output captured during our v1.2.0 release-gate testing.

v1.2.0 also adds the other end of the speed spectrum: Spark-X2.5, thinking models at 1.7B and 4B that run at roughly 26–55 tokens/sec on the same class of CPU hosts. They return reasoning_content too, and they load from the console with one click.

The deployment pattern we recommend is both at once: one server hosts both tiers and routes by the model field. Interactive traffic — chat, extraction, tool-calling agents — goes to Spark; the hard problems queue for Flash-Next. No second endpoint, no second box, no gateway logic beyond a model id.

Four patterns from the private/on-prem deployments this model was built for:

• Regulated-industry contract and policy analysis, overnight. Feed the day's contracts, policy diffs, or filings into a queue after close of business; collect reasoned reviews — with the reasoning_content retained for audit — before the next morning. Nothing leaves the building, which for this workload is usually the entire requirement. Output shape: thousands of reasoning tokens across a document queue — hours of work, not seconds, which is why it runs overnight.
• Deep-research summarization queues. Long technical documents in, focused syntheses out. Fast prompt ingestion plus short, dense outputs plays directly to the model's strengths. Output shape: a few hundred tokens of dense synthesis per document — minutes each, queued through the day.
• Agentic workflows with a strong planner. Flash-Next writes the plan — decomposes the task, sets the steps, defines the checks — and the Spark models execute the steps at interactive speed. The expensive model thinks once; the cheap models run many times. Output shape: a few hundred tokens of plan, produced once, then executed by Spark models at ~55 tok/s.
• Second-opinion review. Let a small model draft at 50 tok/s, then have Flash-Next review the high-stakes outputs — the customer-facing letter, the compliance summary — before anything ships. A one-token-per-second reviewer is fine when it only reads the drafts that matter. Output shape: the draft ingested at prefill speed, then a reasoned verdict of a few hundred tokens — minutes per review.

Flash-Next needs a ~128 GB-RAM host — the engine enforces this with an admission gate rather than letting an undersized box swap itself to death. There's no console fetch group for it yet; you place the model file in the models directory yourself and the server picks it up:

# one 79 GB UD-Q2_K_XL file, on a ~128 GB-RAM host
sudo mv qwen38-flash-next-q2.gguf /var/lib/searchai/models/

# then call it like any other model
"model": "qwen38-flash-next-q2"

It serves with thinking on or off, and everything else about the server applies unchanged: OpenAI-compatible API, API-key auth, sessions and memory, and the small models running alongside. Before release we verified greedy decoding token-parity against llama.cpp on identical files — the engine is running the same model, not an approximation of it.

Frontier-scale reasoning used to mean a GPU cluster or someone else's API. As of v1.2.0 it means one 128 GB CPU box inside your own network: a 125B mixture-of-experts that thinks before it answers, keeps its reasoning auditable, and works through the hard queue while the fast models on the same server handle everything interactive. One token per second — pointed at the work where that's exactly enough.

Speeds stated are from our own runs on Apple M4 Max and Graviton4-class hosts; expect variation with hardware, context length, and settings. Sizing details in the deployment guide.