Right-size the model: one Qwen3.5 family, three speed tiers, one server

With a metered API there's little reason to think about model size — you pay per token either way, so everyone defaults to the biggest model for everything. On your own server the economics invert: every task you route to a smaller model is answered sooner, and every core-second saved is capacity for the next request. The skill that matters becomes right-sizing: match the model to the task, and reach for the big one only when the task earns it.

The server makes this trivial — all the Qwen3.5 sizes serve side-by-side from one process, and routing is just the "model" field on each request. Here's what right-sizing is worth, measured on a single CPU-only server (32-vCPU arm) with three family members loaded at once.

One identical request — "summarize this incident in exactly 3 bullet points for an executive update" over a ~120-word incident note — sent to each tier (temperature 0, warm models):

Model	Wall time	Decode	Summary quality
q35-0.8b	1.0 s	105.6 tok/s	correct, all key facts
q35-2b	1.8 s	64.6 tok/s	correct, all key facts
q35-4b	3.7 s	32.5 tok/s	correct, all key facts

All three produced accurate executive bullets. For this task the 4B model buys you nothing except a 3.7× longer wait — and at batch scale (summarizing every ticket, every call transcript), that difference is your throughput ceiling. A one-word classification request on 0.8B answered in 300 ms.

Size still matters when structure and instruction-following get strict. We asked for a JSON extraction of five fields with a casually-worded prompt:

• q35-4b, naive prompt: clean JSON, right keys, right types.
• q35-0.8b, naive prompt: the facts were right, but the contract broke — markdown fences around the JSON, an invented key name (three_follow_up_actions), a number returned as a string in an array.

Then we optimized the prompt — explicit key list with types, "output ONLY a JSON object, no prose, no markdown fences":

Output ONLY a JSON object, no prose, no markdown fences, exactly these keys:
start_time (string), root_cause (string), rollback_time (string),
customers_with_duplicate_holds (number), follow_up_actions (array of 3 strings).
Text: ...

q35-0.8b, optimized prompt: exact schema, correct types, no fences — at 73 tok/s, three times the 4B's throughput, from a model a fifth the size. The prompt bought a tier down.

{
  "start_time": "09:14",
  "root_cause": "connection pool exhaustion in the payments gateway after a deploy...",
  "rollback_time": "09:47",
  "customers_with_duplicate_holds": 240,
  "follow_up_actions": ["add pool saturation alerting", "cap client retries",
                        "add a canary stage to the deploy pipeline"]
}

Honest footnote: the source text was ambiguous about whether the rollback happened at 09:41 or took effect at 09:47, and different runs picked different readings — model size didn't fix that. Ambiguity is a prompt-and-verify problem at every size, which is why our use-case walkthroughs bake a verify step into every extraction.

Task	Tier	Why
Classification, routing, tagging	0.8B	300 ms answers; quality plateaus early
Summaries, title/description generation	0.8B–2B	3× the throughput of 4B, same output quality
Strict-schema extraction, casual prompts	4B (or 0.8B + a tight prompt)	instruction-following strengthens with size; a precise prompt substitutes
Chat, RAG answers, tool calling, agents	4B–9B	the reliability floor for tool use is ~4B
Long-form drafting, hard reasoning, coding	9B–27B	where extra capacity genuinely shows; pair with the GPU add-on for interactive speeds

This is how the server itself is built, by the way: background session summaries run on sessions-summary-model=q35-0.8b, memory recall embeds with the 2B embedder, and chat defaults to 4B — three tiers cooperating inside one feature, each doing the job its size is right for.

All sizes serve simultaneously from one endpoint, so the router is your own code choosing a string:

model = "q35-0.8b" if task in ("classify", "summarize") else "q35-4b"
client.chat.completions.create(model=model, messages=[...])

The cost of keeping a smaller model resident is RAM, once (the 0.8B weights are under 1 GB) — not a per-token bill. Measure your own tasks with the 380-prompt catalog, tighten the prompts with the prompt-optimization guide, and let the big models earn their seat.

All timings measured 2026-09-05 on one CPU-only 32-vCPU arm server (m9g.8xlarge class) with q35-0.8b/2b/4b resident simultaneously, temperature 0, warm models, single stream. Family-wide single-stream and concurrent throughput tables are on the performance page.