Decision pipelines: chaining calibrated decisions into generation, worked end-to-end

Every production agent flow we've validated — ticket triage, compliance review, RAG with actions — is secretly the same two-step: decide, then generate. Today most teams make one chat model do both jobs in a single generation ("think about the category, then write the answer"). That's slow where it shouldn't be (the decision pays the full decode loop) and uncalibrated where it matters most (an LLM saying "I'm 90% sure" means nothing — it's just text).

The server splits the jobs across two endpoints that already live on the same box. POST /v1/systemone (the strands-decider model) produces calibrated, typed, zero-decode decisions — a probability you can threshold. POST /v1/chat/completions produces the actual words. The console's Pipeline tab ships two of these chains as runnable demos: everything below is the real JSON they send, and you can edit every prompt and threshold in the browser.

Stage 1 asks strands-decider two questions about the raw ticket in one request — the state is prefilled once and forked per question, so the second question is nearly free:

# Stage 1 — decide (POST /v1/systemone)
{ "model": "strands-decider-2b",
  "state": "Help! Since this morning every API call to /v2/orders returns a 500 error and our checkout is down. We are on the enterprise plan.",
  "questions": {
    "team":      { "type": "choice", "instructions": "Which team should handle this ticket?",
                   "criteria": { "billing":   "payments, invoices, refunds, payouts",
                                 "technical": "bugs, outages, API errors",
                                 "sales":     "pricing, plans, upgrades" } },
    "is_urgent": { "type": "noul", "instructions": "Does this ticket convey urgency?" } } }

# → { "answers": { "team":      { "choice": "technical", "confidence": 0.97, "probabilities": {...} },
#                  "is_urgent": { "noul": 0.91 } },
#     "usage": { "input_tokens": ..., "output_tokens": 0 } }

A confidence gate — plain client-side code, no model — applies the standard thresholds before anything is generated:

• ≥ 0.90 — act: auto-route to the chosen team's template;
• 0.50–0.90 — confirm: proceed, but flag the run for review;
• < 0.50 — escalate: stop and hand the ticket to a human.

Stage 2 sends the routed system template plus the ticket to a generative model (any of the chat models on the box — the demo defaults to the fast tier):

# Stage 2 — generate (POST /v1/chat/completions)
{ "model": "q35-2b",
  "messages": [
    { "role": "system", "content": "You are the TECHNICAL support responder. Acknowledge impact, state the next diagnostic step, give a realistic ETA. The customer is on the enterprise plan." },
    { "role": "user",   "content": "" } ] }

Because the gate sits between the stages, the expensive step only runs on tickets the decider is confident about — and the ones it isn't sure about reach a human with the probabilities attached, not buried in prose.

The second demo is the classic search-practice chain — spelling correction → classification → expansion — rebuilt as decide-then-generate. Stage 1 asks strands-decider three questions about the raw query in one request: a gibberish gate, the query's language, and its intent class. Stage 2 corrects spelling and Stage 3 expands synonyms with a small generative model (the validated binding is q35-2b), and the result is assembled into one "understood query" object.

The gibberish gate is where calibration earns its keep: measured keyboard mash lands at P 0.78–0.88, so the demo rejects at P ≥ 0.90 and stops with a human override between 0.50 and 0.90 — thresholds you can edit in the browser and justify in a design review, because the decider's probabilities are calibrated (ECE 0.054 — see the strands-decider post).

Honest latency note: the full three-stage chain measures ~6 s on CPU. Use it for offline and async enrichment — query-log mining, zero-result analysis, building suggestion dictionaries — not inside a live <100 ms search request. The individual stages are available as prompt cards P388–P397 in the console.

Step	Measured cost (M4-class CPU)
systemone, 1 question, short state	~1–2 s (prefill-bound; grows with state length)
each ADDITIONAL question, same state	nearly free (state encoded once, forked per question)
repeated decisions over the same state prefix	sub-second (prefix-cache hit)
generation (2B/4B chat tier)	TTFT ~0.5–2 s + the model's tok/s

Rule of thumb: a decision adds ~1.8 s to a pipeline stage (less when the state repeats), and generation dominates everything else. Bundling all of a stage's gates into ONE multi-question request is the single biggest latency lever.

Route-then-generate and the query chain are patterns (a) and (e) of a catalog that also covers gate-then-generate (a yes/no decision guards an expensive or risky generation), generate-then-verify (the decider reviews what the generator wrote — as of v1.6.0 this one has its own endpoint, POST /v1/evaluate, with deterministic validators and judged rubrics), and refine loops (decide → generate → re-decide until the gate passes). All of them compose from the same two calls you saw above.

To try the shipped ones: open the console at http://localhost:8081/console, install the strands model group, and hit the Pipeline tab. Both stages, the intermediate decision, the gate verdict, and the latency of every run land in the session log, exportable as JSON.