Fast local decisioning: classify & route with /v1/score

Route tickets, gate actions, and label inputs in a single prefill with no decode — a deterministic decision plus a probability distribution, entirely on-box.

Why run it locally: Not every step needs a full generation. Classification and routing want speed and repeatability, not prose. POST /v1/score (new in v1.4.0) prefills the prompt once and reads a restricted softmax over just your allowed answers — no decode loop, fully deterministic, and it rides the prefix cache so repeats over the same choice set land in tens of milliseconds. It runs on the same OpenAI-compatible server as chat, so you can decide cheaply and then generate only on the path you chose.

Model & hardware fit: Qwen 3.5 or Spark X2.5 models, CPU is enough. No extra model to install.

1. The server — any install works; one line on Linux:

curl -fsSL https://inference-server.searchblox.com/install | sudo bash

2. The agent — pi, a minimal open-source coding agent (four tools: read, write, edit, bash), needs Node 22+:

npm install -g --ignore-scripts @earendil-works/pi-coding-agent

3. Connect them — create ~/.pi/agent/extensions/searchai.ts:

import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";

export default function (pi: ExtensionAPI) {
  pi.registerProvider("searchai", {
    baseUrl: "http://127.0.0.1:8081/v1",
    apiKey: "$SEARCHAI_API_KEY",
    api: "openai-completions",
    models: [{ id: "q35-4b", name: "SearchAI q35-4b", reasoning: false,
      input: ["text"], cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
      contextWindow: 32768, maxTokens: 4096 }]
  });
}

export SEARCHAI_API_KEY='your-api-key'   # printed by the installer

Full setup detail (and four more worked tasks) in the pi agents article. The server works with pi stock — no compatibility flags.

Ask the server to route an incoming support message — send the query and the allowed answers, get back a decision and a probability distribution:

curl -s http://127.0.0.1:8081/v1/score \
  -H "Authorization: Bearer $SEARCHAI_API_KEY" -H "Content-Type: application/json" \
  -d '{"model":"q35-4b","query":"My card was charged twice this month.",
       "choices":["billing","shipping","technical support","none of the above"]}'

Measured result, verbatim (~355 ms on a CPU-only server, deterministic across runs):

{
    "object": "score",
    "decision": "billing",
    "probabilities": { "billing": 0.99, "shipping": 0.002,
                       "technical support": 0.004, "none of the above": 0.003 },
    "scores": [0.99, 0.002, 0.004, 0.003]
}

Then decide, then generate: branch on the decision and only pay for a full generation on the path you picked — e.g. draft the reply with a 4B model:

DECISION=$(curl -s .../v1/score -H "Authorization: Bearer $SEARCHAI_API_KEY" \
  -H "Content-Type: application/json" -d '{...as above...}' | python3 -c \
  'import sys,json;print(json.load(sys.stdin)["decision"])')
# route to the right queue / policy, then generate the answer with q35-4b or spark-x25-4b
[ "$DECISION" = "billing" ] && curl -s .../v1/chat/completions ...  # draft billing reply

Probability mass is forced onto the listed choices, so add an explicit escape option like "none of the above" whenever the set may not be exhaustive. Labels map to single tokens internally (2–26 choices). Scoring is validated on the Qwen 3.5 and Spark X2.5 families; a raw label_token_ids form (SGLang /v1/score parity) is available for callers that own the prompt. Full detail in the local-decisioning article.

Measured: ~355 ms per decision (q35-4b) · ~84 ms on an exact repeat.

Next: all the use cases · give the agent memory across sessions · 3–4× the speed with one GPU flag