Show, don't tell: answering questions about images with a calibrated decision model

Vision-language chat is good at describing an image. But most of what production systems need from an image isn't a description — it's a decision: does this screenshot show an error? which team owns it? is this scanned form complete enough to process? A chat model answers those in prose you then have to parse, with a confidence you can't trust.

The strands-decider model takes images directly in POST /v1/systemone: attach up to 4 images (base64 or data-URI, longest side capped at 448 px) alongside the state text, ask typed questions — noul (yes/no probability), choice (options with criteria), score (ordered levels) — and read calibrated probabilities off one prefill, zero generated tokens. Every number below is a real output from the shipped v1.6.0 server, run while writing this post.

The smallest possible example — a plain red square and two questions in one request. This is the exact image sent (300×300):

curl -sS localhost:8081/v1/systemone -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d '{"model":"strands-decider-2b",
       "state":"A single image is attached.",
       "images":[""],
       "questions":{
         "color": {"type":"choice","instructions":"What is the dominant color of the image?",
                   "criteria":{"red":"mostly red","green":"mostly green","blue":"mostly blue"}},
         "is_red":{"type":"noul","instructions":"Is the image predominantly red?"}}}'

# → "color":  { "choice": "red", "probabilities": { "red": 0.9788, "green": 0.0144, "blue": 0.0068 },
#               "confidence": 0.9682 }
#   "is_red": { "noul": 0.8832 }
#   usage: 323 input tokens, 0 output tokens

Both questions shared one image encode and one state prefill — the second question was nearly free.

A support ticket arrives with a screenshot of an error dialog: "Payment failed — Error 402: card declined by issuer. No retry scheduled. Order #58214 is on hold." This is the exact screenshot attached to the request:

Three questions, one request:

{ "model": "strands-decider-2b",
  "state": "Screenshot attached from a customer support ticket.",
  "images": [""],
  "questions": {
    "shows_error": { "type": "noul",  "instructions": "Does the screenshot show an error message?" },
    "team":        { "type": "choice","instructions": "Which team should own this ticket?",
                     "criteria": { "billing":   "payments, charges, card errors",
                                   "technical": "bugs, crashes, API failures",
                                   "sales":     "pricing, plans" } },
    "severity":    { "type": "score", "instructions": "How severe is the issue shown for the customer?",
                     "criteria": [ "cosmetic, no impact", "annoying, has a workaround",
                                   "blocks a key task", "blocks purchase or loses data" ] } } }

# → "shows_error": { "noul": 0.901 }
#   "team":     { "choice": "billing", "confidence": 0.9173,
#                 "probabilities": { "billing": 0.9449, "technical": 0.0509, "sales": 0.0043 } }
#   "severity": { "score": 2.05,
#                 "probabilities": { "0": 0.105, "1": 0.129, "2": 0.374, "3": 0.392 },
#                 "confidence": 0.4493 }
#   usage: 618 input tokens, 0 output tokens · measured 12.1 s on an M4-class CPU

Two things worth staring at. First, the routing: an error dialog looks technical, but the model read the text in the image — card declined by issuer — and put 0.9449 on billing, with technical a distant 0.05. That's the ≥0.90 act band: auto-route it.

Second, the severity answer is honestly unsure — the mass sits almost evenly across "blocks a key task" (0.374) and "blocks purchase" (0.392), and the confidence says so (0.45). A generative model would have picked one and sounded certain. For an ordered score question you also get the expected value (2.05 on the 0–3 legend), which is often the number you actually want to sort a queue by.

A scanned vehicle claim form has the policy number, date, and description filled in — but the signature line is blank. Here is the exact scan sent — note the empty signature line at the bottom:

The intake pipeline asks two questions:

{ "model": "strands-decider-2b",
  "state": "A scanned claim form is attached. Required fields: policy number, date of incident, description, signature.",
  "images": [""],
  "questions": {
    "doc_kind":    { "type": "choice", "instructions": "What kind of document is this?",
                     "criteria": { "claim_form": "an insurance claim form",
                                   "invoice":    "a bill or invoice",
                                   "id_document":"a passport or driving licence" } },
    "is_complete": { "type": "noul",
                     "instructions": "Are ALL required fields filled in (policy number, date, description, signature)?" } } }

# → "doc_kind":    { "choice": "claim_form", "confidence": 0.9957,
#                    "probabilities": { "claim_form": 0.9971, "invoice": 0.0022, "id_document": 0.0007 } }
#   "is_complete": { "noul": 0.5598 }
#   measured 10.8 s · 460 input tokens, 0 output tokens

The classification is emphatic (0.9971 — route it to the claims queue). The completeness answer is the interesting one: 0.5598. Not "yes", not "no" — squarely inside the 0.50–0.90 confirm band from the standard pipeline thresholds. The gate does exactly what it should: the form isn't rejected (three of four fields are plainly filled), and it isn't auto-accepted either — it goes to a human with the probability attached. A form with every field filled clears the band; this one, with a blank signature line, correctly doesn't. That behaviour is what calibration buys you, and it's why the probability — not the argmax — is the product.

• Model: use strands-decider-2b (the f16 entry binds the shipped vision projector). The Q8 variant serves text-only and refuses image requests with a clean error rather than silently ignoring the image.
• Limits: ≤4 images per request, longest side resized to 448 px (≤196 vision tokens per image), oversize inputs refused. Images and state are prefilled once; every additional question is nearly free — the three-question triage measured the same ~12 s as the two-question warm-up.
• Latency: ~11–12 s per request on an M4-class CPU in these runs (vision tower + state prefill + the question forks). Right for intake pipelines, queue triage, and async enrichment; pair with /v1/score when the input is text-only and you need hundreds of milliseconds.
• score questions take an ordered list of level descriptions ("criteria": ["cosmetic", …, "blocks purchase"]) and return the full distribution, the expected value, and a legend.
• Everything stays local: the screenshots and scans in these examples never left the box — the same server, key, and API as the rest of the stack.

All outputs above are unedited responses from a v1.6.0 server (strands-decider-2b, M4-class CPU), captured while writing this post. Vision decisions are parity-gated against the upstream reference implementation on every release (tolerance 0.07 absolute, argmax equality).