Deployment Guide

Operations reference: model management, clustering, GPU acceleration, and security.

CPU-based, OpenAI-compatible inference. The server ships as a single self-contained binary — no companion libraries and no runtime flags to manage. Deployment is: one binary + a properties file + model files.

• Platforms: Linux x86_64 and aarch64. (Windows is not currently supported.)
• One x86_64 binary runs on both Intel and AMD CPUs (runtime CPU-feature dispatch).

New to the server? Getting Started is the adoption guide for business/developer teams: prerequisites and sizing, one-command install, validating prompts for your industry in the console, integrating via the OpenAI-compatible SDKs, and scaling as users grow. This document is the full operations reference.

Get started

Prerequisites: a Linux host (x86_64 or aarch64) with sudo and outbound HTTPS. Artifacts (server, add-ons, models, docs) are served from a public URL — no cloud account or credentials required. (When the aws CLI is present it is used automatically; otherwise everything downloads over plain HTTPS.)

Fetch the deploy scripts (the installer + model fetcher), then run the install:

Install with one line (defaults: full multimodal — 4B chat + vision + speech-to-text + text-to-speech, 32 GB RAM recommended; auto-generated API key; service starts on port 8081). On a 16 GB host use ... | sudo MODELS='4b' MIN_RAM_GB=8 bash for chat + vision only:

curl -fsSL https://inference-server.searchblox.com/install | sudo bash

Options ride as environment variables, e.g. ... | sudo MODELS='4b asr tts' bash or ... | sudo API_KEY='my-secret' BACKEND=cuda bash. The download & docs landing page is https://inference-server.searchblox.com/index.html.

Prefer to inspect before running? Fetch the scripts and run them yourself:

mkdir -p deploy && cd deploy
BASE=https://inference-server.searchblox.com/deploy
curl -fsSLO $BASE/install-searchai.sh -O $BASE/fetch-models.sh \
     -O $BASE/BUILD_AND_DEPLOY.md -O $BASE/GETTING_STARTED.md
chmod +x *.sh
sudo API_KEY='choose-a-long-secret' MODELS='4b' ./install-searchai.sh

That is the whole flow — everything else below is reference (models, cluster, tuning, ops). The install downloads the arch-matching binary + the 4B model and starts the service; jump to the smoke test in §2 to verify.

Docker alternative (any Docker host, including Docker Desktop on Mac/Windows; amd64 + arm64):

docker run -d --name searchai -p 8081:8081 -v searchai-models:/models \
  public.ecr.aws/m0m7b0k9/searchai-inference-server

Same defaults as the installer (MODELS="4b asr tts"); models persist in the named volume, the auto-generated API key is printed in docker logs searchai (and stored in the volume). Configure with -e MODELS=..., -e API_KEY=..., -e SERVER_PORT=.... The container is CPU-serving; for GPU use the native install with BACKEND=cuda (§8).

Sizing & performance

Server requirements — start with the 4B base, add RAM per extra model.

Sizing rules of thumb: - RAM scales with the number of resident models, not requests. Weights are memory-mapped and quantized; the server also builds an optimized weight pack at load whose transient spike exceeds the resident footprint — so budget headroom above steady-state RSS, especially when loading a second/third model. - vCPU raises prefill (compute-bound) and the concurrency ceiling; single-stream decode is memory-bandwidth-bound and does not scale with cores. - 16 GB comfortably serves 4B (text + vision) OR the audio models — not all at once. For reliable multimodal on a small box, load one modality at a time (POST /v1/models/unload between them), or step up to 32 GB. - Always set min-free-mb so the server sheds load with HTTP 429 before the OS out-of-memory killer fires. Under concurrency, memory stays bounded (validated: 4B held 16→31 GB RSS from concurrency 16→64 on a 64 GB node — no OOM).

Customer capacity planning

Two things size a deployment: which models (sets base RAM) and how many concurrent users (sets vCPU + extra RAM for per-request KV cache).

RAM ≈ resident models + (concurrent users × KV per request). Each in-flight request holds a KV cache that grows with context length; at the default 8K-token context with q8 KV, budget ~1–2 GB per concurrent user on top of the models.

Context window: the default is max-context=8192 tokens (prompt + response combined), with responses capped at max-tokens=6144 unless the request asks for less. Both are plain server.properties keys — raise them (e.g. max-context=32768) and restart to work with longer documents; the models themselves support far longer contexts, so the setting is a memory/latency budget, not a model limit. Cost of raising it: per-request KV memory grows with the window (budget accordingly in the RAM formula above) and long prompts prefill longer. A prompt that exceeds the window fails fast with an explicit context_length_exceeded error naming the limit — nothing is silently truncated.

Concurrency controls: max-inflight sets how many requests generate at once (default 4); beyond it, requests queue up to queue-timeout-ms then return HTTP 429. Raise max-inflight as you add vCPU + RAM. When one node can't keep up, add cluster nodes — that is the path to more simultaneous users.

Memory shedding: min-free-mb rejects new requests when the box's MemAvailable (plus the engine's idle decode-state pool, which is reusable headroom) drops below the floor — HTTP 503 with code native_low_memory and the actual free/floor numbers in the message. This is distinct from the 429 "max in-flight" capacity error; if you see the 503, the box is short on RAM for its workload (lower min-free-mb only if you know the headroom is safe). The installer defaults the floor to ~1/8 of RAM, capped at 4096 MB.

The idle decode-state pool itself is capped by SAI_POOL_MAX_MB (set in /etc/searchai/searchai.env); the installer scales it to host RAM — 1024 MB on ≤32 GB hosts, 2048 MB on ≤64 GB, 4096 MB above — so steady-state chat traffic never crowds out an on-demand audio/vision model load. Raise it on big-RAM dedicated text servers if you want maximum state reuse.

Tokens/sec and how to improve it

Decode tokens/sec is the per-user response speed (e.g. 4B ≈ 28 tok/s on a 32-vCPU Graviton node, ≈ 23 tok/s on 16 vCPU; 2B ≈ 50 tok/s — comfortably faster than reading speed). Under concurrency the batcher raises aggregate throughput, but each user's share falls as more users generate at once. To improve it:

• Faster memory bandwidth (biggest lever for decode). Single-stream decode is memory-bandwidth-bound, not core-count-bound — CPUs with higher-bandwidth RAM (more/faster memory channels; newer Graviton/EPYC/Xeon generations) give a near-linear decode speed-up. This is the #1 thing to spec for faster responses.
• More vCPU → faster prefill (shorter time-to-first-token on long prompts) and a higher concurrency ceiling — not faster single-stream decode.
• More RAM → more concurrent users (more KV headroom) and more resident models.
• Smaller model / tighter KV → 2B instead of 4B, or kv-cache=q4, trade a little quality for materially faster decode and lower memory.
• Cluster → the way to serve many users at full per-user speed simultaneously.

Measured throughput (tokens/sec, SAI_FAST_PREFILL=1, q8 KV cache; on 32 vCPU / 64 GB nodes):

Scaling: - Vertical (bigger box): more cores raise prefill and the concurrency ceiling. Single-stream decode is memory-bandwidth-bound and plateaus (≈50 tok/s for 2B) regardless of extra cores — scale up for prefill and more simultaneous users, not for faster single-stream decode. - Horizontal (cluster) — the way to add capacity: add nodes to raise aggregate throughput. Measured 3-node 4B cluster (distinct prompts, balanced):

Scaling improves with concurrency as the load-balancer spreads requests more evenly. Each node is memory-bounded independently, so capacity grows with node count. Mixed CPU types are fine (route by capacity); the fastest node should be the entry/seed.

1. Artifacts

All artifacts are served from the public base URL https://inference-server.searchblox.com — plain HTTPS, no credentials needed:

Each binary and add-on has a .sha256 next to it; the installer verifies checksums automatically.

Building release binaries (maintainers)

Release binaries are produced by deploy/publish-artifacts.sh (SSM-builds on the fleet hosts via publish-node.sh, uploads to the staging bucket, then mirrors to the public bucket). If you ever build one by hand, three rules are mandatory — each was learned from a shipped-class failure:

1. Build natively on the target OS/arch — never cross-compile from macOS. A Mac→Linux cross build cannot link Linux's OpenMP runtime; the kernel pragmas are silently ignored and the kernels run single-threaded (~3.7× lower concurrent throughput, no warning anywhere). Serial tests won't catch it — only a concurrency sweep does.
2. libomp must be present at build time (apt-get install libomp-dev; verify with ldconfig -p | grep libomp and confirm the build log links it). Without it the build succeeds with serial kernels.
3. Pin the portable CPU target — never build with the host's native CPU. x86_64: -Dtarget=x86_64-linux-gnu.2.38 -Dcpu=x86_64_v3 (a native build on an AMX-capable Intel host bakes unguarded AVX-512/AMX into the main binary and SIGILLs on AMD). arm64: -Dtarget=aarch64-linux-gnu.2.38 (native Graviton4 builds bake SVE and SIGILL on non-SVE arm64). macOS: -Dcpu=baseline (M1+). The featured SIMD kernels are unaffected — they compile with their own flags behind runtime CPUID gates. One x86_64 binary serves both Intel (AMX path) and AMD (VNNI path).

Before any binary reaches the public bucket it must pass the release gate (deploy/release-gate/): the 380-prompt pack byte-compared against the currently published binary plus a C=1..8 concurrency sweep, on a box of the target architecture — and for x86, smoke-test on both an Intel (AMX) and an AMD (VNNI) host. The Docker image must always be rebuilt --no-cache after binaries publish (a cached download layer once shipped stale binaries).

TTS voices — presets and cloning: only samantha is published as a preset. To create/clone a voice, drop a reference recording as <name>.wav (mono WAV, 10–30 s of clean speech) into <models-dir>/tts-voices/, then request "voice":"<name>" — the first use computes the speaker embedding from that file. Omitting voice (or "default") uses the model's built-in voice. Only clone voices you have permission to use.

2. Single-node install (4B + vision + ASR + TTS)

On a fresh Linux box with the deploy/ folder present (the box needs the aws CLI, or the installer will fetch it; credentials come from the instance role):

sudo API_KEY='choose-a-long-secret' \
     MODELS='4b asr tts' \
     ./install-searchai.sh

This installs runtime dependencies (OpenMP runtime + ffmpeg), downloads the arch-matching binary (sha256-verified), fetches the model groups into /var/lib/searchai/models, writes /etc/searchai/server.properties (vision + ASR + TTS auto-enabled when their files are present), installs the searchai systemd service, and preloads the default model on start.

Layout produced:

/opt/searchai/bin/searchai-server          (single binary)
/etc/searchai/server.properties            (config)
/etc/searchai/searchai.env                 (environment)
/var/lib/searchai/models/*.gguf
/etc/systemd/system/searchai.service

Smoke test:

KEY=$(sudo sed -n 's/^server.api-key=//p' /etc/searchai/server.properties)
curl -fsS localhost:8081/health
curl -sS localhost:8081/v1/chat/completions -H "Authorization: Bearer $KEY" \
  -H 'Content-Type: application/json' \
  -d '{"model":"q35-4b","messages":[{"role":"user","content":"Hello"}],"max_tokens":64}'

Quality testing with the browser console (380-prompt pack)

The server ships a built-in web console with a 380-prompt pack covering every capability and 13 industries (Financial Services, Insurance, Healthcare, Manufacturing, Energy, Retail, Life Sciences, Government, Technology, Legal, Education, Telecom, Logistics) for eyeballing answer quality. It is an interactive quality check — you run prompts and read the responses; there are no stored expected answers to diff against (it is not an automated pass/fail gate).

1. Open http://<host>:8081/console in a browser (the console page itself is unauthenticated).
2. Paste your server.api-key into the API key field near the top (it is saved in the browser's local storage and sent as Authorization: Bearer …).
3. Pick a Category and/or an Industry in the left nav (counts cross-filter), or use the filter box to search by id / title / text. Optionally add a user Message. Prompts with an image/video input show an upload box with the supported formats and size limits.
4. Click Run to send it to this server and read the streamed response — tool-calling and multi-turn prompts run as-is, with tool calls shown in the output.
5. Use Copy curl / Copy JSON / Copy Python to take any prompt into your own application — the same request works from any language or HTTP client.

Work through a few prompts per category to confirm the model is coherent and on- topic before handing the endpoint to users.

Repeatable sampled outputs: with a fixed seed, repeated identical requests are reproducible run-to-run once warm. If you need byte-identical outputs from the very first request (e.g., automated diff tests with temperature > 0), start the server with -Dsearchai.native.prefixcache=false — this trades away prompt-reuse speedups on repeated prompts. Greedy requests (temperature: 0) are always reproducible.

Test the other modalities (installed with MODELS='4b asr tts'):

# Vision — base64 an image into a chat message
B=$(base64 -w0 picture.png)
curl -sS localhost:8081/v1/chat/completions -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d "{\"model\":\"q35-4b\",\"messages\":[{\"role\":\"user\",\"content\":[
       {\"type\":\"image_url\",\"image_url\":{\"url\":\"data:image/png;base64,$B\"}},
       {\"type\":\"text\",\"text\":\"Describe this image.\"}]}],\"max_tokens\":64}"

# Video — base64 a short clip (mp4/webm) into a chat message; needs ffmpeg
# on the host (the installer installs it). Keep clips small: they are
# sampled at 1 frame/sec, up to 16 frames.
V=$(base64 -w0 clip.mp4)
curl -sS localhost:8081/v1/chat/completions -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d "{\"model\":\"q35-4b\",\"messages\":[{\"role\":\"user\",\"content\":[
       {\"type\":\"video_url\",\"video_url\":{\"url\":\"data:video/mp4;base64,$V\"}},
       {\"type\":\"text\",\"text\":\"Describe what happens in this video.\"}]}],\"max_tokens\":64}"

# TTS — synthesize speech to a WAV file
curl -sS localhost:8081/v1/audio/speech -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d '{"input":"Deployment complete.","voice":"samantha"}' -o out.wav

# ASR — transcribe a WAV file
curl -sS localhost:8081/v1/audio/transcriptions -H "Authorization: Bearer $KEY" -F file=@recording.wav

For just the 4B (text + vision, fastest to stand up): omit MODELS (defaults to 4b).

On a 16 GB box, add MIN_RAM_GB=8 and test text+vision and audio separately (unload one before loading the other) — see Sizing. For all modalities at once, use a 32 GB box.

3. Cluster install (multi-node, shared model)

Same binary + model on each node; one shared secret. Nodes gossip every 2 s, load-balance across {self} ∪ healthy peers, and proxy streaming responses. Routing is model-aware: gossip carries each node's loaded models, and a request for a model is routed to a node that has it resident (local first, then round-robin among resident peers, then any healthy peer) — so nodes can serve different models (e.g. 4B on two nodes, 9B on one) behind one endpoint, and clients send every request to any node. Use a private network; inter-node traffic is plain HTTP — terminate TLS at an external load balancer.

Seed node (10.0.0.10):

sudo API_KEY='public-api-key' MODELS='4b' \
     CLUSTER_ADVERTISE='10.0.0.10:8081' \
     CLUSTER_SECRET='shared-cluster-secret' \
     ./install-searchai.sh

Each joining node (unique advertise address):

sudo API_KEY='public-api-key' MODELS='4b' \
     CLUSTER_ADVERTISE='10.0.0.11:8081' \
     CLUSTER_SEED='10.0.0.10:8081' \
     CLUSTER_SECRET='shared-cluster-secret' \
     ./install-searchai.sh

Firewall: allow TCP 8081 between every pair of nodes (gossip distributes the full peer list and nodes probe each other directly), and from the load balancer to entry nodes. Block /cluster/* from the public internet.

Verify:

# membership + residency (cluster token, not the Bearer key)
curl -sS -H "X-SAI-Cluster: shared-cluster-secret" http://10.0.0.10:8081/cluster/state
# load-balanced chat (send to any node; it routes)
curl -sS http://10.0.0.10:8081/v1/chat/completions -H "Authorization: Bearer public-api-key" \
  -H 'Content-Type: application/json' \
  -d '{"model":"q35-4b","messages":[{"role":"user","content":"hi"}]}'

/cluster/state should list all peers and show q35-4b loaded after one gossip interval. Send distinct prompts under concurrency to see requests spread (the route split appears in each node's journal when SAI_ROUTE_LOG=1).

4. Model management

fetch-models.sh (installed next to the models) pulls groups on demand:

sudo bash fetch-models.sh --dest /var/lib/searchai/models 4b asr tts voice
sudo bash fetch-models.sh --list

Groups: 4b (text+vision, default), asr, tts, voice, 2b, 9b, all. Models load lazily on first request, or eagerly via POST /v1/models/load. /v1/models and /v1/models/{load,unload} act on the receiving node only (not cluster-forwarded).

5. Operations

systemctl status searchai
journalctl -u searchai -f
systemctl restart searchai            # config changes need a restart

• /health may report an empty loaded array even when healthy — check /v1/models for model warmth.
• Upgrade: install the new binary to a temp path, smoke-test, replace /opt/searchai/bin/searchai-server, restart one node at a time. There is no drain endpoint — pull a node from the load balancer and let in-flight requests finish before restarting it.
• Memory: set min-free-mb (default 4096) to shed load with HTTP 429 before the OS out-of-memory killer fires.

6. Security

• Always set server.api-key (Bearer) and, for clusters, cluster.secret.
• Restrict /cluster/* to the private network; the same cluster.secret on every node.
• /health and /console are unauthenticated by design — don't expose the server directly to the internet; front it with an HTTPS load balancer.
• Single-node HTTPS is available in-process — set both keys and restart:

properties server.ssl-cert=/etc/searchai/certs/fullchain.pem server.ssl-key=/etc/searchai/certs/privkey.pem

The API, console, and health endpoint then serve over https:// on the same port. Do not enable TLS on clustered node ports (peer transport is plain HTTP on the private network — terminate TLS at the load balancer instead).

7. GPU acceleration (NVIDIA)

x86_64 hosts with an NVIDIA data-center GPU (e.g. L4, A10G) can serve text generation, vision and video chat, and speech-to-text on the GPU. Measured on one L4: ~70 tokens/sec on the 4B model and ~138 tokens/sec on the 2B model per stream — roughly 10× the per-node throughput of the CPU sizing tables, with sub-second prompt processing. Image and video questions answer in a few hundred milliseconds warm; transcription of short clips runs in ~200 ms warm.

Prerequisite checklist (before running the installer):

1. NVIDIA driver installed and working — nvidia-smi must succeed and show the GPU. Driver installation is a host-admin step (kernel module, possible reboot) and is not performed by the installer. Stock Ubuntu AMIs do not ship the driver; either: - sudo apt-get install -y nvidia-driver-535-server then reboot, or - launch from an AMI that includes it (e.g. the AWS Deep Learning Base GPU AMI), where nvidia-smi works out of the box.
2. Enough host RAM for the model set — see the instance sizing table below. On a 16 GB box (g6.xlarge) set MIN_RAM_GB=8 to pass the install check, and do not include image in MODELS on that size.
3. Disk headroom for the selected models (the installer prints per-model sizes; the 4B + speech set needs ~10 GB free).

Worked example — g6.xlarge test box (everything except image editing):

nvidia-smi   # must succeed first
sudo API_KEY='<key>' BACKEND=cuda MIN_RAM_GB=8 MODELS='4b asr tts voice' ./install-searchai.sh

Then verify with GET /health → "backend": "cuda", "accelerated": true. For image editing use a g6.2xlarge or larger (the pipeline needs ~30 GB of host RAM — see the sizing table and §8).

Install with the GPU add-on:

sudo API_KEY='...' BACKEND=auto ./install-searchai.sh    # use GPU when present, else CPU
sudo API_KEY='...' BACKEND=cuda ./install-searchai.sh    # require GPU; refuse to start without it

This installs the acceleration add-on from build/linux-amd64-cuda/searchai-gpu-adapter.tgz into /opt/searchai/lib/gpu/ and sets backend= + gpu.adapter-path= in server.properties.

Verify after start — GET /health reports the active backend:

{ "backend": "cuda", "accelerated": true,
  "device": { "vendor": "nvidia", "name": "NVIDIA L4", ... } }

With BACKEND=auto, a missing/broken driver falls back to CPU and the reason appears in the health fallback field. Responses served by the GPU carry "system_fingerprint": "SearchAI-Native-cuda".

Notes:

• Text chat/completions, vision (images), video, and speech-to-text all run on the GPU. Text-to-speech continues to run on the CPU in this release (it is fast there and needs no GPU memory).
• Vision/video need the model's projector file (<model>-mmproj.gguf) in the models directory, and speech-to-text needs the ASR model pair — the same files the CPU install uses; nothing extra to configure.
• Non-streaming responses include usage.total_time_ms — the server-side wall time for the request — alongside the token counts (standard OpenAI clients ignore the extra field).
• The first request after installing a model pays a one-time disk read of the model file (~20 s for the 4B model on default gp3 storage); subsequent loads are seconds. The first vision/audio request also loads the projector (a few hundred MB, one time).
• GPU memory sizing: the 2B model uses ~1.5 GB and the 4B ~4 GB of GPU memory plus per-request context state; vision + ASR add ~3 GB combined — a 24 GB GPU comfortably serves 4B with full media at the default concurrency.

Supported GPUs: the shipped adapter includes native code for NVIDIA A100, A10/A10G, L4, RTX 40-series, and H100 (CUDA compute 8.0 / 8.6 / 8.9 / 9.0). Other NVIDIA generations may work via driver JIT but are not validated. Performance figures on the Performance page are measured on L4; A100/H100-class cards are faster.

GPU instance sizing (AWS L4-class examples):

8. Image editing (instruction-based)

Edit images with plain-language instructions ("make the sky sunset orange", "remove the car", "turn this into a watercolor") through POST /v1/images/edits. GPU strongly recommended — an edit is a 20-step diffusion pass over a 20-billion-parameter model. Measured on one L4 GPU (weight streaming): ~81 s per edit warm, ~5 min for the first edit (one-time pipeline load). CPU-only takes many minutes per edit.

Install with the image add-on by including image in MODELS (the model set is ~30 GB, downloaded once):

sudo API_KEY='...' BACKEND=cuda MODELS='4b image' ./install-searchai.sh

Usage (multipart, like OpenAI images/edits):

curl -sS localhost:8081/v1/images/edits -H "Authorization: Bearer $KEY" \
  -F image=@photo.png -F prompt="make the sky a dramatic sunset" \
  | python3 -c 'import json,sys,base64; r=json.load(sys.stdin); open("edited.png","wb").write(base64.b64decode(r["data"][0]["b64_json"]))'

Or JSON with a base64 image, plus optional knobs:

curl -sS localhost:8081/v1/images/edits -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d "{\"image\":\"$(base64 -w0 photo.png)\",\"prompt\":\"remove the text overlay\",\"size\":\"1024x1024\",\"steps\":20,\"seed\":42}"

The response is {"created", "data":[{"b64_json": "<PNG>"}], "total_time_ms"}.

Notes:

• Edits run one at a time (the pipeline owns the GPU while denoising); concurrent requests queue. Keep this endpoint off the hot path of chat traffic, or serve it from a dedicated GPU node.
• The first edit loads the ~17 GB pipeline (one time, ~1–2 minutes from disk).
• Input images up to 10 MB / 50 megapixels; output size follows the input (rounded to multiples of 16, max 1536) unless size is given.
• Host RAM: the streaming pipeline holds ~30 GB of weights in RAM (image-offload=true; GPU memory use is then tiny — ~350 MB). A 32 GB GPU box (g6.2xlarge class) is the working minimum and runs at the edge of RAM; 64 GB (g6.4xlarge class) is the comfortable tier, and required if chat models serve heavy traffic on the same box.
• Disk: the image model set is ~30 GB on top of the chat models — size the volume accordingly (100 GB+ recommended for a full-featured GPU node).

9. Not in this release

• AMD GPU (backend=rocm) is not yet available and fails closed.
• Windows is not supported as a published channel. A validated native build exists (built and regression-tested on-box 2026-08-12, commit 105c57d) but is deliberately unlinked from the landing page — single-node/CPU-only scope: no clustering, no in-process TLS (front with IIS ARR/nginx), no GPU adapters; see docs/KNOWN_ISSUES.md §2. Artifacts (internal / on-request only):
• s3://searchai-inference-server/build/windows-x64/searchai-server.exe (+ .sha256) — unsigned; expect a SmartScreen warning.
• Staging copies + the gcc kernel bundle: s3://searchai-inference-server-zig/build-out/searchai-server-windows-x64-105c57d.exe, i8mm_bundle-windows-x64-105c57d.o.
• Build recipe: mingw-w64 gcc kernels (-mavx512* -mf16c -mfma -mamx-* -fopenmp), then ld -r kernel.o dlstub.o libgcc_eh.a (emutls + dl stubs), then zig build -Doptimize=ReleaseFast -Dkernel-obj=<bundle> natively on Windows per the native-build release rules.
• Measured on c8i.2xlarge (8 vCPU): 2B decode 18.9 tok/s, prefill 93 tok/s, warm latency ~300 ms — consistent with Linux same-core-count scaling.