# SearchAI Inference Server — Deployment Guide

CPU-based, OpenAI-compatible inference. The server ships as a **single
self-contained binary** — no companion libraries and no runtime flags to
manage. Deployment is: one binary + a properties file + model files.

- **Platforms:** Linux `x86_64` and `aarch64`. (Windows is not currently supported.)
- **One `x86_64` binary runs on both Intel and AMD CPUs** (runtime CPU-feature dispatch).

---

> **New to the server?** `GETTING_STARTED.md` is the adoption guide for
> business/developer teams: prerequisites and sizing, one-command install,
> validating prompts for your industry in the console, integrating via the
> OpenAI-compatible SDKs, and scaling as users grow. This document is the
> full operations reference.

## Get started

**Prerequisites:** a Linux host (`x86_64` or `aarch64`) with `sudo` and
outbound HTTPS. Artifacts (server, add-ons, models, docs) are served from a
**public URL** — no cloud account or credentials required. (When the `aws`
CLI is present it is used automatically; otherwise everything downloads over
plain HTTPS.)

**Fetch the deploy scripts** (the installer + model fetcher), then run the
install:

**Install with one line** (defaults: full multimodal — 4B chat + vision +
speech-to-text + text-to-speech, 32 GB RAM recommended; auto-generated API
key; service starts on port 8081). On a 16 GB host use
`... | sudo MODELS='4b' MIN_RAM_GB=8 bash` for chat + vision only:

```bash
curl -fsSL https://inference-server.searchblox.com/install | sudo bash
```

Options ride as environment variables, e.g.
`... | sudo MODELS='4b asr tts' bash` or `... | sudo API_KEY='my-secret' BACKEND=cuda bash`.
The download & docs landing page is
**https://inference-server.searchblox.com/index.html**.

Prefer to inspect before running? Fetch the scripts and run them yourself:

```bash
mkdir -p deploy && cd deploy
BASE=https://inference-server.searchblox.com/deploy
curl -fsSLO $BASE/install-searchai.sh -O $BASE/fetch-models.sh \
     -O $BASE/BUILD_AND_DEPLOY.md -O $BASE/GETTING_STARTED.md
chmod +x *.sh
sudo API_KEY='choose-a-long-secret' MODELS='4b' ./install-searchai.sh
```

That is the whole flow — everything else below is reference (models, cluster,
tuning, ops). The install downloads the arch-matching binary + the 4B model and
starts the service; jump to the smoke test in §2 to verify.

**Docker alternative** (any Docker host, including Docker Desktop on
Mac/Windows; `amd64` + `arm64`):

```bash
docker run -d --name searchai -p 8081:8081 -v searchai-models:/models \
  public.ecr.aws/m0m7b0k9/searchai-inference-server
```

Same defaults as the installer (`MODELS="4b asr tts"`); models persist in the
named volume, the auto-generated API key is printed in `docker logs searchai`
(and stored in the volume). Configure with `-e MODELS=...`, `-e API_KEY=...`,
`-e SERVER_PORT=...`. The container is CPU-serving; for GPU use the native
install with `BACKEND=cuda` (§8).

---

## Sizing & performance

**Server requirements — start with the 4B base, add RAM per extra model.**

| Deployment | vCPU (min → recommended) | RAM (min → recommended) | Notes |
|---|---|---|---|
| **4B — text + vision** (base) | **8 → 16** | **16 GB → 32 GB** | validated on 8 vCPU / 16 GB; ~5–7 GB RSS. On a 16 GB box set `MIN_RAM_GB=8` (the default-16 startup gate refuses to start) |
| **+ ASR** | +0 | **+ ~6–8 GB** | ~2.5 GB resident **plus** a load-time spike (transient RAM > resident) |
| **+ TTS** | +0 | **+ ~4–6 GB** | ~2.7 GB resident + load-time spike |
| **Full multimodal** (4B + vision + ASR + TTS) | **16** | **32 GB** | 16 GB is too tight — all models together on 16 GB runs out of memory; use 32 GB, or unload one modality before loading another |
| 9B text/vision | 16 | 32 GB | ~7 GB weights |
| 35B MoE | 16 | 64 GB | ~13–22 GB RSS |

Sizing rules of thumb:
- **RAM scales with the *number of resident models*, not requests.** Weights are
  memory-mapped and quantized; the server also builds an optimized weight pack at
  load whose **transient spike exceeds the resident footprint** — so budget
  headroom above steady-state RSS, especially when loading a second/third model.
- **vCPU** raises **prefill** (compute-bound) and the **concurrency ceiling**;
  single-stream **decode is memory-bandwidth-bound** and does not scale with cores.
- 16 GB comfortably serves **4B (text + vision) OR the audio models — not all at
  once**. For reliable multimodal on a small box, load one modality at a time
  (`POST /v1/models/unload` between them), or step up to **32 GB**.
- Always set `min-free-mb` so the server sheds load with HTTP 429 before the OS
  out-of-memory killer fires. Under concurrency, memory stays bounded (validated:
  4B held 16→31 GB RSS from concurrency 16→64 on a 64 GB node — no OOM).

### Customer capacity planning

Two things size a deployment: **which models** (sets base RAM) and **how many
concurrent users** (sets vCPU + extra RAM for per-request KV cache).

**RAM ≈ resident models + (concurrent users × KV per request).** Each in-flight
request holds a KV cache that grows with context length; at the default 8K-token
context with q8 KV, budget **~1–2 GB per concurrent user** on top of the models.

| Customer profile | Models | Concurrent users | vCPU | RAM |
|---|---|---:|---|---|
| Small (team / pilot) | 4B text+vision | 1–4 | 8–16 | 16–32 GB |
| Medium (department) | 4B text+vision | 8–16 | 16–32 | 32–48 GB |
| + audio (ASR/TTS) | 4B + ASR + TTS | 8–16 | 16–32 | 48–64 GB |
| Large (org-wide) | 4B | 32–64+ | **cluster** (3+ nodes) | 32–64 GB / node |

**Context window:** the default is **`max-context=8192` tokens** (prompt +
response combined), with responses capped at **`max-tokens=6144`** unless the
request asks for less. Both are plain `server.properties` keys — raise them
(e.g. `max-context=32768`) and restart to work with longer documents; the
models themselves support far longer contexts, so the setting is a
memory/latency budget, not a model limit. Cost of raising it: per-request KV
memory grows with the window (budget accordingly in the RAM formula above)
and long prompts prefill longer. A prompt that exceeds the window fails fast
with an explicit `context_length_exceeded` error naming the limit — nothing
is silently truncated.

**Concurrency controls:** `max-inflight` sets how many requests generate at once
(default 4); beyond it, requests queue up to `queue-timeout-ms` then return HTTP
429. Raise `max-inflight` as you add vCPU + RAM. When one node can't keep up,
**add cluster nodes** — that is the path to more simultaneous users.

**Memory shedding:** `min-free-mb` rejects new requests when the box's
MemAvailable (plus the engine's idle decode-state pool, which is reusable
headroom) drops below the floor — HTTP 503 with code `native_low_memory` and
the actual free/floor numbers in the message. This is distinct from the 429
"max in-flight" capacity error; if you see the 503, the box is short on RAM
for its workload (lower `min-free-mb` only if you know the headroom is safe).
The installer defaults the floor to ~1/8 of RAM, capped at 4096 MB.

The idle decode-state pool itself is capped by `SAI_POOL_MAX_MB` (set in
`/etc/searchai/searchai.env`); the installer scales it to host RAM —
1024 MB on ≤32 GB hosts, 2048 MB on ≤64 GB, 4096 MB above — so steady-state
chat traffic never crowds out an on-demand audio/vision model load. Raise it
on big-RAM dedicated text servers if you want maximum state reuse.

### Tokens/sec and how to improve it

Decode **tokens/sec is the per-user response speed** (e.g. 4B ≈ 28 tok/s on a 32-vCPU
Graviton node, ≈ 23 tok/s on 16 vCPU; 2B ≈ 50 tok/s — comfortably faster than reading speed). Under
concurrency the batcher raises *aggregate* throughput, but each user's share
falls as more users generate at once. To improve it:

- **Faster memory bandwidth (biggest lever for decode).** Single-stream decode is
  **memory-bandwidth-bound**, not core-count-bound — CPUs with higher-bandwidth
  RAM (more/faster memory channels; newer Graviton/EPYC/Xeon generations) give a
  near-linear decode speed-up. This is the #1 thing to spec for faster responses.
- **More vCPU** → faster **prefill** (shorter time-to-first-token on long prompts)
  and a higher concurrency ceiling — *not* faster single-stream decode.
- **More RAM** → more concurrent users (more KV headroom) and more resident models.
- **Smaller model / tighter KV** → 2B instead of 4B, or `kv-cache=q4`, trade a
  little quality for materially faster decode and lower memory.
- **Cluster** → the way to serve many users at full per-user speed simultaneously.

**Measured throughput** (tokens/sec, `SAI_FAST_PREFILL=1`, q8 KV cache; on
32 vCPU / 64 GB nodes):

| Node | Model | Decode | Prefill | Concurrency-8 |
|---|---|---:|---:|---:|
| Graviton c9g (Arm) | 2B | 50 | 145 | 138 |
| Graviton c9g | 4B | ~28 | — | — |
| Intel c8i (x86) | 2B | 40 | 107 | 51 |
| AMD c8a (x86, current gen) | 2B | 47 | 142 | 88 |
| AMD c7a (x86, prior gen) | 2B | 40 | 55 | 66 |
| Graviton c9g | 35B MoE | 13–19 | — | — |

**Scaling:**
- **Vertical (bigger box):** more cores raise **prefill** and the **concurrency
  ceiling**. Single-stream **decode is memory-bandwidth-bound** and plateaus
  (≈50 tok/s for 2B) regardless of extra cores — scale up for prefill and more
  simultaneous users, not for faster single-stream decode.
- **Horizontal (cluster) — the way to add capacity:** add nodes to raise
  aggregate throughput. Measured 3-node 4B cluster (distinct prompts, balanced):

  | Concurrency | Cluster tok/s | vs 1 node | Per-node RSS |
  |---:|---:|---:|---:|
  | 16 | 43.8 | 1.49× | 16 GB |
  | 32 | 43.2 | 1.52× | 23 GB |
  | 64 | 47.7 | 1.65× | 31 GB |

  Scaling improves with concurrency as the load-balancer spreads requests more
  evenly. Each node is memory-bounded independently, so capacity grows with node
  count. Mixed CPU types are fine (route by capacity); the fastest node should be
  the entry/seed.

## 1. Artifacts

All artifacts are served from the public base URL
`https://inference-server.searchblox.com` — plain HTTPS, no
credentials needed:

| Artifact | Path under the base URL |
|---|---|
| arm64 binary | `/build/linux-arm64/searchai-server` |
| x86_64 binary | `/build/linux-amd64/searchai-server` |
| macOS binary (Apple Silicon) | `/build/macos-arm64/searchai-server` |
| GPU add-on | `/build/linux-amd64-cuda/searchai-gpu-adapter.tgz` |
| Image-editing add-on | `/build/linux-amd64-cuda/searchai-img-adapter.tgz` |
| Models | `/models/` (`q35-4b.gguf`, `q35-4b-mmproj.gguf`, `qwen3-asr-1.7b.gguf`, `mmproj-qwen3-asr.gguf`, `qwen-talker-1.7b.gguf`, `qwen-tts-codec.gguf`, …) |
| Deploy scripts + docs | `/deploy/` (`install-searchai.sh`, `install-searchai-macos.sh`, `fetch-models.sh`, this guide, `GETTING_STARTED.md`) |

Each binary and add-on has a `.sha256` next to it; the installer verifies
checksums automatically.

### Building release binaries (maintainers)

Release binaries are produced by `deploy/publish-artifacts.sh` (SSM-builds on
the fleet hosts via `publish-node.sh`, uploads to the staging bucket, then
mirrors to the public bucket). If you ever build one by hand, three rules are
**mandatory** — each was learned from a shipped-class failure:

1. **Build natively on the target OS/arch — never cross-compile from macOS.**
   A Mac→Linux cross build cannot link Linux's OpenMP runtime; the kernel
   pragmas are silently ignored and the kernels run single-threaded
   (~3.7× lower concurrent throughput, no warning anywhere). Serial tests
   won't catch it — only a concurrency sweep does.
2. **libomp must be present at build time** (`apt-get install libomp-dev`;
   verify with `ldconfig -p | grep libomp` and confirm the build log links
   it). Without it the build succeeds with serial kernels.
3. **Pin the portable CPU target — never build with the host's native CPU.**
   x86_64: `-Dtarget=x86_64-linux-gnu.2.38 -Dcpu=x86_64_v3` (a native build
   on an AMX-capable Intel host bakes unguarded AVX-512/AMX into the main
   binary and SIGILLs on AMD). arm64: `-Dtarget=aarch64-linux-gnu.2.38`
   (native Graviton4 builds bake SVE and SIGILL on non-SVE arm64).
   macOS: `-Dcpu=baseline` (M1+). The featured SIMD kernels are unaffected —
   they compile with their own flags behind runtime CPUID gates. One x86_64
   binary serves both Intel (AMX path) and AMD (VNNI path).

Before any binary reaches the public bucket it must pass the **release gate**
(`deploy/release-gate/`): the 380-prompt pack byte-compared against the
currently published binary plus a C=1..8 concurrency sweep, on a box of the
target architecture — and for x86, smoke-test on both an Intel (AMX) and an
AMD (VNNI) host. The Docker image must always be rebuilt `--no-cache` after
binaries publish (a cached download layer once shipped stale binaries).

> **TTS voices — presets and cloning:** only `samantha` is published as a
> preset. To create/clone a voice, drop a reference recording as
> `<name>.wav` (mono WAV, 10–30 s of clean speech) into
> `<models-dir>/tts-voices/`, then request `"voice":"<name>"` — the first use
> computes the speaker embedding from that file. Omitting `voice` (or
> `"default"`) uses the model's built-in voice. Only clone voices you have
> permission to use.

---

## 2. Single-node install (4B + vision + ASR + TTS)

On a fresh Linux box with the `deploy/` folder present (the box needs the aws
CLI, or the installer will fetch it; credentials come from the instance role):

```bash
sudo API_KEY='choose-a-long-secret' \
     MODELS='4b asr tts' \
     ./install-searchai.sh
```

This installs runtime dependencies (OpenMP runtime + ffmpeg), downloads the
arch-matching binary (sha256-verified), fetches the model groups into
`/var/lib/searchai/models`, writes `/etc/searchai/server.properties` (vision +
ASR + TTS auto-enabled when their files are present), installs the `searchai`
systemd service, and preloads the default model on start.

Layout produced:

```
/opt/searchai/bin/searchai-server          (single binary)
/etc/searchai/server.properties            (config)
/etc/searchai/searchai.env                 (environment)
/var/lib/searchai/models/*.gguf
/etc/systemd/system/searchai.service
```

Smoke test:

```bash
KEY=$(sudo sed -n 's/^server.api-key=//p' /etc/searchai/server.properties)
curl -fsS localhost:8081/health
curl -sS localhost:8081/v1/chat/completions -H "Authorization: Bearer $KEY" \
  -H 'Content-Type: application/json' \
  -d '{"model":"q35-4b","messages":[{"role":"user","content":"Hello"}],"max_tokens":64}'
```

### Quality testing with the browser console (380-prompt pack)

The server ships a built-in web console with a **380-prompt pack** covering
every capability and **13 industries** (Financial Services, Insurance,
Healthcare, Manufacturing, Energy, Retail, Life Sciences, Government,
Technology, Legal, Education, Telecom, Logistics) for eyeballing answer
quality. It is an **interactive** quality check — you run prompts and read
the responses; there are no stored expected answers to diff against (it is
not an automated pass/fail gate).

1. Open **`http://<host>:8081/console`** in a browser (the console page itself
   is unauthenticated).
2. Paste your `server.api-key` into the **API key** field near the top (it is
   saved in the browser's local storage and sent as `Authorization: Bearer …`).
3. Pick a **Category** and/or an **Industry** in the left nav (counts
   cross-filter), or use the **filter** box to search by id / title / text.
   Optionally add a user **Message**. Prompts with an image/video input show
   an upload box with the supported formats and size limits.
4. Click **Run** to send it to this server and read the streamed response —
   tool-calling and multi-turn prompts run as-is, with tool calls shown in
   the output.
5. Use **Copy curl** / **Copy JSON** / **Copy Python** to take any prompt into
   your own application — the same request works from any language or HTTP
   client.

Work through a few prompts per category to confirm the model is coherent and on-
topic before handing the endpoint to users.

> **Repeatable sampled outputs:** with a fixed `seed`, repeated identical
> requests are reproducible run-to-run once warm. If you need byte-identical
> outputs from the very first request (e.g., automated diff tests with
> `temperature` > 0), start the server with
> `-Dsearchai.native.prefixcache=false` — this trades away prompt-reuse
> speedups on repeated prompts. Greedy requests (`temperature: 0`) are always
> reproducible.

Test the other modalities (installed with `MODELS='4b asr tts'`):

```bash
# Vision — base64 an image into a chat message
B=$(base64 -w0 picture.png)
curl -sS localhost:8081/v1/chat/completions -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d "{\"model\":\"q35-4b\",\"messages\":[{\"role\":\"user\",\"content\":[
       {\"type\":\"image_url\",\"image_url\":{\"url\":\"data:image/png;base64,$B\"}},
       {\"type\":\"text\",\"text\":\"Describe this image.\"}]}],\"max_tokens\":64}"

# Video — base64 a short clip (mp4/webm) into a chat message; needs ffmpeg
# on the host (the installer installs it). Keep clips small: they are
# sampled at 1 frame/sec, up to 16 frames.
V=$(base64 -w0 clip.mp4)
curl -sS localhost:8081/v1/chat/completions -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d "{\"model\":\"q35-4b\",\"messages\":[{\"role\":\"user\",\"content\":[
       {\"type\":\"video_url\",\"video_url\":{\"url\":\"data:video/mp4;base64,$V\"}},
       {\"type\":\"text\",\"text\":\"Describe what happens in this video.\"}]}],\"max_tokens\":64}"

# TTS — synthesize speech to a WAV file
curl -sS localhost:8081/v1/audio/speech -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d '{"input":"Deployment complete.","voice":"samantha"}' -o out.wav

# ASR — transcribe a WAV file
curl -sS localhost:8081/v1/audio/transcriptions -H "Authorization: Bearer $KEY" -F file=@recording.wav
```

For just the 4B (text + vision, fastest to stand up): omit `MODELS` (defaults to `4b`).

> On a 16 GB box, add `MIN_RAM_GB=8` and test text+vision and audio **separately**
> (unload one before loading the other) — see Sizing. For all modalities at once,
> use a 32 GB box.

---

## 3. Cluster install (multi-node, shared model)

Same binary + model on each node; one shared secret. Nodes gossip every 2 s,
load-balance across `{self} ∪ healthy peers`, and proxy streaming responses.
Routing is **model-aware**: gossip carries each node's loaded models, and a
request for a model is routed to a node that has it resident (local first,
then round-robin among resident peers, then any healthy peer) — so nodes can
serve different models (e.g. 4B on two nodes, 9B on one) behind one endpoint,
and clients send every request to any node. Use
a private network; **inter-node traffic is plain HTTP** — terminate TLS at an
external load balancer.

**Seed node** (`10.0.0.10`):
```bash
sudo API_KEY='public-api-key' MODELS='4b' \
     CLUSTER_ADVERTISE='10.0.0.10:8081' \
     CLUSTER_SECRET='shared-cluster-secret' \
     ./install-searchai.sh
```

**Each joining node** (unique advertise address):
```bash
sudo API_KEY='public-api-key' MODELS='4b' \
     CLUSTER_ADVERTISE='10.0.0.11:8081' \
     CLUSTER_SEED='10.0.0.10:8081' \
     CLUSTER_SECRET='shared-cluster-secret' \
     ./install-searchai.sh
```

Firewall: allow TCP 8081 **between every pair of nodes** (gossip distributes the
full peer list and nodes probe each other directly), and from the load balancer
to entry nodes. Block `/cluster/*` from the public internet.

Verify:
```bash
# membership + residency (cluster token, not the Bearer key)
curl -sS -H "X-SAI-Cluster: shared-cluster-secret" http://10.0.0.10:8081/cluster/state
# load-balanced chat (send to any node; it routes)
curl -sS http://10.0.0.10:8081/v1/chat/completions -H "Authorization: Bearer public-api-key" \
  -H 'Content-Type: application/json' \
  -d '{"model":"q35-4b","messages":[{"role":"user","content":"hi"}]}'
```
`/cluster/state` should list all peers and show `q35-4b` loaded after one gossip
interval. Send distinct prompts under concurrency to see requests spread (the
route split appears in each node's journal when `SAI_ROUTE_LOG=1`).

---

## 4. Model management

`fetch-models.sh` (installed next to the models) pulls groups on demand:

```bash
sudo bash fetch-models.sh --dest /var/lib/searchai/models 4b asr tts voice
sudo bash fetch-models.sh --list
```

Groups: `4b` (text+vision, default), `asr`, `tts`, `voice`, `2b`, `9b`, `all`.
Models load lazily on first request, or eagerly via `POST /v1/models/load`.
`/v1/models` and `/v1/models/{load,unload}` act on the **receiving node only**
(not cluster-forwarded).

---

## 5. Operations

```bash
systemctl status searchai
journalctl -u searchai -f
systemctl restart searchai            # config changes need a restart
```

- `/health` may report an empty `loaded` array even when healthy — check
  `/v1/models` for model warmth.
- **Upgrade:** install the new binary to a temp path, smoke-test, replace
  `/opt/searchai/bin/searchai-server`, restart one node at a time. There is no
  drain endpoint — pull a node from the load balancer and let in-flight requests
  finish before restarting it.
- **Memory:** set `min-free-mb` (default 4096) to shed load with HTTP 429 before
  the OS out-of-memory killer fires.

---

## 6. Security

- Always set `server.api-key` (Bearer) and, for clusters, `cluster.secret`.
- Restrict `/cluster/*` to the private network; the same `cluster.secret` on
  every node.
- `/health` and `/console` are unauthenticated by design — don't expose the
  server directly to the internet; front it with an HTTPS load balancer.
- Single-node HTTPS is available in-process — set both keys and restart:

  ```properties
  server.ssl-cert=/etc/searchai/certs/fullchain.pem
  server.ssl-key=/etc/searchai/certs/privkey.pem
  ```

  The API, console, and health endpoint then serve over `https://` on the same
  port. Do **not** enable TLS on clustered node ports (peer transport is plain
  HTTP on the private network — terminate TLS at the load balancer instead).

---

## 7. GPU acceleration (NVIDIA)

x86_64 hosts with an NVIDIA data-center GPU (e.g. L4, A10G) can serve text
generation, **vision and video chat, and speech-to-text** on the GPU.
Measured on one L4: **~70 tokens/sec** on the 4B model and **~138 tokens/sec**
on the 2B model per stream — roughly 10× the per-node throughput of the CPU
sizing tables, with sub-second prompt processing. Image and video questions
answer in a few hundred milliseconds warm; transcription of short clips runs
in ~200 ms warm.

**Prerequisite checklist (before running the installer):**

1. **NVIDIA driver installed and working** — `nvidia-smi` must succeed and show
   the GPU. Driver installation is a host-admin step (kernel module, possible
   reboot) and is **not** performed by the installer. Stock Ubuntu AMIs do not
   ship the driver; either:
   - `sudo apt-get install -y nvidia-driver-535-server` then reboot, or
   - launch from an AMI that includes it (e.g. the AWS *Deep Learning Base
     GPU* AMI), where `nvidia-smi` works out of the box.
2. **Enough host RAM for the model set** — see the instance sizing table
   below. On a 16 GB box (g6.xlarge) set `MIN_RAM_GB=8` to pass the install
   check, and do **not** include `image` in `MODELS` on that size.
3. **Disk headroom** for the selected models (the installer prints per-model
   sizes; the 4B + speech set needs ~10 GB free).

Worked example — g6.xlarge test box (everything except image editing):

```sh
nvidia-smi   # must succeed first
sudo API_KEY='<key>' BACKEND=cuda MIN_RAM_GB=8 MODELS='4b asr tts voice' ./install-searchai.sh
```

Then verify with `GET /health` → `"backend": "cuda", "accelerated": true`.
For image editing use a g6.2xlarge or larger (the pipeline needs ~30 GB of
host RAM — see the sizing table and §8).

Install with the GPU add-on:

```sh
sudo API_KEY='...' BACKEND=auto ./install-searchai.sh    # use GPU when present, else CPU
sudo API_KEY='...' BACKEND=cuda ./install-searchai.sh    # require GPU; refuse to start without it
```

This installs the acceleration add-on from
`build/linux-amd64-cuda/searchai-gpu-adapter.tgz` into
`/opt/searchai/lib/gpu/` and sets `backend=` + `gpu.adapter-path=` in
`server.properties`.

Verify after start — `GET /health` reports the active backend:

```json
{ "backend": "cuda", "accelerated": true,
  "device": { "vendor": "nvidia", "name": "NVIDIA L4", ... } }
```

With `BACKEND=auto`, a missing/broken driver falls back to CPU and the reason
appears in the health `fallback` field. Responses served by the GPU carry
`"system_fingerprint": "SearchAI-Native-cuda"`.

Notes:

- Text chat/completions, vision (images), video, and speech-to-text all run
  on the GPU. **Text-to-speech continues to run on the CPU** in this release
  (it is fast there and needs no GPU memory).
- Vision/video need the model's projector file (`<model>-mmproj.gguf`) in the
  models directory, and speech-to-text needs the ASR model pair — the same
  files the CPU install uses; nothing extra to configure.
- Non-streaming responses include `usage.total_time_ms` — the server-side
  wall time for the request — alongside the token counts (standard OpenAI
  clients ignore the extra field).
- The first request after installing a model pays a one-time disk read of the
  model file (~20 s for the 4B model on default gp3 storage); subsequent loads
  are seconds. The first vision/audio request also loads the projector
  (a few hundred MB, one time).
- GPU memory sizing: the 2B model uses ~1.5 GB and the 4B ~4 GB of GPU memory
  plus per-request context state; vision + ASR add ~3 GB combined — a 24 GB
  GPU comfortably serves 4B with full media at the default concurrency.

**Supported GPUs:** the shipped adapter includes native code for NVIDIA
A100, A10/A10G, L4, RTX 40-series, and H100 (CUDA compute 8.0 / 8.6 / 8.9 /
9.0). Other NVIDIA generations may work via driver JIT but are not
validated. Performance figures on the [Performance](https://inference-server.searchblox.com/performance.html)
page are measured on L4; A100/H100-class cards are faster.

GPU instance sizing (AWS L4-class examples):

| Instance | Host RAM | Fits |
|---|---|---|
| g6.xlarge | 16 GB | Text + vision + video + speech (set `MIN_RAM_GB=8`; test ASR and TTS one at a time — audio load spikes are the tight spot). **Not image editing.** |
| g6.2xlarge | 32 GB | Everything incl. image editing (image runs at the edge of RAM — working minimum) |
| g6.4xlarge | 64 GB | Everything, comfortably — recommended for image editing or mixed heavy traffic |

---

## 8. Image editing (instruction-based)

Edit images with plain-language instructions ("make the sky sunset orange",
"remove the car", "turn this into a watercolor") through
`POST /v1/images/edits`. **GPU strongly recommended** — an edit is a 20-step
diffusion pass over a 20-billion-parameter model. Measured on one L4 GPU
(weight streaming): **~81 s per edit warm**, ~5 min for the first edit
(one-time pipeline load). CPU-only takes many minutes per edit.

Install with the image add-on by including `image` in `MODELS` (the model
set is ~30 GB, downloaded once):

```sh
sudo API_KEY='...' BACKEND=cuda MODELS='4b image' ./install-searchai.sh
```

Usage (multipart, like OpenAI images/edits):

```bash
curl -sS localhost:8081/v1/images/edits -H "Authorization: Bearer $KEY" \
  -F image=@photo.png -F prompt="make the sky a dramatic sunset" \
  | python3 -c 'import json,sys,base64; r=json.load(sys.stdin); open("edited.png","wb").write(base64.b64decode(r["data"][0]["b64_json"]))'
```

Or JSON with a base64 image, plus optional knobs:

```bash
curl -sS localhost:8081/v1/images/edits -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d "{\"image\":\"$(base64 -w0 photo.png)\",\"prompt\":\"remove the text overlay\",\"size\":\"1024x1024\",\"steps\":20,\"seed\":42}"
```

The response is `{"created", "data":[{"b64_json": "<PNG>"}], "total_time_ms"}`.

Notes:

- Edits run **one at a time** (the pipeline owns the GPU while denoising);
  concurrent requests queue. Keep this endpoint off the hot path of chat
  traffic, or serve it from a dedicated GPU node.
- The first edit loads the ~17 GB pipeline (one time, ~1–2 minutes from disk).
- Input images up to 10 MB / 50 megapixels; output size follows the input
  (rounded to multiples of 16, max 1536) unless `size` is given.
- Host RAM: the streaming pipeline holds ~30 GB of weights in RAM
  (`image-offload=true`; GPU memory use is then tiny — ~350 MB). A 32 GB
  GPU box (g6.2xlarge class) is the working minimum and runs at the edge of
  RAM; **64 GB (g6.4xlarge class) is the comfortable tier**, and required if
  chat models serve heavy traffic on the same box.
- Disk: the image model set is ~30 GB on top of the chat models — size the
  volume accordingly (100 GB+ recommended for a full-featured GPU node).

---

## 9. Not in this release

- **AMD GPU (`backend=rocm`)** is not yet available and fails closed.
- **Windows** is not supported.