# SearchAI Inference Server — Getting Started for Your Organization

From zero to embedded AI in four steps: **install → verify → validate prompts
→ integrate into your applications**. This guide is for the team adopting the
server (developers, architects, IT); the companion `BUILD_AND_DEPLOY.md` is
the full operations reference (model management, clustering detail, GPU,
security hardening).

**What you get:** a private, self-hosted AI server with an
**OpenAI-compatible API** — chat, documents, function/tool calling, JSON
output, vision (images), video, speech-to-text, text-to-speech, and image
editing — running entirely on your own infrastructure. No data leaves your
network; any OpenAI-compatible SDK, framework, or tool works against it by
changing two settings (base URL + API key).

---

## 1. Prerequisites and sizing

**Host requirements:**

- Linux, `x86_64` or `aarch64` (Arm), with `sudo`. (Testing from a Mac or
  Windows PC? The quickest path is the Docker image — one command with
  Docker Desktop, no VM setup:

  ```bash
  docker run -d --name searchai -p 8081:8081 -v searchai-models:/models \
    public.ecr.aws/m0m7b0k9/searchai-inference-server
  ```

  Same defaults as the installer (4B chat + vision + speech); models download
  into the named volume on first start and the auto-generated API key is
  printed in `docker logs searchai`. On a smaller Docker VM use
  `-e MODELS='4b'` and allocate the VM memory in Docker Desktop settings.
  Alternatively use any lightweight Linux VM — UTM, Lima, WSL2 — or a small
  cloud instance. The GPU add-on supports NVIDIA A100, A10/A10G, L4,
  RTX 40-series, and H100 GPUs on Linux hosts.)
- Outbound HTTPS (the installer downloads the server and models from a
  public URL — no cloud account or credentials required)
- Open one TCP port for the API (default **8081**)

**How much CPU and memory?** Start from the model set, then add headroom per
concurrent user:

| Deployment size | Models | Concurrent users | vCPU | RAM |
|---|---|---:|---|---|
| Pilot / small team | 4B text + vision | 1–4 | 8–16 | 16–32 GB |
| Department | 4B text + vision | 8–16 | 16–32 | 32–48 GB |
| Department + voice | 4B + speech-to-text + text-to-speech | 8–16 | 16–32 | 48–64 GB |
| Organization-wide | 4B | 32–64+ | cluster, 3+ nodes | 32–64 GB per node |

Rules of thumb (full tables in `BUILD_AND_DEPLOY.md` §"Sizing & performance"):

- **RAM follows the models you keep loaded** (4B text+vision ≈ 5–7 GB
  resident; audio adds ~10–14 GB with load headroom; 35B needs a 64 GB box),
  **plus ~1–2 GB per concurrent user** for in-flight request state.
- **More vCPU** = faster long-prompt processing and more simultaneous users.
  **Faster memory bandwidth** = faster per-user response speed (the biggest
  hardware lever). Typical per-user speeds: 2B ≈ 40–50 tokens/sec; 4B ≈ 28 tokens/sec on a
  32-vCPU node (c9g.8xlarge-class) and ≈ 23 on a 16-vCPU node
  (c9g.4xlarge-class) — all faster than reading speed.
- **GPU (optional):** an NVIDIA-equipped host serves the same API ~10× faster
  (4B ≈ 70 tok/s on one L4) — see `BUILD_AND_DEPLOY.md` §7 for the one-flag
  install and GPU instance sizing.

**Disk:** ~10 GB for the 4B + audio model set; ~40 GB with the 9B and image
editing added.

## 2. Install (one command)

**Install with one line** (defaults: full multimodal — 4B chat + vision +
speech-to-text + text-to-speech, 32 GB RAM recommended; auto-generated API
key; service starts on port 8081). On a 16 GB host use
`... | sudo MODELS='4b' MIN_RAM_GB=8 bash` for chat + vision only:

```bash
curl -fsSL https://inference-server.searchblox.com/install | sudo bash
```

Options ride as environment variables, e.g.
`... | sudo MODELS='4b asr tts' bash` or `... | sudo API_KEY='my-secret' BACKEND=cuda bash`.
The download & docs landing page is
**https://inference-server.searchblox.com/index.html**.

Prefer to inspect before running? Fetch the scripts and run them yourself:

```bash
mkdir -p deploy && cd deploy
BASE=https://inference-server.searchblox.com/deploy
curl -fsSLO $BASE/install-searchai.sh -O $BASE/fetch-models.sh \
     -O $BASE/BUILD_AND_DEPLOY.md -O $BASE/GETTING_STARTED.md
chmod +x *.sh
sudo API_KEY='choose-a-long-secret' MODELS='4b asr tts' ./install-searchai.sh
```

`MODELS` picks what to install: `4b` (text + vision), plus optional `asr`
(speech-to-text), `tts` (speech), `voice` (voice cloning), `2b`/`9b`
(smaller/larger chat models), `image` (image editing — needs 32 GB+ RAM and
ideally a GPU). The service starts automatically and restarts on reboot.

Sizing note for the install command: on a **16 GB host**, add `MIN_RAM_GB=8` before `./install-searchai.sh`
(the startup safety gate assumes 16 GB free by default) and install
`MODELS='4b'` only — the audio models alongside 4B need a **32 GB** host
(see the sizing table above).

**What happens on an undersized host?** The system fails safely, in
three layers:

1. **At install:** the installer checks available RAM first (`MIN_RAM_GB`,
   default 16) and refuses with a clear error before anything starts —
   overriding it (e.g. `MIN_RAM_GB=8`) is a conscious choice.
2. **At runtime:** before loading a model or admitting a request, the server
   checks free memory against its floor (`min-free-mb`, auto-set to ~1/8 of
   RAM). If there isn't room it returns **HTTP 503** with code
   `native_low_memory` and the actual free-vs-floor numbers in the message —
   the service stays up and everything already loaded keeps working. Seeing
   this error means the host needs more RAM for its workload (or unload a
   model first).
3. **Worst case:** loading a model has a brief memory spike above its final
   footprint; on a saturated box that can trip the operating system's
   out-of-memory protection, which stops the server process — systemd
   restarts it automatically and chat service recovers. This edge is why
   the full multimodal set firmly requires **32 GB**.

The installer also sizes the server's reusable request-state pool to the
host's RAM automatically, so on 16–32 GB machines there is always headroom
left to load an audio or vision model on demand even after long periods of
heavy chat traffic — no tuning needed.

On a 16 GB machine: install `MODELS='4b'` only, and if speech is needed
occasionally, keep one heavy modality resident at a time
(`POST /v1/models/unload` between them).

**Verify:**

```bash
curl -fsS localhost:8081/health
# {"available":true, ...}

KEY='choose-a-long-secret'
curl -sS localhost:8081/v1/chat/completions -H "Authorization: Bearer $KEY" \
  -H 'Content-Type: application/json' \
  -d '{"model":"q35-4b","messages":[{"role":"user","content":"Hello"}],"max_tokens":64}'
```

## 3. Validate prompts for YOUR use cases (the console)

Open **`http://<host>:8081/console`** in a browser and paste your API key
into the key field. The console ships a **380-prompt pack organized by 13
industries** (Financial Services, Insurance, Healthcare, Manufacturing,
Energy, Retail, Life Sciences, Government, Technology, Legal, Education,
Telecom, Logistics) and by use case (document understanding, extraction to
JSON, tool calling, classification, multilingual, vision, and more).

The evaluation workflow for a business:

1. Click your **industry** in the left nav — you'll see ready-made scenarios
   for your domain (e.g., Insurance: claim intake to JSON, policy-document
   Q&A, damage-photo description, claim-status tool calls).
2. **Run** the prompts closest to your intended use, with your own text
   substituted in. Image/video prompts have an upload box that states the
   supported formats and size limits. The metrics line shows response time
   and token counts.
3. Compare models from the dropdown (e.g., 2B vs 4B) for the speed/quality
   trade-off on your content.
4. Audio and image-editing prompts display an **"Example request"** with the
   exact HTTP call and payload for their endpoint, ready to copy.

## 4. Integrate into your application (embedded AI)

When a prompt looks right in the console, take it into your code — every
prompt card has **Copy cURL**, **Copy JSON**, and **Copy Python** buttons that
emit the exact request. The API is OpenAI-compatible, so integration is the
same two-line change in any language or framework:

**Python (official OpenAI SDK):**

```python
# pip install openai
from openai import OpenAI

client = OpenAI(base_url="http://<host>:8081/v1", api_key="YOUR_API_KEY")

resp = client.chat.completions.create(
    model="q35-4b",
    messages=[
        {"role": "system", "content": "You are a claims intake assistant."},
        {"role": "user", "content": "I need to file a claim for hail damage."},
    ],
    temperature=0,
)
print(resp.choices[0].message.content)
```

**JavaScript / TypeScript:**

```js
// npm install openai
import OpenAI from "openai";
const client = new OpenAI({ baseURL: "http://<host>:8081/v1", apiKey: "YOUR_API_KEY" });

const resp = await client.chat.completions.create({
  model: "q35-4b",
  messages: [{ role: "user", content: "Summarize this policy: ..." }],
  stream: true,                       // token streaming works out of the box
});
for await (const chunk of resp) process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
```

**Any OpenAI-compatible framework** (LangChain, LlamaIndex, Spring AI,
Semantic Kernel, low-code tools): point the OpenAI provider at
`http://<host>:8081/v1` with your API key — no other changes.

**The capabilities you validated in the console map to standard API features:**

| Use case | How to call it |
|---|---|
| Structured output for your systems | `response_format: {"type": "json_object"}` — guaranteed-parseable JSON |
| Let the model call your business functions | `tools: [...]` — response returns `tool_calls` with typed arguments; execute them and send results back as `role: "tool"` messages |
| Documents / long content | Put the document in the message; summaries, Q&A, extraction all work to the context limit (default **8,192 tokens** ≈ 20–25 pages of text; raise `max-context` in `server.properties` for longer documents — see the Deployment Guide sizing note) |
| Images (invoices, damage photos, labels, charts) | `image_url` content part with a base64 data URL (PNG/JPEG/BMP/GIF, ≤10 MB each) |
| Video | `video` part (frame images) or `video_url` (MP4, media requests ≤64 MB total) |
| Speech-to-text | `POST /v1/audio/transcriptions` — multipart WAV upload (any sample rate) → `{"text": ...}` |
| Text-to-speech | `POST /v1/audio/speech` — `{"input": "..."}` → WAV audio |
| Image editing | `POST /v1/images/edits` — image + instruction → edited PNG |
| Response timing for your dashboards | non-streaming responses include `usage.total_time_ms` |
| Token streaming | `stream: true` — server-sent events, works with the OpenAI SDKs' streaming iterators |
| Deeper reasoning on demand | `thinking: true` — the model reasons before answering (slower, stronger on hard problems) |
| Reproducible outputs | `temperature: 0` is fully deterministic; with sampling, pass `seed` |
| Agent frameworks / MCP | the `tools` interface drives any agent stack — including MCP-based clients: the model emits the tool calls, your client executes its MCP tools and returns results as `role: "tool"` messages |
| Voice cloning | drop a `<name>.wav` (10–30 s clean speech) into the models `tts-voices/` folder, then request `"voice": "<name>"` |
| Multiple models on one server | all installed models are served; `POST /v1/models/load` / `unload` manages residency at runtime |

**Production integration tips:**

- **Keep the API key server-side** (your backend calls the inference server;
  browsers/mobile apps call your backend).
- **Enable HTTPS** for any cross-network traffic. Single-node TLS is built in —
  point the server at a PEM certificate and key in
  `/etc/searchai/server.properties` and restart:

  ```properties
  server.ssl-cert=/etc/searchai/certs/fullchain.pem
  server.ssl-key=/etc/searchai/certs/privkey.pem
  ```

  ```bash
  sudo systemctl restart searchai
  curl -fsS https://<host>:8081/health   # now HTTPS
  ```

  Any CA-issued or internal-PKI certificate works (e.g. a Let's Encrypt
  `fullchain.pem`/`privkey.pem` pair). For clusters, terminate TLS at your
  load balancer instead — node-to-node ports stay on the private network.
- **Handle capacity signals:** HTTP **429** = concurrency queue full (retry
  with backoff), HTTP **503** with `native_low_memory` = the host needs more
  RAM for the workload. Both are explicit — you'll never hit a silent OOM.
- **Set client timeouts generously** for long documents on CPU (the console's
  metrics line tells you your real latencies).
- **Reproducibility:** `temperature: 0` responses are fully deterministic —
  good for regression-testing your prompts in CI. With sampling, pass `seed`
  for reproducible variety.

## 5. Scaling and adding more users

Adding users is a capacity question — the API and your integration do not
change.

1. **Tune the node you have.** `max-inflight` (in
   `/etc/searchai/server.properties`) sets how many requests generate
   simultaneously (default 4). Raise it as you add vCPU/RAM — validated
   stable to concurrency 64 on a 64 GB node. Keep `min-free-mb` set so the
   server sheds load gracefully instead of exhausting memory.
2. **Scale up** for heavier single requests: more vCPU shortens long-prompt
   processing; faster memory gives every user quicker responses; more RAM
   admits more concurrent users and more loaded models.
3. **Scale out (the path for organization-wide use):** install the same build
   on additional nodes and join them into a **cluster** — any node accepts
   requests and load-balances across the fleet, so aggregate capacity grows
   with node count (measured: a 3-node 4B cluster sustains ~1.5–1.65× a
   single node, improving as concurrency rises). See `BUILD_AND_DEPLOY.md`
   §3 for the two-command cluster join.
4. **Or add a GPU node** for a ~10× per-node throughput jump on the same API
   (`BACKEND=cuda` install; GPU instance sizing table in §7).

**Suggested rollout:** pilot on one 16–32 GB node with your team → validate
your industry's prompts in the console → integrate one workflow via the SDK →
raise `max-inflight` as usage grows → add cluster nodes (or a GPU node) when
sustained concurrency outgrows the box.

## 6. Where everything lives

| | |
|---|---|
| API base URL | `http://<host>:8081/v1` |
| Prompt console | `http://<host>:8081/console` |
| Health/monitoring | `GET /health` (no auth) |
| Config | `/etc/searchai/server.properties` |
| Logs | `journalctl -u searchai -f` |
| Models on disk | `/var/lib/searchai/models/` |
| Load/unload models at runtime | `POST /v1/models/load` / `/v1/models/unload` |
| Full operations reference | `BUILD_AND_DEPLOY.md` |