Getting Started

Sizing, install, validating prompts for your industry, integrating with the OpenAI SDKs, and scaling.

From zero to embedded AI in four steps: install → verify → validate prompts → integrate into your applications. This guide is for the team adopting the server (developers, architects, IT); the companion Deployment Guide is the full operations reference (model management, clustering detail, GPU, security hardening).

What you get: a private, self-hosted AI server with an OpenAI-compatible API — chat, documents, function/tool calling, JSON output, vision (images), video, speech-to-text, text-to-speech, and image editing — running entirely on your own infrastructure. No data leaves your network; any OpenAI-compatible SDK, framework, or tool works against it by changing two settings (base URL + API key).

1. Prerequisites and sizing

Host requirements:

• Linux, x86_64 or aarch64 (Arm), with sudo. (Testing from a Mac or Windows PC? The quickest path is the Docker image — one command with Docker Desktop, no VM setup:

bash docker run -d --name searchai -p 8081:8081 -v searchai-models:/models \ public.ecr.aws/m0m7b0k9/searchai-inference-server

Same defaults as the installer (4B chat + vision + speech); models download into the named volume on first start and the auto-generated API key is printed in docker logs searchai. On a smaller Docker VM use -e MODELS='4b' and allocate the VM memory in Docker Desktop settings. Alternatively use any lightweight Linux VM — UTM, Lima, WSL2 — or a small cloud instance. The GPU add-on supports NVIDIA A100, A10/A10G, L4, RTX 40-series, and H100 GPUs on Linux hosts.) - Outbound HTTPS (the installer downloads the server and models from a public URL — no cloud account or credentials required) - Open one TCP port for the API (default 8081)

How much CPU and memory? Start from the model set, then add headroom per concurrent user:

Rules of thumb (full tables in Deployment Guide §"Sizing & performance"):

• RAM follows the models you keep loaded (4B text+vision ≈ 5–7 GB resident; audio adds ~10–14 GB with load headroom; 35B needs a 64 GB box), plus ~1–2 GB per concurrent user for in-flight request state.
• More vCPU = faster long-prompt processing and more simultaneous users. Faster memory bandwidth = faster per-user response speed (the biggest hardware lever). Typical per-user speeds: 2B ≈ 40–50 tokens/sec; 4B ≈ 28 tokens/sec on a 32-vCPU node (c9g.8xlarge-class) and ≈ 23 on a 16-vCPU node (c9g.4xlarge-class) — all faster than reading speed.
• GPU (optional): an NVIDIA-equipped host serves the same API ~10× faster (4B ≈ 70 tok/s on one L4) — see Deployment Guide §7 for the one-flag install and GPU instance sizing.

Disk: ~10 GB for the 4B + audio model set; ~40 GB with the 9B and image editing added.

2. Install (one command)

Install with one line (defaults: full multimodal — 4B chat + vision + speech-to-text + text-to-speech, 32 GB RAM recommended; auto-generated API key; service starts on port 8081). On a 16 GB host use ... | sudo MODELS='4b' MIN_RAM_GB=8 bash for chat + vision only:

curl -fsSL https://inference-server.searchblox.com/install | sudo bash

Options ride as environment variables, e.g. ... | sudo MODELS='4b asr tts' bash or ... | sudo API_KEY='my-secret' BACKEND=cuda bash. The download & docs landing page is https://inference-server.searchblox.com/index.html.

Prefer to inspect before running? Fetch the scripts and run them yourself:

mkdir -p deploy && cd deploy
BASE=https://inference-server.searchblox.com/deploy
curl -fsSLO $BASE/install-searchai.sh -O $BASE/fetch-models.sh \
     -O $BASE/BUILD_AND_DEPLOY.md -O $BASE/GETTING_STARTED.md
chmod +x *.sh
sudo API_KEY='choose-a-long-secret' MODELS='4b asr tts' ./install-searchai.sh

MODELS picks what to install: 4b (text + vision), plus optional asr (speech-to-text), tts (speech), voice (voice cloning), 2b/9b (smaller/larger chat models), image (image editing — needs 32 GB+ RAM and ideally a GPU). The service starts automatically and restarts on reboot.

Sizing note for the install command: on a 16 GB host, add MIN_RAM_GB=8 before ./install-searchai.sh (the startup safety gate assumes 16 GB free by default) and install MODELS='4b' only — the audio models alongside 4B need a 32 GB host (see the sizing table above).

What happens on an undersized host? The system fails safely, in three layers:

1. At install: the installer checks available RAM first (MIN_RAM_GB, default 16) and refuses with a clear error before anything starts — overriding it (e.g. MIN_RAM_GB=8) is a conscious choice.
2. At runtime: before loading a model or admitting a request, the server checks free memory against its floor (min-free-mb, auto-set to ~1/8 of RAM). If there isn't room it returns HTTP 503 with code native_low_memory and the actual free-vs-floor numbers in the message — the service stays up and everything already loaded keeps working. Seeing this error means the host needs more RAM for its workload (or unload a model first).
3. Worst case: loading a model has a brief memory spike above its final footprint; on a saturated box that can trip the operating system's out-of-memory protection, which stops the server process — systemd restarts it automatically and chat service recovers. This edge is why the full multimodal set firmly requires 32 GB.

The installer also sizes the server's reusable request-state pool to the host's RAM automatically, so on 16–32 GB machines there is always headroom left to load an audio or vision model on demand even after long periods of heavy chat traffic — no tuning needed.

On a 16 GB machine: install MODELS='4b' only, and if speech is needed occasionally, keep one heavy modality resident at a time (POST /v1/models/unload between them).

Verify:

curl -fsS localhost:8081/health
# {"available":true, ...}

KEY='choose-a-long-secret'
curl -sS localhost:8081/v1/chat/completions -H "Authorization: Bearer $KEY" \
  -H 'Content-Type: application/json' \
  -d '{"model":"q35-4b","messages":[{"role":"user","content":"Hello"}],"max_tokens":64}'

3. Validate prompts for YOUR use cases (the console)

Open http://<host>:8081/console in a browser and paste your API key into the key field. The console ships a 380-prompt pack organized by 13 industries (Financial Services, Insurance, Healthcare, Manufacturing, Energy, Retail, Life Sciences, Government, Technology, Legal, Education, Telecom, Logistics) and by use case (document understanding, extraction to JSON, tool calling, classification, multilingual, vision, and more).

The evaluation workflow for a business:

1. Click your industry in the left nav — you'll see ready-made scenarios for your domain (e.g., Insurance: claim intake to JSON, policy-document Q&A, damage-photo description, claim-status tool calls).
2. Run the prompts closest to your intended use, with your own text substituted in. Image/video prompts have an upload box that states the supported formats and size limits. The metrics line shows response time and token counts.
3. Compare models from the dropdown (e.g., 2B vs 4B) for the speed/quality trade-off on your content.
4. Audio and image-editing prompts display an "Example request" with the exact HTTP call and payload for their endpoint, ready to copy.

4. Integrate into your application (embedded AI)

When a prompt looks right in the console, take it into your code — every prompt card has Copy cURL, Copy JSON, and Copy Python buttons that emit the exact request. The API is OpenAI-compatible, so integration is the same two-line change in any language or framework:

Python (official OpenAI SDK):

# pip install openai
from openai import OpenAI

client = OpenAI(base_url="http://<host>:8081/v1", api_key="YOUR_API_KEY")

resp = client.chat.completions.create(
    model="q35-4b",
    messages=[
        {"role": "system", "content": "You are a claims intake assistant."},
        {"role": "user", "content": "I need to file a claim for hail damage."},
    ],
    temperature=0,
)
print(resp.choices[0].message.content)

JavaScript / TypeScript:

// npm install openai
import OpenAI from "openai";
const client = new OpenAI({ baseURL: "http://<host>:8081/v1", apiKey: "YOUR_API_KEY" });

const resp = await client.chat.completions.create({
  model: "q35-4b",
  messages: [{ role: "user", content: "Summarize this policy: ..." }],
  stream: true,                       // token streaming works out of the box
});
for await (const chunk of resp) process.stdout.write(chunk.choices[0]?.delta?.content ?? "");

Any OpenAI-compatible framework (LangChain, LlamaIndex, Spring AI, Semantic Kernel, low-code tools): point the OpenAI provider at http://<host>:8081/v1 with your API key — no other changes.

The capabilities you validated in the console map to standard API features:

Production integration tips:

• Keep the API key server-side (your backend calls the inference server; browsers/mobile apps call your backend).
• Enable HTTPS for any cross-network traffic. Single-node TLS is built in — point the server at a PEM certificate and key in /etc/searchai/server.properties and restart:

properties server.ssl-cert=/etc/searchai/certs/fullchain.pem server.ssl-key=/etc/searchai/certs/privkey.pem

bash sudo systemctl restart searchai curl -fsS https://<host>:8081/health # now HTTPS

Any CA-issued or internal-PKI certificate works (e.g. a Let's Encrypt fullchain.pem/privkey.pem pair). For clusters, terminate TLS at your load balancer instead — node-to-node ports stay on the private network. - Handle capacity signals: HTTP 429 = concurrency queue full (retry with backoff), HTTP 503 with native_low_memory = the host needs more RAM for the workload. Both are explicit — you'll never hit a silent OOM. - Set client timeouts generously for long documents on CPU (the console's metrics line tells you your real latencies). - Reproducibility: temperature: 0 responses are fully deterministic — good for regression-testing your prompts in CI. With sampling, pass seed for reproducible variety.

5. Scaling and adding more users

Adding users is a capacity question — the API and your integration do not change.

1. Tune the node you have. max-inflight (in /etc/searchai/server.properties) sets how many requests generate simultaneously (default 4). Raise it as you add vCPU/RAM — validated stable to concurrency 64 on a 64 GB node. Keep min-free-mb set so the server sheds load gracefully instead of exhausting memory.
2. Scale up for heavier single requests: more vCPU shortens long-prompt processing; faster memory gives every user quicker responses; more RAM admits more concurrent users and more loaded models.
3. Scale out (the path for organization-wide use): install the same build on additional nodes and join them into a cluster — any node accepts requests and load-balances across the fleet, so aggregate capacity grows with node count (measured: a 3-node 4B cluster sustains ~1.5–1.65× a single node, improving as concurrency rises). See Deployment Guide §3 for the two-command cluster join.
4. Or add a GPU node for a ~10× per-node throughput jump on the same API (BACKEND=cuda install; GPU instance sizing table in §7).

Suggested rollout: pilot on one 16–32 GB node with your team → validate your industry's prompts in the console → integrate one workflow via the SDK → raise max-inflight as usage grows → add cluster nodes (or a GPU node) when sustained concurrency outgrows the box.

6. Where everything lives