What is the SearchAI Inference Server?

A private AI inference server from SearchBlox that runs LLMs entirely inside your own network and serves them through one OpenAI-compatible API — chat, RAG, function calling, JSON output, vision, video, speech, and image editing. No data leaves your infrastructure.

Is the SearchAI Inference Server free?

Yes. It is free to use with no license key, under the Elastic License 2.0. Optional commercial support plans and custom distilled models are available from SearchBlox.

Do I need a GPU to run it?

No. It is built to serve enterprise AI workloads on the CPUs you already own — document Q&A, summarization, extraction, function calling, vision, and speech at reading speed or better. An optional one-flag NVIDIA GPU add-on delivers roughly 10x throughput when needed.

What hardware do I need?

A Linux host (x86_64 or arm64) with 32 GB RAM runs the default full multimodal install (4B chat + vision, speech-to-text, text-to-speech). A 16 GB host runs the lean 4B chat + vision install, and the 2B model runs comfortably on 16 GB. Mac and Windows are supported via Docker.

Does it work with the OpenAI SDKs?

Yes. The API is OpenAI-compatible, so any OpenAI SDK or OpenAI-compatible client works unchanged — you only change the base_url to your own host and use your generated API key.

One line on any Linux host with outbound HTTPS: curl -fsSL https://inference-server.searchblox.com/install | sudo bash. On Apple Silicon Macs: curl -fsSL https://inference-server.searchblox.com/install-macos | sudo bash. The installer downloads the models, generates an API key, and starts the service in 2-10 minutes. A Docker image is also available for Mac, Windows, and container platforms, and a native Windows x64 build installs with one PowerShell line: irm https://inference-server.searchblox.com/install-windows | iex.

Yes. The server runs entirely inside your network — on-prem, air-gapped, regulated, or edge environments. Zero bytes of your data leave the network, and there is no metered or per-token billing.

Can it scale across multiple servers?

Yes. Nodes cluster with model-aware, load-aware, and cache-aware routing: any node accepts any request and forwards it to a node with the right model resident, so different nodes can serve different models behind one endpoint.

Free to use · no license key

Run Private LLMs on
CPUs.

SearchAI Inference Server runs private AI models inside your network and serves them through one OpenAI-compatible endpoint — chat, RAG, function calling, JSON output, vision, video, speech, and image editing. No data egress. No metered billing. No GPUs required.

Install in one line Browse 380 tested prompts

Linux x86_64 · arm64 CPU or NVIDIA GPU 2–10 min install

root@host — install

$ curl -fsSL https://inference-server.searchblox.com/install | sudo bash fetching models … 4B chat+vision, asr, tts api key generated → /etc/searchai/server.properties ✓ service started on :8081 $ curl http://<host>:8081/health {"available":true, "models":["q35-4b"]} $ open http://<host>:8081/console

Console ships with 380 ready-made prompts across 13 industries — copy any one as Python, cURL or JSON.

bytes of your data leaving the network

380

tested prompts across 13 industries

OpenAI-compatible endpoint for all modalities

Free

to download and run — support plans optional

Pick your install

A Linux host (Ubuntu 22.04+, Debian 12+, Amazon Linux 2023, RHEL 9+ — x86_64 with AVX2, or arm64), an Apple Silicon Mac (macOS 13+), 64-bit Windows, or Docker — with outbound HTTPS. Details in the supported-platforms table.

full multimodal · 32 GB RAM

curl -fsSL https://inference-server.searchblox.com/install | sudo bash

Installs 4B chat + vision, speech-to-text and text-to-speech, generates an API key, and starts the service. Then open http://<host>:8081/console.

On Windows Server or desktop? A native Windows x64 build installs with one PowerShell line — it takes a few extra steps (elevated PowerShell, SmartScreen on the unsigned binary), so follow the step-by-step Windows install guide →

Runs fully on CPU

You do not need a GPU to run enterprise AI.

On the CPUs you already have: grounded document Q&A, summarization, extraction to JSON, function calling, vision and speech at reading-speed-or-better — with latency you can write into an SLA.

No accelerator to reserve, no GPU dependency, no second system to secure
Built for on-prem, air-gapped, regulated and edge deployments
One self-contained artifact; predictable latency, no cold starts

Where a GPU still wins

Long open-ended generation, very large models, and many concurrent token streams. For those, the same install takes a one-flag GPU add-on.

3–4×

measured decode with BACKEND=cuda — A100, A10/A10G, L4, RTX 40-series, H100. See the 9-model measurements →

One endpoint, everything supported

Every capability below is served from http://<host>:8081/v1 with a Bearer key. Any OpenAI-compatible SDK works unchanged.

/v1/chat/completions

Chat & documents

Token streaming, grounded RAG answers over your own content, thinking mode for deeper reasoning.

tools + mcp

Function calling

Every model in the catalog does function calling — drives agent frameworks including MCP-based clients; guaranteed JSON with response_format.

vision

Images & video

Photos, forms, charts and video understanding — Gemma 4 serves media natively (12B encodes an image in 19 ms) — plus plain-language image editing via add-on.

audio

Speech in and out

Speech-to-text and text-to-speech, including file-based voice cloning. An ASR sidecar gives text-first models audio input on the same endpoint.

/v1/embeddings + /v1/rerank

Search & RAG retrieval

Multimodal embeddings (text and images in one vector space, Matryoshka dimensions) and a second-stage reranker — the complete embed → recall → rerank → generate pipeline on one box.

/v1/score

Classify & route

Fixed-answer scoring — send a query and the allowed choices, get a decision plus a probability distribution in one prefill, no decode. Deterministic intent routing, triage and labeling on Qwen 3.5 and Spark models.

console

Model workbench built in

Side-by-side model comparison — including any external OpenAI-compatible endpoint — a 380-prompt library, and a one-click Prompt Optimizer that critiques outputs and suggests better prompts.

ops

Cluster-ready

Model-aware, load- and cache-aware routing: any node accepts any request, different nodes serve different models.

trust

Auditable by design

Built-in HTTPS, Bearer-key auth, deterministic outputs, usage.total_time_ms on every response.

Model catalog

Full catalog with fetch commands →

Model

Modalities

RAM

Notes

q35-0.8bQwen 3.5 · 0.8B

text · tools

8 GB

Fastest, text-only

q35-2bQwen 3.5 · 2B

text · vision · audio · video · tools

16–32 GB

Light multimodal

q35-4bQwen 3.5 · 4B

text · vision · audio · video · tools

32 GB

Flagship default

q35-9bQwen 3.5 · 9B

text · vision · audio · video · tools

32 GB

Best text + vision Qwen

q35moe-35bQwen 3.5 · 35B MoE

text · vision · audio* · video · tools

64 GB

General mixture-of-experts

coder-nextQwen 3.5 · 35B MoE

text · tools

64 GB

Agentic coding

qwen38-27bQwen 3.8 · 27B

text · vision · audio* · video · tools

64 GB

Thinking mode + reasoning_effort levels

qwen38-flash-next-q2Qwen 3.8 · 125B MoE (6B active)

text · tools

128 GB

Frontier-scale MoE — thinking with reasoning_content

spark-x25-4bSpark X2.5 · 4B

text · tools

8 GB

Apache-2.0 thinking model, reasoning_content

spark-x25-1.7bSpark X2.5 · 1.7B

text · tools

4 GB

Smallest thinking model — fits 4 GB hosts

gemma-4-E2B-itGemma 4 · E2B

text · vision · audio · video · tools

8 GB

Full multimodal, smallest

gemma-4-E4B-itGemma 4 · E4B

text · vision · audio · video · tools

16 GB

Full multimodal

gemma-4-12B-itGemma 4 · 12B

text · vision · audio · video · tools

32 GB

Media flagship — image encode in 19 ms

gemma-4-26B-A4B-itGemma 4 · 26B A4B MoE

text · vision · audio* · video · tools

32 GB

Highest pack quality (329/380), ~10 s answers

Load any of these from the console's Models tab with one click — the server downloads and brings it online. audio* = served via the ASR sidecar (configure asr-model). Low-RAM option: SAI_G4_W4=only runs Gemma 12B in ~7 GB. See the full model catalog for sizes and fetch commands.

Size it in one glance

Measured performance →

Deployment

vCPU

RAM

What to expect

Pilot / small team4B model

32 GB

≈23 tokens/sec per user — faster than reading speed

Department4B model, 8–16 users

32–64 GB

≈28 tokens/sec per user; batching handles concurrency

Fast / light2B model

16 GB

40–50 tokens/sec per user

Multimodal flagshipGemma 12B

32 GB

Native vision, audio and video in one model — image encode in 19 ms

Best-quality MoEGemma 26B A4B

16–32

32 GB

Highest answer quality in the catalog (329/380 pack); ~10 s answers

Deep reasoningQwen 3.8 27B

16–32

64 GB

Thinking mode with reasoning_effort levels; plan for a 64 GB node

Edge thinkingSpark X2.5 1.7B / 4B

4–8 GB

Apache-2.0 thinking models with reasoning_content — ~55 (1.7B) and ~26 (4B) tokens/sec on M4-Max-class Arm

Frontier MoEQwen 3.8 Flash-Next 125B

16–32

128 GB

125B knowledge at 6B-active-parameter cost — thinking with reasoning_content; plan for a 128 GB host

Low-RAM Gemma tierSAI_G4_W4=only

8–16 GB

Gemma E2B on an 8 GB host; Gemma 12B in ~7 GB RSS on a 16 GB host, instant load

Optional NVIDIA GPUone-flag add-on

—

24 GB VRAM

~10× faster — 4B ≈ 70, 2B ≈ 138 tokens/sec

Default install (chat + vision + speech) wants 32 GB; a 16 GB host runs the lean 4B install. Testing on Mac or Windows? Use a local Linux VM (UTM, Lima, WSL2) or Docker Desktop — the console and API work from your normal browser.

From one node to a fleet

Cluster guide →

Start on one box. When you outgrow it, add nodes with the same install and one shared secret — no external load balancer, no coordinator.

step 1

Add a node

Same one-line install, point it at any existing peer. Nodes gossip every 2 seconds and discover the fleet automatically.

step 2

Smart routing, built in

Any node accepts any request and routes it: model-aware (to a node with that model loaded), then load-aware (least busy), then cache-aware (multi-turn chats stick to the node holding their prefix cache — repeat turns skip re-processing entirely).

step 3

Watch it from anywhere

The console’s Admin tab shows the whole fleet from any node: per-node RAM, CPU load and loaded models. Mix specialties — two nodes on the 4B, one on the 27B — and clients still hit one address.

From zero to your own app in six steps

Run the install command

2–10 minutes depending on model download. The service starts automatically and restarts on reboot.

Save the API key

Printed at the end of the install and stored at /etc/searchai/server.properties.

Verify the server

curl http://<host>:8081/health — expect {"available":true, …}.

Open the prompt console

Paste your key, pick your industry in the left nav, and run ready-made prompts on your own content.

Copy a prompt into your app

Every prompt has Copy Python, Copy cURL and Copy JSON.

Grow when ready

Raise concurrency, add cluster nodes, or add a GPU host — see the Getting Started guide.

python · openai sdk

from openai import OpenAI client = OpenAI( base_url="http://<host>:8081/v1", api_key="YOUR_API_KEY", ) resp = client.chat.completions.create( model="q35-4b", messages=[{"role": "user", "content": "Summarize this contract"}], ) print(resp.choices[0].message.content)

Same code, same SDK — only the base_url changes.