Free to use · no license key

Run Private LLMs on
CPUs.

SearchAI Inference Server runs private AI models inside your network and serves them through one OpenAI-compatible endpoint — chat, RAG, function calling, JSON output, vision, video, speech, and image editing. No data egress. No metered billing. No GPUs required.

Install in one line Browse 380 tested prompts
Linux x86_64 · arm64 CPU or NVIDIA GPU 2–10 min install
root@host — install
$ curl -fsSL https://inference-server.searchblox.com/install | sudo bash fetching models … 4B chat+vision, asr, tts api key generated → /etc/searchai/server.properties ✓ service started on :8081 $ curl http://<host>:8081/health {"available":true, "models":["q35-4b"]} $ open http://<host>:8081/console
Console ships with 380 ready-made prompts across 13 industries — copy any one as Python, cURL or JSON.
0
bytes of your data leaving the network
380
tested prompts across 13 industries
1
OpenAI-compatible endpoint for all modalities
Free
to download and run — support plans optional

Pick your install

A Linux host or Apple Silicon Mac with outbound HTTPS — that is the whole prerequisite list.

full multimodal · 32 GB RAM
curl -fsSL https://inference-server.searchblox.com/install | sudo bash
Installs 4B chat + vision, speech-to-text and text-to-speech, generates an API key, and starts the service. Then open http://<host>:8081/console.
Runs fully on CPU

You do not need a GPU to run enterprise AI.

On the CPUs you already have: grounded document Q&A, summarization, extraction to JSON, function calling, vision and speech at reading-speed-or-better — with latency you can write into an SLA.

  • No accelerator to reserve, no GPU dependency, no second system to secure
  • Built for on-prem, air-gapped, regulated and edge deployments
  • One self-contained artifact; predictable latency, no cold starts
Where a GPU still wins

Long open-ended generation, very large models, and many concurrent token streams. For those, the same install takes a one-flag GPU add-on.

~10×
throughput with BACKEND=cuda — A100, A10/A10G, L4, RTX 40-series, H100

One endpoint, everything supported

Every capability below is served from http://<host>:8081/v1 with a Bearer key. Any OpenAI-compatible SDK works unchanged.

/v1/chat/completions
Chat & documents
Token streaming, grounded RAG answers over your own content, thinking mode for deeper reasoning.
tools + mcp
Function calling
Drives agent frameworks including MCP-based clients; guaranteed JSON with response_format.
vision
Images & video
Photos, forms, charts and video understanding — plus plain-language image editing via add-on.
audio
Speech in and out
Speech-to-text and text-to-speech, including file-based voice cloning.
ops
Cluster-ready
Model-aware, load- and cache-aware routing: any node accepts any request, different nodes serve different models.
trust
Auditable by design
Built-in HTTPS, Bearer-key auth, deterministic outputs, usage.total_time_ms on every response.

Size it in one glance

Measured performance →

Deployment
vCPU
RAM
What to expect
Pilot / small team4B model
16
32 GB
≈23 tokens/sec per user — faster than reading speed
Department4B model, 8–16 users
32
32–64 GB
≈28 tokens/sec per user; batching handles concurrency
Fast / light2B model
8+
16 GB
40–50 tokens/sec per user
Optional NVIDIA GPUone-flag add-on
—
24 GB VRAM
~10× faster — 4B ≈ 70, 2B ≈ 138 tokens/sec

Default install (chat + vision + speech) wants 32 GB; a 16 GB host runs the lean 4B install. Testing on Mac or Windows? Use a local Linux VM (UTM, Lima, WSL2) or Docker Desktop — the console and API work from your normal browser.

From zero to your own app in six steps

1
Run the install command
2–10 minutes depending on model download. The service starts automatically and restarts on reboot.
2
Save the API key
Printed at the end of the install and stored at /etc/searchai/server.properties.
3
Verify the server
curl http://<host>:8081/health — expect {"available":true, …}.
4
Open the prompt console
Paste your key, pick your industry in the left nav, and run ready-made prompts on your own content.
5
Copy a prompt into your app
Every prompt has Copy Python, Copy cURL and Copy JSON.
6
Grow when ready
Raise concurrency, add cluster nodes, or add a GPU host — see the Getting Started guide.
python · openai sdk
from openai import OpenAI client = OpenAI( base_url="http://<host>:8081/v1", api_key="YOUR_API_KEY", ) resp = client.chat.completions.create( model="q35-4b", messages=[{"role": "user", "content": "Summarize this contract"}], ) print(resp.choices[0].message.content)
Same code, same SDK — only the base_url changes.

Direct downloads

Models download automatically during install — 4B text+vision ≈ 3.4 GB, speech ≈ 5 GB extra.

linux arm64
Server binary
Download →sha256
linux x86_64
Server binary
Download →sha256
macos arm64
Server binary (Apple Silicon)
Download →sha256
macos arm64
macOS installer script
Download →
linux x86_64 + nvidia
GPU acceleration add-on
Download →sha256
linux x86_64
Image-editing add-on
Download →sha256
all platforms
Installer script
Download →
all platforms
Model fetcher
Download →

Need a model that speaks your domain?

Distilled and custom models — trained on your terminology, documents and tasks, sized for your hardware — are generated with the SearchAI platform and served by this inference server unchanged.

Talk to SearchBlox See the SearchAI platform
Getting Started
Sizing, install, validating prompts, SDK integration, scaling
Deployment Guide
Model management, clustering, GPU, security
Performance
Measured tokens/sec by model and hardware
380 Tested Prompts
The industry prompt catalog in the console
License & Support
Free to use; optional support plans