Run private RAG on one EC2 box — and stop paying by the token

Retrieval-augmented generation has a quiet cost problem: it re-sends the same tokens over and over. Every query ships your system prompt, your formatting rules, and the same hot document chunks back to a metered API that charges you for them every single time. Multiply by every employee, every day, and "pennies per query" becomes the fastest-growing line on your cloud bill.

There's a simpler shape for this workload: put the model inside your VPC on a fixed-price CPU instance, point your existing RAG stack at it, and let the meter stop. Here's exactly how we run it, what it costs, and the measured numbers behind every claim — from a full benchmark day on a single c9g.4xlarge (16 vCPU Graviton4, 32 GB, about 70 cents an hour).

On a fresh Ubuntu 24.04 arm64 instance:

curl -fsSL https://inference-server.searchblox.com/install | sudo bash

That one command — measured at 2m12s including all 8 GB of model downloads — leaves you with a systemd service, a generated API key, and an OpenAI-compatible endpoint at http://<host>:8081/v1 serving text, vision, speech-to-text, and text-to-speech from one process. Your RAG stack doesn't change: LangChain, LlamaIndex, or a custom pipeline connects by swapping base_url. Images ride in as standard image_url parts; audio has its own endpoints. Lock port 8081 to your app subnet, front it with your ALB for TLS, snapshot the volume, done.

Nothing — not prompts, not documents, not images — ever leaves your VPC.

Operation	Measured on the c9g.4xlarge
Warm short chat response	~0.3 s
Typical chat / extraction answers	2–7 s
Image + question (vision RAG)	14.0 s avg
Transcribe a ~10 s audio clip	4.0 s avg
Speak a reply (text-to-speech)	10.2 s avg
Read a 5,400-token document, first time	58 s
Same document, any follow-up question	0.5 s
16 concurrent users	zero failures · p95 12.5 s

That last pair is the RAG story in two rows. Prefix caching — on by default — means the expensive part of RAG (re-processing the same system prompt and document context) happens once. Follow-up questions against hot context return in half a second. A per-token API charges full price for those same repeated tokens on every call; here they're free after the first pass. Structure prompts stable-parts-first ([system][documents][question]) and steady-state RAG skips most prompt processing entirely.

Our assumptions, stated plainly (change them and the math moves — the formulas are all here):

• A RAG query = 3,000 input tokens (system prompt + retrieved chunks + question) + 300 output tokens. Image queries fit the same envelope (an image bills as ~1.6–2k input tokens on token-metered APIs).
• The box: c9g.4xlarge at the exact us-east-1 rate ($0.6955/hr on-demand; ~$0.50/hr on a 1-yr compute savings plan) + 100 GB gp3 → $373/month flat on the 1-yr plan.
• Capacity from measured throughput (93 tok/s prompt processing, 39 tok/s batched generation): ~40 s of compute per cold query, ~24 s with a 50% prefix-cache hit rate — conservative for real RAG traffic.
• Per-token tiers at public list prices: premium ($3/M input + $15/M output), mid ($1/M + $5/M), budget ($0.06/M + $0.24/M).
• We compare on-demand API pricing because that's the like-for-like: managed reserved-capacity tiers start at thousands per month.

RAG queries / month	Premium per-token	Mid per-token	One c9g.4xlarge
10,000	$135	$45	$373
28,000	$378 ← break-even	$126	$373
60,000	$810	$270	$373
83,000	$1,120	$374 ← break-even	$373
200,000	$2,700	$900	$746 (2 nodes)
1,000,000	$13,500	$4,500	~$2,100 (6–8 nodes, 3-yr)

Read it honestly: below ~28k queries a month, on-demand APIs are cheaper — self-hosting earns its keep from roughly 65 daily users (20 queries each) upward, wins 3–6× against premium tiers at scale, and reaches parity with mid tiers at one box's utilization. Against budget-tier models the argument isn't price — it's that your regulated documents never touch a shared endpoint, and your inference bill is a number you know in advance.

And one box replaces four meters: the same $373 node that answers text queries also reads screenshots (~$2.30 per 1k images vs ~$10.50 on a premium token API), transcribes calls (~$0.58 per 1k clips vs $4 on transcription services), and speaks replies — with idle text capacity absorbing the media work for free.

Cheap inference nobody trusts is worthless, so we test in public: a 380-prompt pack spanning 13 industries — banking KYC extraction, insurance claim photos, clinical protocols, manufacturing safety videos, call-center audio, government forms — exercising every API surface: chat, tool calling, strict JSON, 1,000-word document recall, multilingual content, images, video, speech in and out. (Browse the pack →)

On the $0.70/hour box: 360 of 380 passed (95%) with zero server errors, including 20/20 speech-to-text, 12/12 text-to-speech, and every tool-calling and structured-JSON prompt — the categories RAG pipelines live on (long-context recall 19/19, extraction 13/13). The pack ships with the product, so you can run the same 380 prompts against your own install and check us.

Scale-out is built in, not bolted on. On a second instance, install the same way and add one line:

cluster.seed=<first-node-ip>:8081

The new node joins by gossip, health-checks its peers, and every node routes every request to the best home — model-residency aware (requests go where the model is loaded) and prefix-cache aware (follow-up questions route to the node that already holds that document context, protecting the 0.5 s repeat latency as you grow). We validated this live on a three-node Graviton cluster: even load spread, streaming through the cluster, and automatic failover when we killed a node mid-traffic.

So the growth path costs what the first step cost: node two is another $373, doubling capacity and adding redundancy. No external router tier, no re-architecture — and if a workload eventually wants GPU speed, the same binary takes a one-flag CUDA add-on (~10× decode) with the same API and models.

For a 500-person organization running text + image RAG (~10k queries/day), a premium per-token API bills roughly $2,700–3,000 a month — and the number grows with adoption. One c9g.4xlarge at $373 flat serves that load privately, answers warm queries in ~0.3 s and cached document follow-ups in 0.5 s, reads images, transcribes and speaks — validated by a published 380-prompt, 13-industry test at 95% pass. When you need more, the next node is one command and one config line away.

The install is one line. The meter never starts.

curl -fsSL https://inference-server.searchblox.com/install | sudo bash

Numbers: August 2026 benchmark campaign on c9g.4xlarge; EC2 rate from the AWS pricing API; public list prices for per-token tiers. Assumptions stated above — rerun the math with your own query shapes. See Measured Performance for the full architecture matrix and Getting Started for sizing.