Give your local LLM a memory: sessions + long-term recall, built into the server

Chat APIs are stateless: every request re-sends the whole conversation, every app reinvents history storage, and nothing is remembered between sessions. Cloud vendors solve this with hosted conversation state — which means your users' conversations live in someone else's database.

As of v1.1.0, the SearchAI Inference Server owns this server-side, on your box: sessions (the server keeps the turns), background summarization (old turns fold into a rolling summary so context never explodes), and long-term memory (durable facts extracted in the background and recalled across sessions by hybrid vector + keyword search). It's all additive — requests without these fields behave exactly as before — and all of it, including the embeddings, runs on the models already on your server. Nothing leaves your network.

Add a session id to a normal /v1/chat/completions call. Send only the new message each turn — the server prepends the stored history:

curl -s http://127.0.0.1:8081/v1/chat/completions \
  -H "Authorization: Bearer $SEARCHAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "q35-4b",
    "session": "support-ticket-4821",
    "messages": [{"role":"user","content":"The customer name is Dana Reyes and she prefers teal."}]
  }'

Next turn, just ask — no history resend:

-d '{
  "model": "q35-4b",
  "session": "support-ticket-4821",
  "messages": [{"role":"user","content":"What color should the mockup use?"}]
}'
# → "teal" — the server supplied the context

With the OpenAI Python SDK it's one extra parameter:

client.chat.completions.create(
    model="q35-4b",
    messages=[{"role": "user", "content": "What color should the mockup use?"}],
    extra_body={"session": "support-ticket-4821"},
)

Sessions are manageable over the API — GET /v1/sessions lists them, GET /v1/sessions/<id> shows one, DELETE removes it — and in the console's Memory tab.

One contract to remember: because the server owns the history, re-sending your full message list would duplicate turns. New messages only.

The server assembles session context stable parts first — system prompt, rolling summary, stored turns, then your new message. That ordering means every follow-up turn re-uses the server's prefix cache instead of re-ingesting the conversation: in our gate runs a follow-up turn that recalled earlier session facts answered in 622 ms end-to-end on a CPU box.

And because the prefix cache itself now persists (v1.1.0), session warmth survives restarts: a 4,400-token context that took 24.1 s to ingest cold re-loaded from disk in 2.9 s after a full server restart — token-exact. In a cluster, nodes advertise and pull each other's largest caches: in our two-node test, a node that died mid-conversation had its session continued by the surviving peer, full history intact, and the peer served the transferred cache in 268 ms — about 100× faster than recomputing it.

Long conversations don't grow without bound. When a session exceeds a size threshold (sessions-compact-bytes, default 24000) or a turn count you choose (sessions-max-turns), the server folds the oldest half of the turns into a rolling summary — in the background, off the request path, using whichever model you point sessions-summary-model at. The 0.8B model group is ideal: summarization is exactly what a small model is good at, and it leaves your serving model alone.

Recent turns always stay verbatim — research on conversational memory is clear that verbatim recent context beats summaries for recall, so only the old tail is compacted.

During that same background pass, the server extracts durable facts — preferences, entities, standing constraints — into a per-user memory store. Scope memories to a user by the standard OpenAI user field and they follow that user across sessions:

# session A, Monday
-d '{"model":"q35-4b", "session":"onboarding-1", "user":"dana",
     "messages":[{"role":"user","content":"We are migrating Acme Corp to the enterprise plan this quarter."}]}'

# session B, a NEW session days later
-d '{"model":"q35-4b", "session":"checkin-7", "user":"dana",
     "messages":[{"role":"user","content":"Which customer am I migrating, and to what plan?"}]}'
# → "Acme Corp, to the enterprise plan" — recalled from long-term memory

You can also write and manage memories directly:

curl -X POST http://127.0.0.1:8081/v1/memories \
  -H "Authorization: Bearer $SEARCHAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"scope":"dana","text":"Always answer in metric units."}'

curl http://127.0.0.1:8081/v1/memories/dana \
  -H "Authorization: Bearer $SEARCHAI_API_KEY"      # list dana's memories
# DELETE /v1/memories/dana/<id> removes one

Recall is hybrid: the query is embedded (with the on-box embedding model) and matched by vector similarity, and ranked by exact BM25 keyword scoring — the two lists fused so that exact identifiers ("ticket ACME-4821") are found even when embeddings would miss them. Optionally, a cross-encoder reranker (memories-rerank-model) orders the final candidates. Recalled memories are injected into the context automatically — you just ask.

Fresh installs of v1.1.0 wire this up automatically. On an existing install, add to /etc/searchai/server.properties:

sessions-dir=/var/lib/searchai/sessions
memories-dir=/var/lib/searchai/memories
sessions-summary-model=q35-0.8b          # small model for background work
memories-embed-model=qwen3vl-embed-2b    # powers vector recall

Sessions work with no extra models at all; summarization needs the 0.8B group and memory recall needs the embed model (the same one that powers /v1/embeddings). Both are one installer group away.

Privacy note, because it's the point: session and memory files hold conversation content. They live on your disk, 0700, under the service user — treat them with the same care as logs. There is no second copy anywhere.

One request field gives your applications server-side conversation state; the user field gives them memory that crosses sessions; background summarization keeps context bounded; and the whole stack — storage, embeddings, keyword search, recall — runs on the hardware you already own, surviving restarts and following your cluster.

curl -fsSL https://inference-server.searchblox.com/install | sudo bash
# then: add "session": "my-first-session" to any chat request

Timings measured 2026-09-02/03 during our release gates: turn-2 session recall on an Apple M4 (CPU); restart reload (24.1 s → 2.9 s, 4,418-token context, q35-2b) on the same box; cluster cache transfer (268 ms) and session fail-over on a two-node AWS Graviton/x86 pair. Full details in the deployment guide.