Function calling on a 2B model: build a local agent that actually runs tools

Agents are just a loop: the model decides to call a tool, your code runs it, the result goes back, and the model uses it to answer. The only hard requirement is a model that emits clean, correct tool calls — and the assumption is that you need a big one. You don't. Every model the server ships does OpenAI-style function calling, down to the 2B — the one that fits a 16 GB box. This post builds a working agent loop on q35-2b, with the exact API calls and real captured output, including the one small-model gotcha and its one-line fix.

Tools are declared exactly as in the OpenAI API — a JSON-schema description of each function. Send them alongside the message:

curl http://<host>:8081/v1/chat/completions \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{
    "model": "q35-2b",
    "messages": [{"role":"user","content":"What is the weather in Paris? Use celsius."}],
    "tools": [{"type":"function","function":{
      "name":"get_weather",
      "description":"Get the current weather for a city.",
      "parameters":{"type":"object",
        "properties":{"city":{"type":"string"},
                      "unit":{"type":"string","enum":["celsius","fahrenheit"]}},
        "required":["city"]}}}]
  }'

When the model decides the tool applies, it replies with content: null and a structured tool_calls array — name and JSON-string arguments, ready to execute:

{"role":"assistant","content":null,
 "tool_calls":[{"id":"call_18cedd1d…","type":"function",
   "function":{"name":"get_weather",
               "arguments":"{\"city\":\"Paris\",\"unit\":\"celsius\"}"}}]}

Your code parses arguments, runs the real function, and hands the result back — that's the loop.

Return each tool result as a role: "tool" message keyed by the tool_call_id, call the model again, and repeat until it stops calling tools and answers. Here's the whole loop:

def run_agent(user_msg, tools, run_tool):
    msgs = [{"role":"user","content":user_msg}]
    while True:
        m = chat("q35-2b", msgs, tools=tools)   # POST /v1/chat/completions
        msgs.append(m)
        if not m.get("tool_calls"):
            return m["content"]                  # final answer
        for tc in m["tool_calls"]:
            result = run_tool(tc["function"]["name"],
                              json.loads(tc["function"]["arguments"]))
            msgs.append({"role":"tool",
                         "tool_call_id": tc["id"],
                         "name": tc["function"]["name"],
                         "content": json.dumps(result)})

Give the 2B a request that needs two tools and watch it work — this is the real captured run:

> "I'm traveling to Tokyo. What's the weather there,
   and how much is 100 USD in JPY?"

[step 1] get_weather({"city":"Tokyo","unit":"celsius"})
             -> {"city":"Tokyo","temp_c":18,"condition":"partly cloudy"}
[step 1] currency_convert({"amount":100,"from":"USD","to":"JPY"})
             -> {"result":14820.0}
[step 2] FINAL:
   The current weather in Tokyo is 18°C (64°F) with a partly cloudy
   condition. For your travel budget, 100 USD is equivalent to 14,820 JPY.
   Enjoy your trip to the vibrant city of Tokyo!

The 2B called both tools in one turn (parallel tool calls), took both results, and composed a correct final answer. That's a complete agent, driven by a 1.9 GB model on your own hardware.

There's one place a small model trips: a single question it thinks it can answer from memory. Ask the 2B for Paris weather with the default tool_choice: "auto" and it sometimes skips the tool and guesses:

> "What's the weather in Paris? Use celsius."
  {"role":"assistant","content":"The weather in Paris is currently 15°C."}
  # no tool call — that number is invented

When you know a tool must run, don't hope — require it. Set tool_choice to force a call:

"tool_choice": "required"     # must call some tool
# or pin a specific one:
"tool_choice": {"type":"function","function":{"name":"get_weather"}}

Same 2B, same question, with tool_choice:"required" — now it calls the tool, and after you return the result it answers from real data:

get_weather({"city":"Paris","unit":"celsius"})
   -> {"temp_c":18,"condition":"partly cloudy"}
FINAL: The weather in Paris is partly cloudy with a temperature of 18°C.

Rule of thumb: leave tool_choice on auto for open agent loops (the 2B handles multi-tool tasks fine), and switch to required — or pin the function — on the steps where a tool is mandatory (a lookup, a write, a calculation). That single parameter is the difference between a small model that guesses and one that grounds every answer in your tools.

First, the honest part: there is no MCP switch on the inference server. The server is a pure OpenAI-compatible endpoint — it speaks the tools / tool_calls protocol and nothing more. MCP (the Model Context Protocol) lives on the client side: an MCP host connects to MCP servers that expose tools, and hands those tools to a language model. Because our tool-call format is exactly the format MCP hosts already emit, the server slots in as the model behind any MCP setup — you point the host's model endpoint at us; you don't configure MCP in the engine.

Two ways that looks in practice:

1. An MCP-aware client/framework. If your agent framework already speaks MCP (many do), just set its OpenAI base URL and key to the server and pick a model:

OPENAI_BASE_URL=http://<host>:8081/v1
OPENAI_API_KEY=<your-key>
OPENAI_MODEL=q35-2b

The framework discovers tools from its MCP servers, sends them in the tools array, and relays our tool_calls back to the MCP servers — no server-side change.

2. A tiny bridge, if you're wiring it yourself. An MCP client lists a server's tools; convert each to an OpenAI function schema, run the loop from earlier, and dispatch calls back over MCP:

# pseudocode — MCP tools in, tool_calls out, results back over MCP
mcp = MCPClient("stdio://your-mcp-server")        # connect the MCP server
tools = [to_openai_schema(t) for t in mcp.list_tools()]   # MCP -> OpenAI tools

def run_tool(name, args):                          # dispatch a call back to MCP
    return mcp.call_tool(name, args)

answer = run_agent(user_msg, tools, run_tool)      # the same loop, unchanged

The run_agent loop is identical to Step 2 — the only new code is translating tool definitions and routing calls to MCP. The model, the endpoint, and the tool-call protocol don't change.

• Agents are chatty. A single user turn can fan out into many model calls. Per-token, that's the workload you least want on a metered API — and the one where a fast, free local model compounds the most.
• Tools do the heavy lifting. In an agent, the model is a router and composer, not a knowledge base — exactly the job a 2B does well. The function fetches the fact; the model just needs to call it correctly and phrase the result.
• Your tools touch private systems. Agent tools hit your databases, your APIs, your files. Running the model locally keeps the whole loop — prompts, tool arguments, results — inside your network.
• Fast turns matter more here. Every tool round-trip adds a model call; the 2B's speed (≈45 tokens/sec on a Graviton node) keeps multi-step agents responsive.

Start on the 2B, add tool_choice where a call is mandatory, and step up to the 4B only if your tools or reasoning get genuinely harder — the same test-locally-first move from our prompt-optimization post.

Install the server (Linux, one line) — function calling is built into every model, no flag required:

curl -fsSL https://inference-server.searchblox.com/install | sudo bash

Then declare your tools, run the loop above against http://<host>:8081/v1/chat/completions, and you have a private agent — no per-call bill, no data leaving the box.

Every tool call and answer above was captured from q35-2b on a running SearchAI Inference Server, greedy decoding, August 2026, with stubbed tool backends. See the model catalog (every model does function calling) and the companion posts on vision extraction and small-model efficiency.