ANKIT YADAV

LLM Engineering · 19 Nov 2026 · 6 min read

Understanding LLM Tool Calling

How an application asks a model for a structured tool call, checks the arguments, runs the tool itself, and sends the result back to the model.

Introduction

Tool calling is the point where a language model stops being a text box and becomes a client of your application. The model does not query your database. It proposes a function name and a JSON argument object. Your process validates that object, runs the function, and gives the result back so the model can write the sentence a person actually reads.

That split is the whole design. If you let the model "call" a tool by emitting shell text, you have skipped the only step that makes the feature safe to put next to customer data. This note stays on the application side of the line: schemas, validation, timeouts, and what to do when the arguments are almost right.

Provider field names move. The shapes below are deliberately generic. The current request format for a given vendor belongs in that vendor's docs, not in a blog post that will go stale. OpenAI's function-calling guide and Google's Gemini function-calling guide are the two references I would open while implementing a specific SDK.

Problem

A support or operations workflow often needs one fact the model cannot know: a claim status, a policy date, an offer that is still valid. Prompting the model to "remember" that fact produces a fluent answer and a wrong number. Asking the user to paste the record into the chat pushes the integration problem onto a person who does not have the record.

The naive fix is to paste the entire database row into the prompt on every turn. That works for a demo and fails as soon as the row is large, private, or one of thousands. Tool calling exists so the model can request the one lookup it needs, and so the application can refuse any request that is not on the list.

A second problem shows up the moment the model is allowed to ask. Models invent argument names, wrap JSON in prose, and call a tool that was described in an old prompt and removed last week. The application has to treat every tool call as untrusted input, the same way it treats a form post.

Technical explanation

A tool definition has three parts the model is allowed to see: a name, a short description of when to use it, and a JSON schema for the arguments. The implementation, the credentials, and the query live only on the server. The model never receives a connection string just because it might need to look something up.

A turn then has two model calls, or one if no tool is required. The first call includes the user message and the tool list. The response is either ordinary text or a tool call. A tool call is a name plus an arguments payload. You parse the payload, check it against the schema you advertised, run the function, and append a tool result message. The second call sees the original user message, the tool call, and the result, and it writes the reply.

Several details fall out of that sequence.

The schema is a contract, not a hint. If the model sends a string where the schema says integer, you do not coerce it quietly and hope. You reject the call or you ask the model once, with the validation error in the tool result, to try again. Silent coercion is how a status code and a phone number trade places.

The tool runs with the caller's authority, not the model's. If the signed-in user cannot read a policy, the tool returns a permission error even when the model asked politely. The model is not an admin.

Side effects need an idempotency key. A "send the offer" tool that posts a message should accept a key the application generated, so a retried model call does not send the offer twice. Read-only tools can be looser. Anything that writes cannot.

Timeouts are part of the tool, not an afterthought around the HTTP client you happened to use. A tool that hangs will hold the chat turn open until the user's client gives up, and the fallback logic you add later will not know whether the first attempt is still running.

Architecture

Keep a registry. Each entry is the public schema and a function. The router that talks to the model only sees the schema. The worker that executes a call looks up the name in the registry and refuses anything else. That single allowlist is what stops a prompt injection hidden in a user message from reaching a function you did not mean to expose.

Diagram

The registry, the validator, and the timeout sit in your process. The model vendor sees the schema and the tool result you chose to send, which should already be the minimum the reply needs. Do not echo an entire internal record back into the prompt if three fields answer the question.

Implementation approach

Start with one read-only tool. Give it a boring name, a schema with required fields, and a function that returns a small object or a clear not-found. Log the tool name, a hash or id of the subject, the latency, and the error class. Do not log the full prompt if the prompt can contain a policy number you would not put in an analytics tool.

Add a second model call only after the first tool is dull and correct. Then add the repair path: one retry when validation fails, with the validator's message as the tool result, and a hard stop after that. Unlimited repair loops are a bill and a stuck chat.

When you add a write tool, generate the idempotency key in your code before the model is involved. Pass it in a field the model is not supposed to invent. If the model changes it, reject the call.

Provider SDKs differ on whether arguments arrive as an object or as a JSON string, and on what the tool-result role is called. Parse at the edge of your client wrapper and hand the rest of the application a single ToolCall type. The official guides linked from a provider's docs are the place to check the wire format the week you ship. Pinning a guessed JSON envelope in an article would be obsolete on purpose.

Code examples

The snippet is provider-agnostic TypeScript. complete is a function you own. It hides whichever SDK you are calling this month.

tool-turn.ts
type ToolCall = { name: string; arguments: unknown };
 
type Tool<T> = {
  name: string;
  description: string;
  parameters: Record<string, unknown>;
  parse: (input: unknown) => T;
  run: (args: T, signal: AbortSignal) => Promise<unknown>;
};
 
async function handleTurn(userText: string, tools: Tool<unknown>[]) {
  const first = await complete({
    messages: [{ role: "user", content: userText }],
    tools: tools.map(({ name, description, parameters }) => ({
      name,
      description,
      parameters,
    })),
  });
 
  if (!first.toolCall) return first.text ?? "";
 
  const tool = tools.find((item) => item.name === first.toolCall?.name);
  if (!tool) return "I can't complete that request.";
 
  const controller = new AbortController();
  const timer = setTimeout(() => controller.abort(), 8000);
  try {
    const args = tool.parse(first.toolCall.arguments);
    const result = await tool.run(args, controller.signal);
    const second = await complete({
      messages: [
        { role: "user", content: userText },
        { role: "assistant", toolCall: first.toolCall },
        { role: "tool", name: tool.name, content: JSON.stringify(result) },
      ],
    });
    return second.text ?? "";
  } finally {
    clearTimeout(timer);
  }
}

parse is where Zod, or any other schema library, belongs. If it throws, catch it and send one tool result that says the arguments were invalid. Do not include a stack trace. The model does not need your file paths, and the user does not either.

A Python service would look the same: a dict of callables, jsonschema or Pydantic on the arguments, and an HTTP timeout on the outbound call. The language is not the design.

Lessons learned

The useful version of tool calling is smaller than the demo version. Three tools with strict schemas beat fifteen tools with descriptions that overlap. Overlapping descriptions are why a model calls the wrong one and then apologizes.

Validation is the product. The sentence the model writes afterward is formatting. If the tool result is wrong, a well-written sentence makes the bug harder to see.

Keep a human exit for anything that changes money, coverage, or a claim. A tool that files a change should be callable only when a person has already confirmed, or it should create a draft a person approves. That is an application rule. The model will not invent it for you.

Conclusion

Tool calling works when the application owns the function, the schema, the timeout, and the permission check. The model owns a proposal. Treat the proposal like any other untrusted request, return the smallest result that answers the turn, and keep a single retry between a bad argument list and a plain failure. The SDK you wrap this week is a detail. The allowlist is the system.

The fallback note on this site covers what to do when the model call itself fails. The RAG note covers the case where the "tool" is really a document search and the answer has to stay inside the passages you retrieved.

Related articles

LLM Engineering · 19 Nov 2026 · 6 min

Multi-LLM Fallback Architecture

A practical way to call a second model when the first one times out or returns an unusable response, without repeating the same side effect.