Tool Use in LLMs: How Function Calling Actually Works Under the Hood

From Text Completion to Tool Invocation

Large language models were originally designed to predict the next token in a sequence. Yet in 2025-2026, tool use has become a first-class capability across GPT-4o, Claude, Gemini, and open-source models like Llama 3.3. Understanding how function calling works beneath the surface is critical for anyone building AI-powered applications.

How Tool Definitions Reach the Model

When you define tools in an API call, the provider serializes your function schemas into the model's context. For example, with OpenAI's API:

{
  "tools": [{
    "type": "function",
    "function": {
      "name": "get_weather",
      "description": "Get current weather for a location",
      "parameters": {
        "type": "object",
        "properties": {
          "location": { "type": "string" },
          "unit": { "type": "string", "enum": ["celsius", "fahrenheit"] }
        },
        "required": ["location"]
      }
    }
  }]
}

This JSON schema gets converted into a structured prompt segment that the model sees as part of its system context. The model has been fine-tuned (via RLHF and supervised fine-tuning on tool-use datasets) to recognize when a user query requires tool invocation and to emit a structured JSON response matching the schema.

The Token-Level Mechanics

Under the hood, function calling works through constrained decoding:

1. Intent recognition: The model determines that the user's request maps to one of the available tools rather than a direct text answer
2. Schema-guided generation: The model generates a JSON object with the function name and arguments, constrained by the provided schema
3. Stop sequence: The model emits a special stop reason (e.g., tool_use or function_call) instead of the normal end-of-turn token
4. Execution loop: The calling application executes the function and injects the result back into the conversation for the model to synthesize

Parallel and Sequential Tool Calls

Modern LLMs support parallel tool calling, where the model requests multiple function invocations in a single turn:

# Claude's tool_use response may contain multiple tool blocks
for block in response.content:
    if block.type == "tool_use":
        result = execute_tool(block.name, block.input)
        tool_results.append({
            "type": "tool_result",
            "tool_use_id": block.id,
            "content": result
        })

Sequential tool calls happen when the model needs the output of one tool to determine the input of the next. The model handles this by making a single tool call, receiving the result, then deciding whether to call another tool or respond to the user.

Reliability Challenges

Tool use introduces several failure modes:

• Schema hallucination: The model invents parameters not in the schema or passes invalid types
• Tool selection errors: The model picks the wrong tool for the task
• Argument extraction failures: Ambiguous user input leads to incorrect parameter values
• Infinite loops: The model repeatedly calls the same tool without making progress

Production Hardening Patterns

Teams shipping tool-use systems in production adopt several patterns:

• Strict mode: OpenAI and Anthropic both support strict schema validation that guarantees the output conforms to the JSON schema
• Retry with feedback: When a tool call fails, inject the error message back into the conversation so the model can self-correct
• Tool call limits: Cap the number of tool calls per turn to prevent runaway loops
• Fallback responses: If tool execution fails after retries, have the model respond gracefully without the tool result

The Bigger Picture

Tool use transforms LLMs from knowledge retrieval systems into action-taking agents. As tool ecosystems mature through standards like Anthropic's Model Context Protocol (MCP), the boundary between "chatbot" and "software agent" continues to blur.

Sources: Anthropic Tool Use Documentation | OpenAI Function Calling Guide | Gorilla LLM Research