Reliable Structured Output from LLMs: Schemas, Tool Calling, and Repair Loops

Integrating large language models into software pipelines requires deterministic outputs. While LLMs excel at processing unstructured natural language, downstream application logic expects structured data like typed JSON objects, strictly typed database records, or valid function calls.

Relying purely on prompt engineering (for example, telling the model “Return JSON only with keys x, y, z”) is notoriously fragile in production. Models can output markdown code blocks ( ``json `), insert conversational intros or introspective preamble, omit required fields, hallucinate extra keys, or emit malformed syntax on long strings.

Building production-ready structured LLM outputs requires a multi-layered approach combining provider-level constrained decoding, client-side schema validation, backoff strategies, and automated repair loops.

1. Why Free-Form Prompting Fails

When asking a model for JSON without schema enforcement, several failure modes emerge:

1. Syntax Corruption: Truncated outputs, unescaped quotes within text strings, or trailing commas cause standard JSON parsers (json.loads or JSON.parse) to throw syntax errors.
2. Formatting Wrapping: The model wraps its output in markdown code fences ( json ... ) or prepends friendly text like “Here is the extracted JSON:”, breaking strict parsers.
3. Schema Drift: The model omits nested keys, alters property casing (switching from camelCase to snake_case), or changes primitive types (emitting "123" as a string instead of an integer 123).
4. Hallucinated Attributes: Unexpected fields appear in the output, polluting application memory or violating downstream database schemas.

Prompting can mitigate these issues to some degree, but non-zero error rates persist. In automated pipelines processing thousands of requests, even a 2% failure rate creates operational noise and requires manual intervention.

2. Provider-Native Schema Enforcement

Modern LLM provider APIs offer constrained decoding (also called structured outputs or grammar-guided decoding). Instead of post-filtering text, the provider’s inference engine masks the logit probabilities at each token generation step. Tokens that would violate the specified JSON schema receive a probability of zero, mathematically guaranteeing syntactically valid JSON output.

OpenAI Structured Outputs

OpenAI’s Responses API takes a Pydantic model directly via text_format and hands back a parsed object, with no manual JSON decoding step.

from openai import OpenAI
from pydantic import BaseModel, Field

client = OpenAI()

class UserExtraction(BaseModel):
    name: str = Field(description="Full name of the user")
    age: int = Field(description="Age in years")
    skills: list[str] = Field(description="List of technical skills")

response = client.responses.parse(
    model="gpt-5.6",
    input=[
        {"role": "system", "content": "Extract user details from text."},
        {"role": "user", "content": "Alex is a 32-year-old backend engineer specializing in Go and Python."}
    ],
    text_format=UserExtraction,
)

user_data: UserExtraction = response.output_parsed
print(user_data.name)  # Output: Alex

Under the hood this still forces every field in the JSON schema to be required and sets additionalProperties: false, which is what eliminates schema drift.

Gemini API Structured Outputs

Google’s Gemini API accepts a JSON schema through the Interactions API’s response_format, letting a Python Pydantic model or a raw JSON schema dictate the response structure.

from google import genai
from pydantic import BaseModel, Field

client = genai.Client()

class InvoiceItem(BaseModel):
    description: str
    amount: float

class Invoice(BaseModel):
    vendor: str
    items: list[InvoiceItem]
    total: float

interaction = client.interactions.create(
    model="gemini-3.6-flash",
    input="Extract invoice: Acme Corp provided 2 servers for $500 each and setup for $150.",
    response_format={
        "type": "text",
        "mime_type": "application/json",
        "schema": Invoice.model_json_schema()
    }
)

invoice = Invoice.model_validate_json(interaction.output_text)
print(invoice.total)  # Output: 1150.0

Anthropic Tool Calling

Anthropic handles structured extraction via tool calling (function calling). By defining a tool with a strict input_schema and setting tool_choice={"type": "tool", "name": "your_tool"}, Claude is compelled to invoke that tool with matching parameters.

import Anthropic from "@anthropic-ai/sdk";

const anthropic = new Anthropic();

const response = await anthropic.messages.create({
  model: "claude-sonnet-5",
  max_tokens: 1024,
  tools: [
    {
      name: "record_user_sentiment",
      description: "Record customer sentiment analysis",
      input_schema: {
        type: "object",
        properties: {
          sentiment: { type: "string", enum: ["positive", "neutral", "negative"] },
          confidence: { type: "number" },
          summary: { type: "string" }
        },
        required: ["sentiment", "confidence", "summary"]
      }
    }
  ],
  tool_choice: { type: "tool", name: "record_user_sentiment" },
  messages: [{ role: "user", content: "The service was fast and helpful!" }]
});

// The output parameters match the tool schema precisely
const toolUse = response.content.find((block) => block.type === "tool_use");
console.log(toolUse.input);

3. Client-Side Validation with Pydantic and Zod

Even when using constrained decoding at the API layer, client-side validation remains mandatory for three primary reasons:

• Token Truncation: If a response hits max_tokens (finish_reason == "length"), the JSON stream cuts off abruptly, leaving an incomplete syntax payload.
• Semantic Limits: JSON schemas check basic types (string, number), but domain rules (such as age >= 18 or email regex matches) require client evaluation.
• Provider Fallbacks: Applications running across multiple models or local LLMs (via Ollama or vLLM) need a uniform client validation layer regardless of backend provider support.

TypeScript Validation with Zod

In Node.js and browser environments, Zod provides runtime validation and static type inference.

import { z } from "zod";

export const TaskSchema = z.object({
  id: z.string().uuid(),
  title: z.string().min(3),
  priority: z.enum(["low", "medium", "high"]),
  tags: z.array(z.string()).default([])
});

export type Task = z.infer<typeof TaskSchema>;

export function parseLLMResponse(rawText: string): Task {
  // Strip potential markdown code block artifacts
  const cleanedText = rawText.replace(/^```json\s*/i, "").replace(/\s*```$/, "").trim();
  const rawJson = JSON.parse(cleanedText);
  
  // Validate schema and throw ZodError on violation
  return TaskSchema.parse(rawJson);
}

Using z.safeParse() allows software to handle validation failures gracefully without unhandled promise rejections or process crashes.

4. Building Resilient Repair and Retry Loops

When client validation catches a malformed response or semantic error, failing immediately drops the task. Instead, feed the invalid output and exact error diagnostics back to the LLM in a self-correction loop.

[User Request] 
      |
      v
[LLM API Call] 
      |
      v
[Client Validation (Pydantic / Zod)]
      |
      +---> Valid? ---> Return Structured Object
      |
   Invalid
      |
      v
[Append Error & Original Output to Prompt]
      |
      v
[Retry Loop (Max Retries: 2-3)]

Self-Correction Loop Implementation

Below is a Python pattern implementing schema validation, automated repair, and exponential backoff for transient rate limits.

import json
import time
from typing import Type, TypeVar
from pydantic import BaseModel, ValidationError
from openai import OpenAI

T = TypeVar("T", bound=BaseModel)

def extract_with_repair(
    client: OpenAI,
    model: str,
    prompt: str,
    schema: Type[T],
    max_retries: int = 3
) -> T:
    messages = [
        {"role": "system", "content": f"Extract structured data adhering strictly to this JSON schema: {schema.model_json_schema()}"},
        {"role": "user", "content": prompt}
    ]

    for attempt in range(1, max_retries + 1):
        try:
            response = client.chat.completions.create(
                model=model,
                messages=messages,
                temperature=0.1
            )
            
            raw_content = response.choices[0].message.content or ""
            
            # Clean markdown code fences if present
            cleaned_content = raw_content.strip()
            if cleaned_content.startswith("```"):
                lines = cleaned_content.splitlines()
                cleaned_content = "\n".join(lines[1:-1])

            # Validate against Pydantic schema
            data = schema.model_validate_json(cleaned_content)
            return data

        except (json.JSONDecodeError, ValidationError) as err:
            if attempt == max_retries:
                raise RuntimeError(f"Failed to extract valid schema after {max_retries} attempts: {err}")
            
            # Feed errors back into context for repair
            messages.append({"role": "assistant", "content": raw_content})
            messages.append({
                "role": "user",
                "content": (
                    f"Your output failed validation with the following error:\n{err}\n"
                    "Please fix the format and return ONLY the valid JSON object."
                )
            })
            
            # Brief backoff pause before retrying
            time.sleep(attempt * 0.5)

    raise RuntimeError("Extraction unexpected exit")

This repair loop routinely resolves edge-case errors on the second attempt, bringing overall extraction reliability near 100%.

5. Streaming vs. Structured Output

When low Time-To-First-Token (TTFT) is critical for user interfaces, streaming model responses introduces complexity for structured JSON parsing.

Standard JSON parsers cannot evaluate incomplete JSON strings like {"user": {"name": "Al.

To bridge streaming and structured objects, choose between two strategies:

1. Buffered Completion: Stream the response to the client or log stream, but buffer the full string in memory before running JSON.parse and schema validation. This is simple, safe, and works with all validation libraries.
2. Incremental Parsing: Use specialized partial JSON parsers (such as jsonrepair or custom streaming parsers) to reconstruct partial objects on every chunk. This allows UIs to render fields as they complete, but requires handling partial state changes gracefully.

Architectural Recommendations

For high-confidence production deployments:

• Prefer provider-native constrained decoding (strict: true or response_format) whenever supported by your chosen model family.
• Always validate on the client using Pydantic or Zod to guard against token truncation or semantic boundaries.
• Implement a 2-step repair loop that passes exact error tracebacks back to the model upon validation failure.
• Use low temperature settings (0.0 to 0.2) when generating structured data to minimize hallucinated keys and syntax drift.