Gen AI

Structured Outputs

Structured outputs force a model to return data in an exact, predictable shape a program can trust, instead of prose you have to parse and hope is right.

Asking a model to reply in JSON only requests a valid shape; restricting which tokens it can generate at each step guarantees the order record always parses.

Why Does This Exist?

Say you're building a system that reads customer emails and turns them into orders. A real one looks like this: "Hey, could you get me two of the standing desk converters by next Friday if possible? Thanks — Dana." A person reads that in two seconds. A downstream order system can't — it wants a record with an exact shape: customer_name a string, item a string, quantity an integer, requested_delivery_date a date. Somebody has to bridge the gap between a paragraph and that shape, and for years that somebody was a pile of regex and keyword rules that broke every time a customer phrased the request slightly differently.

Large language models are good at the bridging part — read the email, figure out what's being asked. The trouble shows up one step later, getting the answer back out in a form the order system can consume. Asking the model to "reply in JSON" gets you something that looks right most of the time, and that's exactly the problem. Most of the time isn't a guarantee, and a pipeline running unattended needs one.

Think of It Like This

A printed intake form versus a blank sheet of paper

Handing someone a blank sheet and asking them to write down an order gets you a letter: full sentences, maybe a comment about the weather, the quantity written as "a couple" instead of the numeral 2. Handing them a printed form with four boxes — Name, Item, Quantity, Delivery date — and a line just wide enough for one word each changes what they're physically able to write. The box for quantity is too narrow for a sentence. It constrains the output through the paper's shape, not a polite request.

Prompting a model to "please reply in JSON" is the blank sheet, with better handwriting. Schema-constrained decoding is the printed form: the shape of each field limits what can go into it before a single character gets written.

How It Actually Works

The problem with parsing free text

Ask a model to reply with the order as JSON and most of the time you'll get something close to {"customer_name": "Dana", "item": "standing desk converter", "quantity": 2, "requested_delivery_date": "next Friday"}. Fine, until it isn't. The model might wrap that object in a markdown code fence, because that's how it's seen JSON formatted thousands of times in training. It might add a caveat sentence before or after it, because that reads as helpful. It might name the field deliveryDate instead of requested_delivery_date, since both are reasonable English for the same idea and nothing forced it to pick the one your code expects. And it might hand back quantity as the string "two" instead of the integer 2, because the email said "a couple" and the model rendered that faithfully instead of numerically.

Any one of those breaks a system that runs json.loads() on the raw response and reaches into record["requested_delivery_date"]. The parser doesn't know the model was trying to be helpful. It just throws — or worse, it doesn't, and a string quietly sits where an integer belongs three steps further down the pipeline.

Schema-constrained decoding

Asking nicely and constraining the decoding are doing different jobs, even though both requests end with the word "JSON." Asking nicely is a prompting trick: you describe the shape you want in English and hope the model complies, the same way you'd hope a person filled out that intake form correctly if you only described it out loud instead of handing them the form.

Schema-constrained decoding works underneath the prompt, at the token level. You give the system an actual schema — a JSON Schema document, a Pydantic model, a grammar — and at every step of generation, the decoder checks which tokens are even legal next, given what's been generated so far and what the schema allows. A token that would open a fifth key when the schema defines only four is never a candidate. A token that would place a letter where the schema demands a digit is never a candidate. The model doesn't choose to comply; it's mechanically restricted to a path through token-space that stays valid the whole way, so the output ends up syntactically correct by construction, not by hope.

What structured outputs do not guarantee

Constrained decoding fixes exactly one problem: shape. It says nothing about content. Run the order-extraction example through a fully schema-constrained setup and you'll always get an object with customer_name, item, quantity, and requested_delivery_date, each with the right type — that part is now guaranteed. Whether quantity holds the 2 the customer actually asked for is a separate question. If the email never states a number, the model still has to put something in that integer field: a sensible default, an empty placeholder, or — if nothing stops it — a plausible-looking number it invented because the schema demanded a number and left no room for "unclear."

That's the split worth holding onto: syntactic validity means the JSON is well-formed and matches the shape; semantic correctness means the values inside it are true. Structured outputs solve the first completely and do nothing for the second.

Show Me the Code

Even after decoding guarantees valid JSON, a careful downstream consumer still checks the parsed dict against its own expectations before trusting it:

1from typing import Any2
3REQUIRED: dict[str, type] = {"customer_name": str, "item": str, "quantity": int, "requested_delivery_date": str}4
5
6def validate_order(record: dict[str, Any]) -> list[str]:7    """Return problems found; an empty list means the record is usable."""8    errors: list[str] = []9    for field, kind in REQUIRED.items():10        if field not in record:11            errors.append(f"missing field: {field}")12        elif not isinstance(record[field], kind):13            errors.append(f"{field} should be {kind.__name__}")14    return errors15
16
17order = {"customer_name": "Dana", "item": "desk converter", "quantity": "2", "requested_delivery_date": "2025-08-15"}18print(validate_order(order))  # -> ['quantity should be int']

Watch Out For

Treating schema-valid as fact-checked

A response that satisfies the schema perfectly can still be wrong. The quantity field being an integer says nothing about whether it's the right integer — a model under pressure to fill a required field can produce a confident, well-typed guess instead of admitting the email didn't say. Schema validation is a gate for shape, not a substitute for checking the actual values against the source, especially on fields nothing in the input ever specified.

Parsing 'JSON-ish' text with regex instead of a real validator

When a model occasionally wraps its JSON in a code fence or adds a trailing sentence, the tempting fix is a regex that strips the fence and grabs "the part that looks like an object." That's fragile in the same way the original free-text parsing was — it just moved the guesswork down one layer. Use real schema-constrained decoding so the fence can never appear, or run every parsed result through an explicit validator, and treat a failure as a signal to retry, not a warning to ignore.

The Quick Version

Asking a model to "reply in JSON" is a request, not a guarantee — markdown fences, stray prose, wrong key names, and type mismatches can all still show up.
Schema-constrained decoding restricts which tokens the model is even allowed to generate at each step, so the output is valid JSON by construction, not by hope.
The schema comes from something concrete — a JSON Schema document, a Pydantic model, a grammar — not from a sentence in the prompt.
Syntactic validity and semantic correctness are separate properties. Structured outputs solve the first; they do nothing for the second.
A downstream consumer should still validate field-by-field, because "parses" and "correct" are not the same claim.