Structured Output with JSON Schemas

Constrained decoding, not polite requests

Asking a model "please reply in JSON" works most of the time, which is not good enough for a program. Local runtimes can do better with constrained decoding: at each step, the sampler masks out every token that would make the output invalid, so only tokens that keep the text on a valid path can be chosen. llama.cpp implements this with GBNF grammars, a grammar notation it can also generate from a JSON Schema (--json-schema or the json_schema field in requests). Ollama exposes it through the format field: "json" for any valid JSON, or a full JSON Schema object to fix field names and types. The guarantee is about shape, not truth: the model can still put a wrong value in a correctly typed field. So validate the result in code, and keep the schema description in the prompt too, because the model writes better values when it knows what each field means.

Schema from a Pydantic model

Requires pip install ollama pydantic and a running server.

from ollama import chat
from pydantic import BaseModel

class Course(BaseModel):
    name: str
    credits: int
    has_lab: bool

text = "Operating Systems is a 4-credit course with a weekly lab."
resp = chat(
    model="llama3.2",
    messages=[{"role": "user", "content": f"Extract the course details: {text}"}],
    format=Course.model_json_schema(),
    options={"temperature": 0},
)
course = Course.model_validate_json(resp.message.content)
print(course)

Leave room for the closing brace

If num_predict is too small, generation can stop mid-object and you get truncated JSON even with a schema. Set a generous token limit and always wrap parsing in error handling.

Quick check: What does constrained decoding with a JSON schema guarantee?

• That every value in the JSON is factually correct
• That the output's structure is valid for the schema, as long as generation completes
• That the model runs faster than unconstrained decoding in all cases
• That no tokens are sampled