Schema-First LLM Systems

The prompt is not the interface. The schema is.

The model returned a beautiful paragraph. Ops still emailed the wrong founder.

The ranking child finished. The parent run was green. The rationale read like something a recruiter might actually write: strong communication, thrives in fast-paced environments, good culture fit.

What it didn't contain was a score the orchestrator could gate on, a closed set of red flags, or a contract the rest of the pipeline could trust.

Someone had "improved" the prompt. Instead of returning the structured object the ranking step expected, the model returned prose. Downstream code parsed what it could, defaulted what it couldn't, and quietly queued a lead that looked fine until a human opened the inbox.

Valid English. Broken software.

I'd seen the same failure months earlier while building DocPilot. A single generation step produced documentation that read well and confidently described endpoints that never existed. Different product. Same bug.

The model wasn't hallucinating because it was "bad." It was doing exactly what we'd asked: generate text. The mistake was expecting free-form language to behave like an API.

That was when I stopped designing LLM systems around prompts.

Why everyone obsesses over prompts

Prompts feel like the product.

You change a sentence, the demo gets better, and it looks like progress. Schemas feel like plumbing. Validation feels like bureaucracy. So teams spend weeks polishing instructions and treat the response shape as an afterthought: "just ask it to return JSON."

That inversion is how you get systems that sound smart and break at the boundary between the model and your code.

Stripe does not return a thoughtful essay about a payment. Postgres does not narrate a row. Both return contracts your software can depend on. LLMs can do the same thing. Most products just never ask them to.

If the rest of your stack is typed and the LLM boundary is a string, you do not have an AI product. You have a chatbot taped to a pipeline.

Start with the object, not the prompt

Schema-first development flips the first question.

Instead of:

What prompt should I write?

Ask:

What object does my application need after this call?

In the outbound recruiting engine, the ranking step does not need a paragraph. It needs a match result the orchestrator can sort, gate, and log:

interface MatchResult {
  score: number;
  rationale: string;
  redFlags: string[];
}

That interface is the product decision. Score range, rationale length, red-flag cardinality: those are business rules dressed as types. The prompt exists to fill this object. If you write the prompt first, you are inventing an API response by chatting with the model until it roughly cooperates.

I encode the same shape in Zod, because TypeScript interfaces do not run at the boundary:

// src/trigger/outbound-rank.ts
import { z } from "zod";

export const matchResultSchema = z.object({
  score: z.number().min(0).max(100),
  rationale: z.string().max(500),
  redFlags: z.array(z.string()).max(5),
});

export type MatchResult = z.infer<typeof matchResultSchema>;

Now the question "did the model succeed?" has a mechanical answer. Parse failed? The child failed. Score missing? Invalid. redFlags came back as a string? Invalid. You do not need a human to decide whether the output "looks okay."

The prompt comes after, and it becomes narrow on purpose: fill MatchResult given this resume and this job description. No preamble. No markdown fences. No helpful closing sentence.

The architecture that actually ships

Every production LLM step I trust looks like this:

Schema defines what success looks like. Prompt is how you ask. The model is a worker. Validation is the gate. Business logic never sees raw text.

Compare that to the shape most demos encourage:

Once you draw it this way, a lot of AI architecture arguments get quieter. Agents vs workflows, two-step pipelines, fan-out orchestration, graceful degradation: they all assume you can tell whether a step produced a usable object. Without a schema, those posts are aspirational. With one, they are implementable.

Prompts and versions are implementation details

In a normal backend, the HTTP handler is not your public API. The response type is.

Schema-first LLM systems steal that discipline. The schema is the interface between "AI stuff" and the rest of the product. The prompt is an implementation detail behind that interface, the same way SQL is an implementation detail behind a repository method. That has practical consequences: you can swap models without rewriting the orchestrator, rewrite the prompt without touching business gates, and reason about what changed by reading a diff instead of a paragraph.

Prompt v27 is meaningless to ops, to tests, and to the next engineer who joins. Schema v2 means something: redFlags became required, or score moved from 0–10 to 0–100, or you added rankingMode. Those are migrations, and I treat them like migrations, versioned explicitly when stored runs need to stay readable:

const rankedApplicantEventSchema = z.discriminatedUnion("schemaVersion", [
  z.object({
    schemaVersion: z.literal(1),
    score: z.number(),
    reason: z.string(),
  }),
  z.object({
    schemaVersion: z.literal(2),
    score: z.number().min(0).max(100),
    rationale: z.string().max(500),
    redFlags: z.array(z.string()).max(5),
    rankingMode: z.enum(["job", "employees"]),
  }),
]);

In the outbound system, job-mode ranking and employee-mode ranking share one matchResultSchema with different prompts and different child tasks. That only works because the orchestrator depends on the schema, not on prompt identity. I keep the schema in source control next to the task that uses it. Prompt text can live in the same file or a template. What I refuse to do is treat prompt edits as the source of truth for "what this step returns."

DocPilot: structured stages, not free-form magic

DocPilot looks like one AI feature from the outside: connect a repo, get API docs. Under the hood it is a pipeline where each stage returns a contract, not a blob of markdown.

Classification does not "think about the repo." It returns candidates:

export const routeCandidateSchema = z.object({
  path: z.string(),
  confidence: z.enum(["high", "medium", "low"]),
  reason: z.string().max(200),
});

export const classificationResultSchema = z.object({
  candidates: z.array(routeCandidateSchema),
  frameworkHint: z.string().optional(),
});

Extraction is mostly not an LLM step at all. Code reads the file and either finds a handler or vetoes the candidate. Generation receives a bounded context object and returns a RouteDoc, not a README the user has to clean up:

export const routeDocSchema = z.object({
  method: z.enum(["GET", "POST", "PUT", "PATCH", "DELETE"]),
  path: z.string(),
  summary: z.string().max(240),
  params: z.array(
    z.object({
      name: z.string(),
      type: z.string(),
      required: z.boolean(),
    })
  ),
  responseShape: z.string(),
});

Each stage can fail independently. Bad classification is a bad candidate list. Bad extraction is a skip. Bad generation is a schema error or a retry. That decision log is what one-shot prompts cannot give you.

That is the same thesis as the two-step pipeline post, pushed one level deeper: the split only works if each LLM stage has a schema sharp enough to fail loudly. Free-form classification text cannot feed deterministic extraction. Free-form "docs" cannot land in a database without another guessing layer.

When DocPilot breaks, I do not ask "was the AI weird today?" I ask which contract broke.

Failures become observable

Without a schema, failure language stays mushy.

"The AI did something weird."

With a schema, failures look like software failures:

• missing field
• wrong enum
• invalid array shape
• number out of range
• validation error after a "successful" provider response

const parsed = matchResultSchema.safeParse(JSON.parse(response.text));

if (!parsed.success) {
  logger.warn("rank_schema_invalid", {
    applicantId,
    issues: parsed.error.issues.map((i) => ({
      path: i.path.join("."),
      code: i.code,
      message: i.message,
    })),
  });
  throw new RankingSchemaError(parsed.error);
}

That log line is debuggable at 2 AM. A transcript of vibes is not.

Schemas also draw a clean line between two different bugs that teams constantly conflate:

I have written at length about the second class in graceful degradation: valid JSON that still should not queue a lead. Schemas do not replace those gates. They make the first class visible so you stop blaming "AI randomness" for a parse error, and stop treating a parse success as proof of quality.

A green provider response is not a green product step. Validation is.

Schemas improve prompts; they do not cage them

People worry that structured output makes the model dumber. In practice, the opposite happens.

When the contract is explicit, the prompt stops wasting tokens on formatting instructions and starts spending them on judgment. You are not begging for JSON. You are specifying what good judgment looks like inside a known shape.

Compare these asks:

Prompt-first: "Rank this candidate and explain your thinking. Return JSON if possible."

Schema-first: "Return score (0–100), a rationale under 500 characters grounded in resume evidence, and up to five concrete redFlags. Prefer precision. If the job description is generic, lower the score and say so in redFlags."

The second prompt is freer where it matters. The model can still reason. It just has to land the plane on a runway you already poured.

I also stopped stuffing examples of JSON into prompts once the schema was enforced at the API layer. Few-shot format examples fight the schema when they drift. Few-shot judgment examples help: a thin job description that should score low, a stack mismatch that belongs in redFlags, a strong match that still notes a location risk. Teach the decision policy. Let the schema teach the shape.

Structured output modes from providers help, but I still validate. Provider schemas reduce drift; they do not absolve you from treating the boundary as hostile. Models occasionally omit optional fields you thought were obvious, stuff enums with near-miss strings, or wrap arrays as a single newline-separated string when the prompt gets chatty. Your Zod parse is the last honest checkpoint before business logic:

import { zodTextFormat } from "openai/helpers/zod";

export async function rankAgainstJobDescription(input: {
  resume: string;
  jobTitle: string;
  jobDescription: string;
}): Promise<MatchResult> {
  const response = await openai.responses.parse({
    model: "gpt-5.4",
    input: buildRankingPrompt(input),
    text: {
      format: zodTextFormat(matchResultSchema, "match_result"),
    },
  });

  // Provider structured output is necessary, not sufficient.
  return matchResultSchema.parse(response.output_parsed);
}

If that parse throws, the child task retries or fails in isolation. The orchestrator never sees a half-object and invents defaults. That is the whole point of treating the schema as the interface.

Testing becomes possible

You cannot usefully regression-test "whatever the model said." You can regression-test objects.

Schemas unlock the testing loop senior engineers already expect from every other integration:

Snapshot fixtures. Save validated outputs for a fixed resume × job pair. When the prompt changes, diff the object, not the vibe.

Replay. Re-run a stored provider payload through safeParse and business gates without spending tokens. This is how I debug a bad Tuesday without re-paying for the original fan-out.

Mocks. Child tasks become mockable. Return a MatchResult. The orchestrator does not need a live model to test skip logic, confidence gates, or degraded ranking modes.

Deterministic pipelines. Given the same validated objects, aggregation should be identical. If pickTopMatch flips, that is your bug, not the model's.

describe("pickTopMatch", () => {
  it("skips when the winner has too many red flags", () => {
    const batch = mockBatch([
      {
        applicantId: "a1",
        output: {
          score: 88,
          rationale: "Keyword overlap on React",
          redFlags: [
            "wrong seniority",
            "no backend evidence",
            "location mismatch",
          ],
        },
      },
      {
        applicantId: "a2",
        output: {
          score: 71,
          rationale: "Partial stack match",
          redFlags: ["thin resume"],
        },
      },
    ]);

    expect(pickTopMatch(applicants, batch)).toBeNull();
  });

  it("rejects provider payloads that violate the contract", () => {
    const raw = {
      score: "high",
      rationale: "Looks good",
      redFlags: "seniority",
    };

    expect(matchResultSchema.safeParse(raw).success).toBe(false);
  });
});

The second test looks trivial. It is the one that catches silent prompt regressions: someone asks for a "richer" response, the model starts returning strings where you needed numbers, and CI fails before staging emails anyone.

This is also why batch fan-out works in production: children return one schema, the parent aggregates, gates decide. Without the schema, your tests are either live-model flaky or theater.

I still do occasional live evaluations for prompt quality. Those are product evaluations on fixed fixtures with human review. They are not a substitute for contract tests, and they should not be the only thing standing between a prompt edit and a cron schedule.

When you should not force a schema

If another function will consume this output, give it a schema. If a human will read it and move on, let it be text.

Schemas are for software boundaries. Not every LLM call is a software boundary. Skip rigid schemas when the output is the product for a human: brainstorming, drafting prose, summarizing a document for reading, open-ended chat. In those cases, free-form text is the interface. Adding Zod around a blog outline is ceremony.

DocPilot's internal stages are schema-first. A "rewrite this paragraph" helper does not need to be. The outbound engine's ranking children are schema-first. A founder-facing email draft can be text that a template wraps, as long as the decision to send was based on structured judgment upstream.

Mixing the two without noticing is how teams either over-constrain creative work or under-constrain automation.

The contract is the product

Most AI systems I review still begin in the wrong place. They open a prompt file and negotiate with a model until the demo looks good. Then they discover, in staging, that nobody can test the boundary, version the behavior, or explain a failure without reading transcripts.

Start with the object your application needs. Write the schema. Put validation in front of business logic. Let the prompt become the worker instructions behind that interface. Use pipelines, not vibes: classify, extract, generate; fan out; degrade; skip. Every one of those patterns assumes a contract you can point to.

We do not design APIs before deciding the response shape.

We should not design LLM systems that way either.

Related: Why Most AI Products Don't Need Agents · Two-Step LLM Pipelines · Orchestrating Parallel LLM Workloads · Graceful Degradation in Cron Pipelines · DocPilot case study