Voice Mock Interviews as a Pipeline, Not an Agent

I didn't build an AI interviewer.

I built a pipeline that happens to contain a voice conversation.

That distinction ended up shaping almost every architectural decision in the product.

The problem

Job seekers do not lack interview advice. They lack a closed loop they can run before the real thing: pick a role, get a credible question set, practice out loud, and get scored feedback that is specific enough to act on.

The tempting architecture is an "AI interviewer agent." Give the model a job description, a microphone, and tool access, and let it improvise. That demo looks alive for five minutes. Then the model invents questions you never asked for, drifts off the seniority you set, and writes feedback that does not fit the UI.

I wanted a practice loop that behaves the same way twice.

The core loop

configure -> generate script -> voice session -> score transcript -> store feedback

Three stages. Two structured model calls with schemas. One hosted realtime voice session in the middle.

1. The user picks role, seniority, interview type, tech tags, and question count.
2. A structured generation call produces the question list plus metadata.
3. The product stores that interview and injects the questions into a fixed voice assistant.
4. The user speaks. The client owns call state and accumulates the transcript.
5. On hang-up, a second structured generation call scores the transcript into a fixed rubric and replaces any prior feedback for that session.

Software -> LLM -> software -> voice runtime -> software -> LLM -> software.

That is the whole product, if you squint.

What shipped

• Custom interview creation with client-side validation before the server spends tokens
• Role-specific question scripts generated as structured objects, not free-form markdown
• Realtime voice practice through a managed voice platform (STT + TTS + conversational model)
• Live transcript during the call
• Post-call feedback with a total score, five fixed categories, strengths, improvements, and a short assessment
• History dashboard to retake, open feedback, or delete a session and its feedback together
• Managed auth so identity was not a custom side quest

Architecture

Engineering highlights

The script is the product object

The interview document is not a chat thread. It is a typed record: role, level, type, tech stack, questions, description, duration, tags, difficulty, user id, timestamps.

User inputs stay user inputs. Model outputs fill only the fields that need judgment. Persistence merges both. The UI renders the object. The voice session consumes questions as template variables. Feedback is a separate collection keyed by user + interview.

Once the script is a document, retakes, history, delete cascades, and score chips become ordinary CRUD. If the "interview" had lived only inside a model context window, none of that would be reliable.

Feedback is a contract, not an essay

Free-form "here's how you did" is easy to generate and hard to productize. Cards need a number. Detail pages need the same five categories every time. Retakes need overwrite semantics, not duplicate rows that fight in the UI.

So feedback is a schema:

• totalScore
• exactly five categoryScores with enum names
• strengths[]
• areasForImprovement[]
• finalAssessment

Before writing a new score, the server deletes prior feedback for that user + interview. Replace, don't accumulate. The rubric is the UI contract. The prompt reinforces the enum names, but Zod is what makes the page safe to render.

What happens when Gemini returns junk

Early on, I let the generation prompt run wild. "Generate interview questions for a senior React developer." The model would return beautifully formatted markdown with 15 questions, explanations, and a suggested timeline.

None of it was structured. The UI couldn't parse it. The voice assistant couldn't inject it. The feedback stage couldn't score against a script it couldn't read.

Switching to generateObject with Zod schemas fixed that immediately. Now invalid responses fail fast at the generation boundary, not when someone tries to start a call with malformed data.

const { object: interviewDetails } = await generateObject({
  model: google("gemini-2.5-flash"),
  schema: interviewSchema,
  prompt: `Prepare questions for a job interview...`
});

How voice sessions actually work

The voice runtime receives a system prompt with template variables:

You are a professional job interviewer...
Follow the structured question flow:
{{questions}}
...
Do not ask any questions that are not in the questions list.

The client injects the generated question list via variableValues when starting the call. The conversational model has the script; it doesn't invent one.

When users try to steer the conversation ("Can you ask about databases instead?"), the assistant politely redirects: "I have a set interview structure I need to follow. Let me ask you about [next question from the list]."

This constraint sounds rigid. In practice, it keeps 20-minute practice sessions on track instead of wandering into general career advice.

Voice is orchestration, not autonomy

The client owns the call state machine: idle -> active -> ended. It listens for transcript messages, buffers them, and only then POSTs to the feedback endpoint.

Scoring is a side effect the product decides to run after a completed call. It is not something the conversational model chooses when it feels done. If you let the voice model own hang-up semantics and persistence, you get "kind of saved" interviews. Kind of saved is useless when someone opens the dashboard later.

What one interview costs to run

Generation: ~2,000 input tokens + ~500 output tokens with Gemini Flash = $0.002

Voice session: ~$0.15 for a 15-minute call (VAPI pricing includes STT, TTS, and the GPT-4 conversation)

Feedback scoring: ~1,500 input tokens + ~300 output tokens = $0.0015

Total per interview: ~$0.15. The voice session dominates cost, not the structured AI stages.

Product decisions

1. Practice loop over hiring OS. No recruiter console, no candidate marketplace, no calendar sync. One person practices for a role.
2. Questions as data, interviewer as template. The script is pre-generated and stored; the voice model executes it, not plans it.
3. Structured feedback over chat-after-chat. Post-call scoring lands in a fixed shape. There is no open-ended "coach chatbot" competing with the dashboard.
4. Retake means replace. New feedback for the same interview overwrites the old score. History stays readable.
5. Managed voice instead of DIY realtime. Latency, barge-in, and voice quality are vendor problems until the loop is proven.

What I learned

1. Schema-first is what makes the pipeline possible. Generation without a contract produces mush you cannot score consistently. Scoring without a contract produces mush you cannot render.

2. An interview product fails when the model owns the interview. Once the script and rubric are product objects, the live call becomes a narrow performance surface instead of the whole architecture.

3. Hosted voice is a runtime, not a product strategy. It is the right place for conversational uncertainty. It is the wrong place for persistence, ACL, and workflow state.

What I'd do differently

1. Verify identity on every Nitro handler with a real Clerk session token, instead of treating a user id header/body as sufficient. Application ownership checks stay; they should not be the only gate.

2. Add evaluation fixtures for generation and scoring — fixed role inputs and transcript fixtures asserting schema shape and rubric completeness. Schema makes this cheap; not having the suite is underbuilt.

3. Gate interview and feedback routes the same way as the dashboard, so deep links are not accidentally public surfaces.

Technical appendix

Uncertainty boundary

Related: The LLM Should Be the Dumbest Part of Your System · Schema-First LLM Systems · Two-Step LLM Pipelines · Why Most AI Products Don't Need Agents · The Stack Is a Product Decision