Calling an LLM is almost embarrassingly easy now.
You can go from a vague idea to this in about ten minutes:
const response = await model.generate(...)
The output looks intelligent. Someone in the room says wow. It feels like you skipped the boring parts of product development.
You didn't. You built a function call.
A demo asks whether the model can do the thing. A product has to survive the next ten thousand of those calls: the ones that time out, return valid JSON with nonsense inside, run twice because the cron overlapped, or cost more than the feature is worth.
The model is one component.
The product is the system around it.
The demo is lying to you
AI demos are deceptively easy because they hide everything that happens before and after the model.
I keep learning this the same way. The version that works on my laptop is not the version that survives someone else's data.
DocPilot looked finished on my test project. Then it hallucinated endpoints on everyone else's. The model could write documentation. The product could not be trusted to know which files were actually routes.
The outbound recruiting engine ranked eight test resumes just fine. In staging, with a real pool, one company took four minutes if I awaited each call. Five companies sequential meant twenty minutes of ranking before discovery, enrichment, or Instantly setup. The model could score a resume. The product could not finish a twelve-hour cron.
Peergrowth Pulse could have shipped a copilot that looked smart in a screenshot: dump campaign JSON into a prompt, ask why CPL moved, get a confident paragraph. That version invents numbers in a client conversation.
A prototype can hide complexity. Production exposes it.
The demo never has to answer the questions that actually matter. What happens when the request times out? When the same job runs twice? When the model changes its output shape? When Fiber is down, or Meta rate-limits you, or forty ranking children 429 at once? How much does this cost when it runs twice a day for a month?
"Can the model do this once?" is a research question.
"What happens when it fails at 2am?" is a product question.
Those are not questions about the model. They are questions about the rest of the system. Which is why, in anything I have actually shipped, the LLM is usually the smallest part.
The LLM is usually the smallest part
I have already argued that the model should be the smallest part of the system. That post was about authority. Don't let the LLM own the workflow.
This is the other half of that argument. Once you put the model in a small box, you discover that the box was never the hard part. The workflow is usually already known. You don't need an agent to rediscover it. You need the known steps to survive production.
The generate call is often twenty lines. The product is everything that makes those twenty lines safe to run.
input
-> validation
-> deterministic processing
-> context retrieval
-> LLM judgment
-> validation
-> business rules
-> persistence
-> retries
-> idempotency
-> observability
-> user-facing result
The interesting engineering is usually not "which model." It is the pipeline that decides what the model is allowed to see, whether its answer is usable, and what the rest of the system does next.
In FinCrew, duplicate payments and weekend spend are predicates. Only the ambiguous remainder goes to Claude Haiku. If Haiku is unavailable, the checks keep working. A review queue still exists.
Pulse is the same shape from the other direction. Lead sync, attribution, freeze-once reports, and workspace ACL were the product before anyone typed into a copilot. The hardest AI problem there was permissioning the model's world, not picking Gemini.
The LLM should usually make a small, bounded judgment. Normal software should run the rest.
If you cannot point to that split in the codebase, you do not have an AI architecture. You have a prompt with a UI.
Once the model is a small box, the remaining work is not glamorous. It is the work that decides whether you can trust the box.
The boring parts become the important parts
This is the part demos skip, and the part I actually spend time on.
If it can run twice, it will
A twelve-hour outbound cron that discovers companies will eventually rediscover the same founders. That is not an LLM problem. That is an exclusion list.
After each run, contacted domains go on a Fiber exclusion list. I do this even when outreach was skipped. Discovery already spent budget. The domain should not show up tomorrow. The model should not be asked whether we already emailed this person. The system should already know.
Pulse has the same class of bug in a different costume. Hourly Meta lead sync cannot be "just run again." Trigger.dev tasks use idempotency keys per account-hour and skip-if-in-flight tags so two overlapping syncs do not fight over the same leads.
Idempotency is unglamorous. It is also the difference between a pipeline and a spam cannon.
Retrying the model is not retrying the product
Retrying an LLM call is not always as simple as calling it again.
If a ranking child times out after OpenAI actually succeeded, a retry spends the tokens twice and you get another score. For ranking, that is annoying and usually acceptable. The child is isolated. Siblings are not blocked. Trigger.dev retries with backoff. The orchestrator stays dumb.
If you retry the parent instead, you redo discovery, enrichment, ranking, and campaign activation. That is how you email a founder twice with a very confident match score.
The rule I ended up with: retries belong on the unit of work that is safe to repeat. Ranking children retry. Instantly side effects stay in the orchestrator, after aggregation, after the gates. FinCrew's ingest and fraud scans run as background jobs for the same reason. A failed scan should recover without losing the import, and it should not block the request path.
A timeout is not a single failure mode. You have to know whether the previous attempt might already have done the thing.
Valid JSON is not a product result
The schema is the contract. It is not a quality guarantee.
The failure that taught me more was not a parse error. It was a green run.
The outbound parent was green. leadsQueued: 3. Then ops asked why we emailed a founder about a mobile engineer when the company was hiring backend. Every child had returned valid JSON. On a staging company with a boilerplate job description, about forty percent of the batch was the same failure: high scores, generic rationales, the model latched onto "fast-paced team." Nothing in matchResultSchema was violated. pickTopMatch picked the least-bad survivor.
That is the production version of "the model worked."
Business gates sit after validation: a score floor, too many red flags, a tiny gap between first and second. Graceful degradation is the longer version. The point here is simpler. Don't trust structured output just because it parsed. Parsing means the model spoke your language. It does not mean it said something you should act on.
Concurrency changes the question
"Can the model rank this resume?" is a different problem from "can I rank forty applicants against five companies before the cron's next run, without blowing the TPM cap?"
Sequential await was the obvious first version. It died in staging. Fan-out with batchTriggerAndWait was the fix for wall-clock time. Companies stayed sequential on purpose. That is backpressure. The pool was capped at query time, not at fan-out time.
The metric that changed how I thought about it was cost per successful lead, not cost per run. A cheap run that queues two embarrassing emails costs more than a slower run that queues one you can stand behind.
Once you are processing many items, the interesting work is batching, rate limits, partial failure, and budgets. The model call is the payload inside that problem, not the problem itself.
Take retries, gates, and skip paths seriously and you keep arriving at the same decision. Sometimes the right call is no call.
A good AI product knows when not to call the model
The best architecture I keep shipping uses the LLM for a small judgment, and only if the rest of the system has something worth judging.
Thin job description, no employee profiles? Skip ranking. Don't burn two hundred calls to learn nothing.
Classifier said this file is a route and extraction finds no handler? Skip generation. Don't ask the model to guess.
Duplicate vendor payment on the same day? Fire the rule. Don't ask Haiku if two identical rows look suspicious.
Pulse's dashboard still shows numbers if Gemini is having a bad hour. FinCrew's fraud rules still run. DocPilot still knows which routes exist if generation fails.
That is not pessimism. It is product design. Uncertainty belongs in one place. If the model owns the whole system, provider downtime, format drift, and a bad prompt become downtime, format drift, and a bad prompt for the whole product.
A prototype calls the model because it can.
A production system calls the model because this specific step needs judgment, the context is good enough, and the rest of the workflow can survive a no.
AI products are still software products
AI does not eliminate traditional software engineering. It makes the boundary between deterministic and probabilistic systems more important, because you can now put a fuzzy step inside a product that used to stop at the WHERE clause.
Auth is still auth. A unique constraint is still a unique constraint. Permissions are still not a prompt. Pulse's copilot is not allowed to invent CPL or mutate a lead because a sentence sounded like a follow-up. ORACLE can only reach FinCrew's own services. Numbers it does not track stay unavailable rather than estimated.
The best AI products I have shipped are not the ones using the biggest model. Gemini for DocPilot was a cost decision at volume. Haiku for FinCrew fraud was a cost and latency decision. GPT-4o for outbound ranking was a structured-judgment decision. The products were the pipelines those calls sat inside.
I do not think this makes the work less interesting. I think it is why building software got interesting again. You finally have a place to put judgment. You still have to build the machine that can afford to have one.
That machine has a simple requirement. It has to work even when you do not believe the model.
You don't have an AI product until you can distrust the model
I can get a model to look smart in an afternoon. Making the rest of the system honest about that call is the work.
Retries that don't double-send. Idempotency that doesn't depend on the model remembering. Validation that can reject a beautiful paragraph. A skip path that is allowed to produce nothing. Logs that can explain why nothing happened. A budget that survives the week.
You don't get there by trusting the model more. You get there by putting enough software around it that you don't have to.
The interesting part of an AI product isn't getting the model to answer.
It's everything that still has to be true after it does.
Related: The LLM Should Be the Dumbest Part of Your System · AI Makes Software More Interesting to Build · Why Most AI Products Don't Need Agents · Schema-First LLM Systems · Two-Step LLM Pipelines · Graceful Degradation in Cron Pipelines · Orchestrating Parallel LLM Workloads · FinCrew · DocPilot · Peergrowth Pulse · Automated Recruiting Outreach