When 40% of Your LLM Children Return Junk: Graceful Degradation in Cron Pipelines

The parent task was green. Three leads queued. Then someone on the ops side asked a very reasonable question: why did we email a founder about a mobile engineer when the company was clearly hiring backend?

The cron hadn't crashed. OpenAI hadn't rate-limited us. Every child task returned valid JSON. The orchestrator logged leadsQueued: 3 and closed the book on a successful run. That's the failure mode I almost shipped: not a red dashboard, not a thrown exception, just output that looked fine in the logs and wasn't fine in the inbox.

I'd been building an automated outbound recruiting pipeline and had just finished the concurrency half of the problem: fan-out with batchTriggerAndWait, backpressure, surviving partial batches. That story lives in a separate post. This one is about everything that happens when the data feeding those two hundred ranking calls is thin, wrong, or missing, and the cron still has to finish on schedule anyway.

Fan-out got me throughput. Degradation is what kept the pipeline from lying to me.

The green checkmark that lies

I used to think there were two ways an LLM pipeline could fail.

The loud way is easy: a 429 from OpenAI, a Zod parse error, Fiber down. The task goes red. You open Trigger.dev and figure it out.

The quiet way is worse. The model hands you { score: 78, rationale: "Strong fit for fast-paced teams", redFlags: [] } for a backend candidate against a job description that literally says "join our fast-paced team" and nothing else. On one staging company, about forty percent of the ranking children looked like that: high scores, generic rationales, obviously wrong if you actually read the resume.

Valid JSON. Useless judgment.

The orchestrator still completed. pickTopMatch picked the least-bad survivor and moved on. Nobody threw an error because nothing in the schema was violated. Without explicit gates, a passing run becomes evidence of quality when it isn't.

A cron that never crashes but sometimes emails the wrong person isn't production-ready. It's production-dangerous.

Where things actually break

Every twelve hours the outbound engine does the same loop: find companies, enrich them, rank the resume pool, queue Instantly campaigns. What took me a while to internalize is that each layer fails differently. Discovery might return nothing. Enrichment might return half a picture. Ranking might return confident nonsense. Outreach might have nobody to email.

The decision at each step isn't really success or failure. It's closer to: do I retry this, skip it, take a worse path, or stop the whole run?

I ended up with a table like this taped to my mental model. Most rows end in "run continues." That bothered me at first. Shouldn't a broken pipeline fail loudly? Then I watched a twice-daily cron throw on every company with a missing job post and realized I wasn't protecting quality. I was just making the dashboard red for normal Tuesdays.

Four verbs I kept confusing

For a while my pipeline either crashed too often or shipped garbage. The fix was being boringly precise about four words.

Retry

Retry when the same input might work if you wait a beat.

OpenAI 429s. Fiber timeouts on a single lookup. Instantly rate limits during campaign activation. That stuff is transient.

I keep retries on child tasks, not wrapped around the orchestrator. If the parent retries a whole company loop, you redo work siblings already finished. Child tasks get isolation; the orchestrator stays dumb.

export const outboundRankApplicantTask = task({
  id: "outbound-rank-applicant",
  retry: {
    maxAttempts: 3,
    factor: 2,
    minTimeoutInMs: 1_000,
    maxTimeoutInMs: 30_000,
  },
  run: async (payload) => {
    // one LLM call, one schema parse
  },
});

Skip

Skip when this particular company or lead can't be processed right now, but the rest of the run is still worth doing.

No founder email. No confident match after ranking. Company already on the exclusion list.

Skipping used to feel like failure to me. It isn't. It's the pipeline saying "nothing useful to do here." The important part is logging why, so ops can tell the difference between "nothing to do" and "something broke."

if (!founderEmail) {
  logger.info("outreach_skipped", {
    companyId: company.id,
    reason: "no_founder_email",
  });
  continue;
}

Degrade

Degrade when you don't have the ideal signal, but you have some signal.

The main example in this pipeline: no indexed job post, but recent employee profiles exist. Early on I skipped those companies entirely. Later I switched ranking mode instead.

// src/trigger/outbound.ts, runtime ranking strategy
if (hasJobData) {
  batchResult = await outboundRankApplicantTask.batchTriggerAndWait(
    applicants.map((applicant) => ({
      payload: {
        applicantId: applicant.id,
        jobDescription: jobDescription!,
        jobTitle: jobTitle!,
      },
    }))
  );
} else if (employees.length > 0) {
  batchResult = await outboundRankApplicantByEmployeesTask.batchTriggerAndWait(
    applicants.map((applicant) => ({
      payload: {
        applicantId: applicant.id,
        employees,
        companyName,
      },
    }))
  );
} else {
  logger.info("rank_skipped", {
    companyId: company.id,
    reason: "no_job_no_employees",
  });
  return null;
}

That turned "no job post, skip" into something sendable: "Here's a candidate similar to [Engineer You Just Hired]." Same Zod schema for scores, different prompt, same orchestrator. Not a second pipeline, a fallback path. I'll dig into that branching pattern in a follow-up post. This one is more about when to stop.

Fail

Fail sparingly. I only hard-fail when continuing would corrupt state or burn money pointlessly.

Revoked API credentials. A database table that's genuinely unreadable. Cases where running again in an hour won't change anything.

Empty applicant pool is different: I exit early with { ok: true, skipped: "empty_pool" } instead of throwing. Ops still gets a green run with a reason attached. I reserve red runs for things that actually need a human.

What actually happened in staging

The company with the useless job description

Staging had been going fine. Scores clustered between 55 and 85. Rationales made sense. Then we hit a company whose job description was pure boilerplate: "join our fast-paced team," no stack, no seniority, no actual role.

About forty percent of the ranking children came back with valid schema JSON and nonsense inside. High scores for clearly mismatched candidates. The model had latched onto "fast-paced" and called it a day.

Valid JSON. Failed business logic. Same problem as the mobile-engineer email.

The fix that helped: don't fan out at all if the context is too thin. If the job description is under ~120 characters and there are no employee profiles to fall back on, skip ranking for that company.

if (hasJobData && jobDescription!.length < 120 && employees.length === 0) {
  logger.info("rank_skipped_thin_context", { companyId: company.id });
  return null;
}

On a typical run (five companies, ~45 applicants in the pool), that's up to 225 ranking calls you never make for a target that can't give you signal. Roughly $0.40–$0.80 saved per thin-context company at GPT-4o pricing. Not exciting per run. Adds up across twice-daily staging over a few weeks.

Missing data is the default, not the exception

I went in assuming most companies would have a usable job post. They don't. Across staging, roughly 30–35% of discovered companies showed up without one.

Before degradation, that was an automatic skip. After employee-profile ranking, about two-thirds of those became viable, not as strong as a real job description match, but good enough to send a "similar to your recent hire" angle.

If your pipeline only works when enrichment is complete, you built it for a demo.

The run that queued zero leads

Ops enabled the schedule and asked why the first real run produced nothing. Parent task: green. Logs: leadsQueued: 0. No child failures. No errors.

I had to dig into skip reasons. Two companies gated out for thin context. One failed the confidence check: top scorer had three red flags, second place was only two points behind. One had no founder email. Four perfectly reasonable skips, one summary integer that made it look like the pipeline was broken.

logger.info("outbound_run_complete", {
  companiesDiscovered: companies.length,
  leadsQueued: leads.length,
  skipped: {
    thin_context: skipCounts.thin_context,
    low_confidence: skipCounts.low_confidence,
    no_email: skipCounts.no_email,
    no_ranking_context: skipCounts.no_ranking_context,
  },
});

That was the moment I stopped trusting leadsQueued alone.

Gates after aggregation

Skipping before fan-out saves tokens. Gating after aggregation saves you from emailing a founder about the wrong person.

The match schema already includes redFlags: string[]. I added business rules on top that the schema can't express:

function isConfidentWinner(
  top: ScoredApplicant,
  runnerUp: ScoredApplicant | null
): boolean {
  if (top.score < 65) return false;
  if (top.redFlags.length > 2) return false;
  if (runnerUp && top.score - runnerUp.score < 5) return false;
  return true;
}

That last one pairs with per-child retries. If more than 40% of children fail outright, I don't trust the batch enough to pick a winner. Degrade to skip.

const succeeded = batchResult.runs.filter((r) => r.ok).length;
const successRate = succeeded / batchResult.runs.length;

if (successRate < 0.6) {
  logger.info("rank_skipped_degraded_batch", {
    companyId: company.id,
    successRate,
    succeeded,
    total: batchResult.runs.length,
  });
  return null;
}

Dedup counts too

Degradation isn't only about missing fields. Sometimes the right move is not doing work that shouldn't happen at all.

After each run, every contacted company goes on a Fiber exclusion list. Without that, a twice-daily cron eventually finds the same founders again.

await fiberRequest("/exclusions/companies/add-to-list", {
  listId: process.env.FIBER_EXCLUSION_LISTID,
  companies: companies
    .map((company) => ({ domain: company.domains?.[0] }))
    .filter((c): c is { domain: string } => !!c.domain),
});

I do this even when outreach was skipped. Discovery already spent budget. The domain shouldn't show up tomorrow.

Logging that answers "why zero?"

My early mistake was one log line on the parent and silence everywhere else. When ranking quality drifted, I'd see leadsQueued: 3 and learn nothing about the two companies that got skipped.

Parent run = ops-channel summary. Child runs = LLM debugging. Per-company skip reasons = answering "why zero today?"

Numbers I actually watch

You don't need a warehouse for this. A few counters per run, looked at over a week, tell you if your gates are too tight or too loose.

From roughly three weeks of twice-daily staging (~45-applicant pool, ~5 companies per run):

The metric that changed my headspace was cost per successful lead, not cost per run. A $4 run that queues one good lead beats a $2 run that queues two embarrassing ones. The second kind costs more than tokens.

Same idea, other pipelines

This isn't outbound-specific. Any cron that enriches data, asks an LLM to judge it, then triggers a side effect has the same shape.

DocPilot skips generation when deterministic extraction can't find a handler; don't ask the model to guess. Classification jobs can degrade to lower-confidence paths instead of one-shotting the whole repo. Scoring pipelines can refuse to email, bill, or alert on weak winners.

Different code. Same question at every layer: retry, skip, degrade, or stop?

Coming next

Job ranking and employee ranking share one matchResultSchema but different prompts and child tasks. One orchestrator picks the path at runtime based on what enrichment returned. The next post is about that branching: when to take the fallback path vs when to skip entirely, and how score distributions differ between modes.

If you only remember one thing

Load shared inputs once. For each company: enrich, decide, gate before the expensive fan-out, aggregate carefully, log why you skipped. Side effects (emails sent, domains excluded) happen in the orchestrator only after everything passes. A green parent run means the schedule held. It doesn't mean every company deserved an email.

Related: Orchestrating parallel LLM workloads with batchTriggerAndWait · Automated outbound recruiting case study · Two-step LLM pipelines