I shipped a pipeline change last month that I was genuinely proud of. A prompt rewrite. A full-context review step. A detailed rubric to catch repetition and enforce structural variety across segments.
I deployed it. Observed it for six weeks. Told myself it was working.
Then I built an eval framework. Ran it on 40 real production episodes across 7 languages. And discovered that the review step — the one I'd been so careful to design — was making scripts measurably worse.
Not neutral. Worse.
The per-segment dialogue scored 3.34 out of 5. The outline scored 3.01. And the final script, after the review pass? 2.91.
The step designed to improve scripts was reducing quality by over 12%.
This shift — from "I think it's working" to "I know it's not" — started a few weeks ago. I was watching Andrew Ng's Agentic AI course on DeepLearning.AI, where he stresses something that sounds obvious but isn't: evals and traces are the foundation of any agentic system. Without them, you're debugging in the dark. With them, you know exactly which link in the chain broke.
I'd been running ad-hoc script quality checks for months — structural metrics like turn length ratios and speaker balance, plus an LLM judge comparing A/B prompt variants. They caught obvious regressions. But they weren't comprehensive. They covered one stage (dialogue), one language at a time, and were designed for prompt A/B testing, not pipeline-wide quality audits.
Andrew's point about traces landed hard. My pipeline has five major LLM calls per episode — research, outline, per-segment dialogue, intro/outro, and review — plus an ElevenLabs TTS pass. I had quality metrics for one of them. The other four were being trusted on vibes alone.
So I built a real eval system. Complete. Multi-dimensional. Re-runnable. Here's what it found, the six fixes I shipped in one day, and what actually improved.
The Baseline: Three Numbers That Changed Everything
I evaluated 40 completed episodes from my production database — covering all 7 languages (en, vi, ja, ko, es, zh, fr) and 10 podcast styles — against three rubrics with five dimensions each. An LLM judge (GPT-5.6-terra) graded every stage.
| Stage | Score | Worst Dimension |
|---|---|---|
| Dialogue (per-segment) | 3.34 | Factual Grounding: 2.17 |
| Outline (research + structure) | 3.01 | Citation Credibility: 1.28 |
| Final Script (post-review) | 2.91 | Publishability: 1.95 |
Three things jumped out immediately:
1. The review pass was degrading scripts by 0.43. Dialogue scored 3.34. After review? 2.91. The step designed to improve quality was removing it.
2. Citation URLs were all opaque redirects. Every "source URL" in every outline pointed to vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQ.... These work for humans clicking them, but look completely fabricated to automated reviewers. 344 facts had source URLs. Zero had resolved_url — a field that didn't even exist yet.
3. Publishability was the lowest dimension across the entire pipeline (1.95). Real podcasters, the eval judge determined, would not publish these scripts under their own name. Detectable AI patterns: identical segment openers, generic transitions, missing host personality.
The eval gave me a ranked list of exactly what to fix. I spent the rest of the day fixing it.
The Six Fixes (And Their Measured Impact)
Fix 1: Citation URLs — 0% to 100% Resolution
Before: 0 out of 344 research facts had a resolved_url. Every URL looked like it was fabricated.
Fix: A 60-line URL resolver that runs during outline generation, follows the vertex AI redirect to the actual source, and stores both URLs (original for Google ToS compliance, resolved for everything else).
def resolve_grounding_url(url: str) -> str:
if "grounding-api-redirect" not in url:
return url
response = requests.head(url, allow_redirects=True, timeout=5)
return response.url if response.url != url else url
After: 23 out of 23 research facts across today's new episodes have resolved_url populated — real URLs like simonwillison.net, axios.com, thinkingmachines.ai. 100% resolution.
Fix 2: Swapped the Review Model (Gemini → DeepSeek)
Before: Gemini 3.5 Flash ran the review pass. It made light, conservative edits: changed "Exactly!" to "No question about it!" but mostly preserved the original text. It added filler that flattened the dialogue. Audio tag density dropped. Type-token ratio dropped. Turn count bloated.
Fix: Swapped the review pass to DeepSeek v4-pro (384K max output, set to 96K cap). Made the provider configurable:
provider = get_provider("review") # reads REVIEW_PROVIDER env var
response = provider.generate(
prompt=prompt, temperature=0.0,
max_output_tokens=96000,
)
Verification: Deployed to production. A Vietnamese tech news episode ran through the full pipeline with DeepSeek review — 10.3K → 11.6K tokens, completed in 121 seconds, 0 tightening passes needed. A second episode (English) processed 7.9K → 17K tokens in 203 seconds. The DeepSeek API key lives in Google Cloud Secret Manager, injected automatically at deploy time via service.yaml.
Fix 3: Outline Self-Critique (+0.50 Improvement)
Before: Outlines went straight from Gemini to the user with no internal quality check. Average score: 3.01.
Fix: Added an automated critique step. DeepSeek v4-pro evaluates the outline against a 6-dimension rubric (the 5 outline dimensions plus segment count), returns scores + specific feedback, and stores the critique in the podcast metadata.
Then I ran a controlled experiment: take 6 episodes whose outlines scored below 3.0, run the DeepSeek critique, feed the feedback back to Gemini as regeneration instructions, and re-score.
After: +0.50 overall improvement across the 6 episodes.
| Dimension | Before | After | Delta |
|---|---|---|---|
| Content Boundaries | 3.33 | 4.17 | +0.83 |
| Research Depth | 2.67 | 3.17 | +0.50 |
| Citation Credibility | 1.00 | 1.50 | +0.50 |
| Structural Logic | 3.67 | 4.17 | +0.50 |
| Audience Fit | 2.17 | 2.33 | +0.17 |
The worst episode went from 1.80 to 3.20 (+1.40). Only 1 of 6 got worse. The critique is now enabled by default in production — every new outline gets this quality check before the user sees it.
Fix 4: One-Segment Outlines Now Fail Hard
Before: The eval showed episodes with a single segment and zero research facts scoring as low as 2.07. These were failure cases the pipeline wasn't treating as failures.
Fix: If the outline generator produces fewer than 3 valid segments after a recovery attempt, the podcast is marked FAILED. No more one-segment outlines reaching users. On the next creation attempt, the user's credit is freed to retry.
Fix 5: Chinese Localization Rules
Before: Chinese episodes scored 2.88 — the lowest across all 7 languages. Translation artifacts (English idioms translated literally), identical segment openers, default English host names (Alex/Maya) instead of Chinese names (明辉/晓雯).
Fix: Wrote Chinese-specific dialogue rules: native conversational patterns (不是...而是... constructions), proper sentence-final particles (吧/啊/呢), anti-translation guidelines (never translate "medical miracle" literally), Chinese football metaphors for World Cup episodes, and locale-specific host profiles with distinct personalities.
The Chinese dialogue template went from reusing English defaults to having its own 80-line guide covering conversational patterns, disagreement phrasing, and audience-specific framing for every podcast style.
Fix 6: Host Name Defaults for Non-English
Before: Multiple non-English episodes had hosts named Alex and Maya. The locale-specific host profiles existed but the fallback path in the outline generator was hardcoded to English defaults.
Fix: Traced the fix through the locale-aware profile resolution pipeline. Recent production episodes now use 翔太/健一 (ja), Mạnh/Nga (vi), Hugo/Camille (fr), 明辉/晓雯 (zh). The July commits already fixed the edge function and creation flow — the remaining fallback was in the outline generator itself.
The Eval Framework (Reusable)
The framework that found all of this is ~500 lines of Python, three rubric files (5 dimensions each), a shared GPT-5.6-terra judge, and an orchestrator:
docs/pipeline/evals/
├── run.py # Orchestrator: extract → judge → report
├── judge.py # Shared LLM judge (rate with rubric)
├── rubric_outline.py # Stage 1: 5 dimensions, 1-5 scale
├── rubric_dialogue.py # Stage 2: 5 dimensions, 1-5 scale
├── rubric_final_script.py # Stage 3: 5 dimensions, 1-5 scale
├── dataset_40.json # Extracted production data
├── scores.json # Machine-readable scores
└── report.md # Auto-generated report
I also built an experiments layer for A/B comparisons:
docs/pipeline/evals/experiments/
├── review_ab.py # Compare two review models
├── outline_critique_ab.py # Critique → regenerate → compare
├── shared.py # Score comparator, report generator
└── results/ # Archived by date
Run python docs/pipeline/evals/run.py --full after any pipeline change and get a report in ~8 minutes. Run experiments/review_ab.py --episodes 10 to compare models. Results are archived by date for trend tracking — want to know if quality improved this month vs last month? Diff the scores.
The full 40-episode eval costs about $1-2 in API calls. The experiments cost less. For the signal you get back — a ranked list of exactly which pipeline stages and dimensions need attention — it's the cheapest debugging tool I've built.
What's Still Broken
The DeepSeek review A/B experiment needs the full production prompt. The experiment script used a simplified 15-line prompt. The production review uses an 80-line prompt with anti-repetition rules, catchphrase quotas, editorial guardrails, and duration policy. With the simplified prompt, DeepSeek showed essentially neutral results (-0.04). With the full prompt in production, the pipeline is completing successfully across English and Vietnamese. I need to update the experiment to mirror production and rerun.
Chinese localization needs validation. The rules look right on paper, the host profiles are written, the dialogue guides are in place. But I haven't yet run the eval on new Chinese episodes to confirm the score moved from 2.88. That's a priority for this week.
I need to re-run the full 40-episode eval on new episodes. All six fixes are deployed. The next step is to extract 40 episodes generated after these changes, run them through the same evals, and compare against the historical baseline. If the final script score moves from 2.91 to above 3.0, and citation credibility moves from 1.28 to something reasonable, I know the changes worked end-to-end.
The Three-Question Pipeline Audit
After this experience, I now audit every pipeline change against three questions:
1. Is there a quality gate between each step? My outline was going from Gemini directly to the user. Adding the critique step caught the worst outlines before anyone saw them. Every pipeline step needs a quality check — automated LLM evaluation against explicit criteria.
2. Does each step actually improve quality when measured? Running it and assuming it helps isn't enough. You need literal before/after scores on the same content. My review step degraded scripts for six weeks before I measured it. Measure first, assume nothing.
3. Can I tell which dimension broke? "Quality is bad" is not actionable. "Citation credibility is 1.28 because 100% of URLs are opaque redirects" is. A multi-dimensional rubric per stage tells you exactly where to look. I spent hours investigating my review step before the eval. With the eval, I spent seconds — the numbers pointed directly at publishability and the review model.
Frequently Asked Questions
Doesn't running LLM evaluations get expensive?
The full 40-episode eval costs about $1-2 in API calls (GPT-5.6-terra, ~420K total tokens). The A/B experiments cost even less. For the signal you get back — a ranked list of exactly which pipeline stages need attention — it's the cheapest debugging tool I've built, and cheaper than the credits users would waste on broken episodes.
Why 5 dimensions per stage? Why not just "overall quality"?
"Overall quality" tells you something is wrong. It doesn't tell you what. When I saw Citation Credibility at 1.28, I immediately knew the URLs were broken. When I saw Publishability at 1.95, I knew AI tells were detectable. Five dimensions give you a differential diagnosis. One dimension gives you a fever thermometer.
How do you keep the LLM judge consistent?
Same model (GPT-5.6-terra), same temperature (0.0), same reasoning effort (medium), same rubric prompts every time. The scores are archived with timestamps so you can compare across dates. Changing the judge changes the baseline — pick one and stick with it.
What happens when the judge itself is wrong?
The rubric includes required justifications for every score — the judge has to cite specific evidence from the content. I also did a manual qualitative review of 10 episodes across all 7 languages to calibrate the judge. The LLM scores matched my human judgment closely. For critical dimensions where the LLM might be unreliable (like "publishability," which requires cultural judgment), human spot-checks are essential.
I build DIALOGUE, an AI podcast platform, solo in my evenings and weekends. I write about what I learn along the way.
If you've built an eval system for your own LLM pipeline, I'd be curious: how do you measure quality, and what did you discover that surprised you?
That's it from me for now.
Cheers, Chandler