Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
TravisMuhlestein 
posted an update 18 days ago
Post
133
AI orchestration > bigger models?

Over the past few months I've been experimenting with a different approach to AI-assisted creation.

Instead of asking one frontier model to do everything, I explored what happens when specialized capabilities are orchestrated into a single production workflow.

The project coordinates drafting, editing, illustration, multilingual translation, narration, publishing outputs, and marketing assets into one end-to-end system.

The first real-world validation is The Isaiah Chronicles, the first release in a planned 10-book series.

Over roughly two months the workflow processed more than 14 billion tokens, with continuous experimentation around evaluation metrics, quality gates, reproducibility, and orchestration strategies.

One observation kept repeating itself:

Improving the orchestration often produced bigger gains than changing the underlying model.

I'm curious whether others building AI systems are seeing the same pattern.

How much of the next leap comes from better models—and how much comes from better orchestration?

🔗 https://theisaiahchronicles.com
dipankarsarkar

Orchestration beat the model in your runs, and the way to know is to freeze one of them.

14 billion tokens over two months is a lot of evidence, but it is evidence about one moving system. The orchestration and the model both changed across it, so the gains got attributed to whichever one you happened to be touching that week.

The cheap test is a held constant. Take one stage you have already shipped, drafting or translation. Keep the graph, the prompts and the quality gates frozen, swap only the model. Then freeze the model and change only the orchestration. Same inputs, same gates, same scorer. Two deltas you can put side by side.

My guess is orchestration still wins, but for a reason that reframes the claim. Orchestration mostly buys retries, checks, and narrower asks. Those pay off in proportion to how often the model is wrong in a way your gate can actually see. That makes the win a property of your gates as much as of the model. A stronger model moves the same number only if the gate can see the errors it still makes.

Which of your quality gates has ever rejected a stage output, and how often does it fire?

·
TravisMuhlestein

Fair challenge—and the last question is the one I can actually answer with data.

The gates fire constantly, and the firing pattern turned out to be the real story. One example: the Spanish edition of Book 2 (~119k words). A first-pass audit rejected 46 of 55 sections, finding 188 confirmed defects—137 of them silent multi-word drops introduced by TTS. Aggregate WER looked acceptable, but word-level alignment exposed the failures. Running the same workflow on Portuguese found 140 defects, while French had only 4. Same orchestration, same prompts, same models—very different error profiles.

That led to two important lessons. First, the failures that mattered were often invisible to the layer producing them, so we combined transcript, acoustic, and alignment-based gates. Second, the gates themselves need validation—ASR can hallucinate defects, so a re-render is only triggered after a second VAD-filtered verification pass.

So I'd refine my original claim rather than double down on it: orchestration's advantage is really model error rate × gate visibility. Better models reduce the first term; better orchestration improves the second. In my experience, the second is still the more underexplored lever in production systems.

And yes, I agree the frozen-graph experiment is the right next step. Translation is probably the cleanest stage to test by holding the graph, prompts, gates, and scorer constant while swapping only the model. I'll run that experiment—thanks for the suggestion.
dipankarsarkar

French having 4 is the number I would not trust yet, and it is the cheapest one to check.

Same orchestration, same prompts, same models. 188 defects in Spanish, 140 in Portuguese, 4 in French. Two readings fit that. French TTS dropped fewer words. Or the French gate saw fewer of the drops it had.

Your stack is transcript, acoustic and alignment. All three are ASR-shaped, and ASR quality is not constant across languages. So gate recall varies across exactly the axis you are comparing.

That is your own second term moving while you read the first one. model error rate x gate visibility, with visibility unpinned.

The control is injection. Take French audio you already shipped, delete N known word spans programmatically, run the unchanged gate stack, count how many it catches. Same N on Spanish. Now you have gate recall per language, and 4 and 188 become 4/r_fr and 188/r_es.

If the recalls come back near equal, French really is cleaner, and that is a fact about the voice worth knowing. If French recall is 0.9 and Spanish 0.5, most of the gap was never the language.

It is the same frozen-constant move as the model swap, pointed at the gate instead. And it is the one you can run without re-rendering anything.

What is your gate's false-negative rate on a defect you planted yourself?

·
TravisMuhlestein

You called it — and it cracked even faster than injection testing. It was a denominator problem: the "4" was a mid-audit snapshot logged at 2 of 55 sections. The completed French pass found 179 defects (141 drops, 38 garbles) — right in line with Spanish (188) and Portuguese (140). No French exception; just a partial audit quoted as a full one. Your general point, proven on contact: the visibility term was unpinned.

Direct answer to your question: unknown — we built the false-positive control (VAD-verified second pass before any re-render) but never the false-negative one. No planted defect has ever tested these gates.

That's a great callout, and I'm adding the injection harness now — planted word-span deletions, silence gaps, and tail loops per language — so defect counts ship as count/recall and future deviations get caught by the instrument instead of by a sharp reader. Thanks for the push. This is exactly the kind of feedback that makes these systems better.
dipankarsarkar

Thank you, but no. Keep the book.

A review is worth more if nothing changes hands, and I would rather the next one read the same way this one did.

The thing I would actually want is the recall table. You are building the injection harness now, so per-language recall becomes a number you can publish: planted spans, caught, missed, split by language and by defect type. Right now 179 French, 188 Spanish, 140 Portuguese reads as a language comparison, and it only becomes one once the denominators are pinned. Once they are, those stop being counts and start being estimates.

Two things worth planting beyond word drops, since they are what an ASR-shaped gate is least likely to see: a correctly transcribed but wrong number, and a swapped speaker turn. Both survive word-level alignment intact, so all three of your gates can agree and still be wrong together.

That also gives you a cheap check on the thing you just found. A partial audit quoted as a full one is a denominator bug, and an injection harness is the instrument that catches it without needing a sharp reader.

What does French recall come out at?
In this post