controlmt-v2.3 / docs /top-lessons.md

docs: move to release/docs/, neutralize internal-doc refs, add prerequisites + data source links + cost estimate + first-week checklist + architecture diagram, rewrite working-with-claude opening in natural voice

31eae1a verified 17 days ago

preview code

Raw

History Blame Contribute Delete

15 kB

Top 10 lessons from building ControlMT

The highest-density version. If you only read one doc in this folder, read this one.

The other docs (the journey, what didn't work, how it was built, working with Claude) each describe pieces of the picture. This one is the synthesis — the ten things I'd carve into stone before starting another project like this.

Ordered roughly by impact-if-violated.

0. Rule Zero — audit your tokenizer before training anything

A broken tokenizer is a foundation crack. The model only sees token IDs and can never recover from a bad vocabulary.

We learned this the hard way. v1/v2 used a 64K SentencePiece Unigram with no curated vocabulary. The model output Fucklands for the input Falklands — same root cause for PyTorch → 4 mangled subwords, dark matter → "ಗಾಢ ಪಥ" (dark path), Shika → "Chicago". Each is a tokenizer fault. No amount of model retraining fixes these — the broken paths are encoded in the vocabulary.

The audit is non-negotiable (script: scripts/tokenizer_audit.py). Required pass criteria:

No obscene tokens in vocab
Single token for top-50K English words (99.7% real-world coverage)
Single token for top-30K Kannada words
Single token for top-1K named entities (curated)
Single token for top-500 tech terms
Single token for currency/units
Native script coverage for all targets — never byte-fallback
No bigram pairs that form slurs
Compound word integrity ("dark matter", "loss function")
Char/token ratio: ≥ 3.0 English, ≥ 2.5 Kannada

If any gate fails, retrain the tokenizer before starting model training. The 14 days of v1 training I did on the bad vocabulary was wasted — we had to retrain the tokenizer for v2.1 anyway.

Lesson written into PROTOCOLS.md as Rule Zero. Don't repeat my mistake.

1. Lock your niche on day one — and don't unlock it

The single highest-value decision in this whole project was made before any code was written: I locked the positioning in a strategy memo on 2026-04-30. The key sentence:

We will NOT beat them on raw general translation quality. But we CAN beat them on specific niches where their general focus leaves gaps.

Every public claim I made about ControlMT in the months that followed was a restatement of that day's framing. Without the lock, I would have wasted weeks chasing general-translation parity with IndicTrans2 (which I would have lost). With the lock, I had clear answers to "should I add this feature?" — yes if it serves the niche, no if it doesn't.

The corollary: if your project's pitch hasn't survived multiple weeks of pressure to add scope, you don't have a niche yet — you have a wish list. Tighten it. ControlMT's pitch fits in one tweet: "139M Kannada↔English translator, code-mix native, Anti-LM contrastive decoding, Apache 2.0, runs on consumer GPU." Three months later, that's still exactly what shipped.

2. Data is the bottleneck. Plan for 60% of your time there.

I budgeted my time as "60% modeling, 30% data, 10% deployment." The actual breakdown was closer to "25% modeling, 60% data, 15% deployment."

Things that took longer than expected:

CometKiwi-scoring 6.7M parallel pairs (57 hours of wall-clock GPU time)
Misalignment scan that found 5 structural regions in the corpus (~2 days of debug + scan tuning)
Adult-content scan that surfaced 1,628 regex hits, all false positives after manual review (a full day of "is this really safe to release?")
Specialized stream curation (transliteration, code-mix, numerical augmentation)
Three iterations of bad-pair quarantine + audit

The modeling code, in contrast, has barely changed since v2.1 in May. Once the architecture worked, it worked. The dataset work never stopped.

If you're starting a from-scratch model project, double your initial data-time estimate. Then double it again if your target language has fewer high-quality public datasets than English.

2a. Encode asymmetric data patterns in the dataset code, not the training-loop hopes

v2.1 trained code-mix Kannada as both source AND target. Result: the decoder learned to emit code-mix patterns for plain inputs (catch → ಕ್ಯಾಚ್). Once trained that way for several epochs, fine-tuning couldn't unlearn it. v2.2 had to be from scratch — 3.5 days of GPU instead of a 1-2 day fine-tune.

The fix was a 10-line kn_is_mixed flag in model/preprocess_pairs.py that gates the EN→KN swap. It should have been there from the start of v2. I learned the asymmetry from the failure mode; if you can predict the asymmetry from the data, encode it in the loader.

General rule: if your dataset has a pattern that's only valid in one direction, prove that in code, don't hope the training averages it out. Common asymmetric patterns:

Code-mix source → clean target (yes), code-mix target (no)
Long source → long target (yes), short→long pair (suspicious)
Numeric source → numeric target (must match), numerical-drift (catastrophic)

Once the decoder learns a wrong signal across epochs, warm-starting won't fix it. Plan for from-scratch retraining as a possibility when designing the data pipeline.

3. Ship the stepping stones

v2.1 was already shipworthy by month 3. I held it back to ship v2.2 with style tokens as a "strict superset." v2.2 was retracted. Months later, v2.3 finally shipped. Net loss: ~2 months of public visibility.

In hindsight: if a version hits your eval gate, release it before you start the next ambitious experiment. The release process surfaces real-world issues (mobile UX, install paths, deployment quirks) that you want to discover with a known-good baseline, not while you're also debugging a new training run.

The general principle: avoid combinatorial risk. "Add a new feature while also retraining" doubles your failure surface. If you must do both, do them in separate releases.

4. The first thing you cut is usually the most distinctive feature

I cut style tokens. They were literally the "Control" in ControlMT. Three months of architecture, corpus prep, training, eval — and they didn't work as advertised at this scale.

This wasn't a freak event. It happens because the distinctive feature is the one you didn't have prior data for. The well-understood baseline features (just translate KN↔EN) had decades of prior research. The distinctive feature (style-controlled translation at small scale) had no prior art to predict the failure mode from.

Brace for this. Have an exit strategy. When I realized style tokens weren't working, the recovery path was clean (v2.3 = same weights, force NATURAL) because the architecture had been written to keep the control signal optional. If I'd hard-coded style routing into every forward pass, the cleanup would have taken weeks.

The corollary: don't put the distinctive feature on the critical path. Make it pluggable. Make it ablate-able. Make it cheap to remove if it doesn't pay off.

5. Anti-LM contrastive decoding is the highest-ROI single trick I used

Two papers (Li et al. 2022 Contrastive Decoding, Yang et al. 2024 Anti-LM Decoding), ~30 lines of code, 2× decoder compute at inference. Result on our 100-pair curated code-mix eval: +5 BLEU on KN→EN, +1 BLEU on EN→KN. No retraining required.

Sketch:

logits_main = decoder(decoder_inputs, encoder_states, normal_mask)
logits_anti = decoder(decoder_inputs, encoder_states, zero_mask)  # masks out the encoder
final_log_p = log_softmax(logits_main) - 0.5 * log_softmax(logits_anti)

The mechanism: logits_anti is what the decoder would say based only on its language model prior. Subtract a fraction of that from the main distribution and you penalize tokens the model "would have said anyway," biasing toward tokens that genuinely require the source context.

Inference-time tricks like this are vastly underused in the small-specialized-model space. Most public Indic MT models don't bother. If you're building a small specialized model, hunt for these. They have a much better ROI than scaling parameters.

6. Specialized small models can match multilingual generalists on their one pair

The hypothesis I started with — that a focused 139M model can match a 1.1B 22-language model on Kannada↔English — broadly survived. ControlMT v2.3 doesn't beat IndicTrans2 1.1B on every sentence, but on FLORES KN↔EN COMET-DA it's within 0.01-0.03 points despite being 8× smaller.

The mechanism: the 1.1B multilingual model spreads its capacity across 22 languages. Most of those parameters aren't doing useful work for KN↔EN specifically. A focused 139M model with all parameters dedicated to one pair can recover most of the quality.

Implication for picking a project: if you're building for one Indian language (or one language pair), don't try to compete with NLLB or IndicTrans2 on their territory. Build the specialized model and you can be competitive with 8-30× less parameters. If you're building a multilingual model... well, do you have $10M of training compute? If not, the math doesn't work in your favor.

7. CometKiwi-DA for quality filtering, BLEU only for headlines

I scored 6.7M parallel pairs with CometKiwi-DA before any training and removed the bottom ~1% (kiwi < 0.50). That single filter likely improved the model more than any architectural decision after v2.1.

Why CometKiwi specifically:

Reference-free (you don't need a "correct" translation to score)
Semantic (catches bad pairs that BLEU's surface-token matching misses)
Fast (~30 hours on RTX 5060 Ti for 6.7M pairs — long but tractable on one GPU)
Open-weight (Unbabel/wmt22-cometkiwi-da on HuggingFace, Apache 2.0)

BLEU and chrF are still useful for reporting because they're conventional and let readers compare to literature. But for filtering and quality decisions during development, BLEU is noise. Use a learned metric.

Caveat: a learned metric can be gamed. If you train your model and then evaluate with the same metric you used to filter your training data, you're optimistic. Hold out some data, score it with both CometKiwi and a separate metric (COMET-DA, which uses references; or human eval on 50 samples), and check that they agree.

8. The release surface area is bigger than the research surface

I estimated 10% of project time for "deployment + docs." It was 15-20%, even with an AI assistant doing first drafts of most of the docs.

Things in the release surface area that aren't part of "training a model":

Model card (~3 hours to write honestly without overselling)
HuggingFace trust_remote_code integration (configuration / modeling / tokenization wrappers — ~6 hours)
Demo Space (Gradio first, then switched to Docker FastAPI — total ~8 hours including the mobile-UX debugging)
PyPI SDK (~4 hours including tests and verification suite)
Quantized sibling repo (~2 hours)
Deployment recipes (DEPLOYMENT.md — verified each one end-to-end, ~3 hours)
Competitor benchmark (~2 hours including downloads)
Privacy doc + opt-in logging pipeline (~2 hours)
This documentation folder (~4 hours)

Total: ~34 hours, or about a full work week of focused effort. Don't be surprised when "the model is trained, now I just need to publish it" is another week.

9. AI assistant as collaborator, with persistent memory

Specific to my workflow: I built ControlMT with Claude as an AI assistant throughout. The 3-month timeline is roughly 2x faster than I would have managed alone — but the speedup is uneven:

Modeling code: ~1.5× speedup (assistant occasionally suggests subtly-wrong architectural changes, so I review carefully)
Deployment + docs: ~5× speedup (assistant writes excellent first drafts, I edit)
Debugging: ~3× speedup (assistant can read stack traces faster than me)
Architecture decisions: 1× speedup (these stayed human)

The thing that makes this work at multi-month timescales is persistent memory. Without it, every new session restarts from zero context. With it, the assistant remembers our decisions, our conventions, our prior failures, and applies them automatically.

If you're going to use an AI assistant for a multi-week project, set up persistent memory first. Otherwise you'll repeat the same "we use bf16, not fp16" conversation forty times. (See working-with-claude.md for the specific patterns I converged on.)

10. Honest documentation is competitive moat

The deployment doc has a section called "Not directly supported" that names — by framework — every popular LLM-serving tool ControlMT can't use (vLLM, Ollama, GGUF/llama.cpp, HF TGI, bitsandbytes int8), with the architectural reason for each. That section took 30 minutes to write.

It saves users hours of failed integration attempts. It tells the truth about the model. And in a market where every other project claims to "support all the major frameworks" (most of which they haven't actually tested), being honest about what doesn't work is a differentiator.

Same applies to the competitor benchmark. I didn't curate the benchmark to make ControlMT look best. I included sentences where ControlMT loses to IndicTrans2 and to Sarvam-Translate, alongside cases where it wins. The honest scorecard is more useful to a deployer than a marketing-doctored one. It also means when ControlMT does win, the win is credible.

The general principle: users are smart and notice overselling. A model card that says "best Kannada translator!" gets ignored. A model card that says "competitive with IndicTrans2 at 8× smaller size on most cases; loses on multi-token code-mix" gets bookmarked. The latter is more useful, more honest, and ultimately more persuasive.

What I'd do differently next time

If I were starting v2.4 (or a totally new specialized-model project) tomorrow:

Week 1: lock the niche, write the README first (just the pitch, ~500 words), write the smoke-test before writing the training loop
Weeks 2-6: data prep — scoring, filtering, alignment, specialized streams. Don't touch the model code. Just data.
Week 7: smoke train (1% data, 2 epochs) to validate the pipeline
Weeks 8-10: full training. Multiple checkpoints, EMA + SWA, per-checkpoint eval logs
Weeks 11-12: ship. Model card, Space, SDK, deployment recipes. Treat this as a full sprint, not an afterthought.
Week 13 onward: real-world usage feedback drives v2.5 priorities. Build the benchmark before you need it.

The thing I'd most change vs my actual ControlMT timeline: release earlier and iterate publicly. v2.1 sat private for 2 months while v2.2 failed. If v2.1 had shipped at the end of week 8 instead, every subsequent iteration would have had real-world usage feedback.

The thing I'd keep exactly the same: niche-lock on day one. Nothing else matters until that's settled.