README.md

---
license: cc-by-sa-4.0
language:
- ja
library_name: transformers
pipeline_tag: text-generation
tags:
- romaji
- japanese
- ime
- romaji-to-japanese
- transduction
- qwen3
---

# romaji2ja

Romaji → Japanese conversion (ローマ字→日本語変換). A compact Qwen3-style causal
LM plus a **hybrid IME-style fast path** that resolves most inputs from
dictionaries/segmentation in well under a millisecond and only falls back to the
neural model for genuinely novel input.

> Code in the companion GitHub repository is MIT-licensed. Model weights and
> model artifacts are published under CC BY-SA 4.0 because the training corpus
> is derived from Japanese Wikipedia.

## What this is

- **Task**: transduce romaji (incl. wapuro/kunrei/Hepburn, typos, IME small-tsu,
  case/space/hyphen noise) into Japanese (kanji/kana).
- **Model**: `Qwen3ForCausalLM`, hidden 1024 · 20 layers · 16 heads / 4 KV (GQA)
  · intermediate 3584 · vocab 8000 · context 1024 · tied embeddings (~0.3B,
  fp32). Trained on rule-generated romaji↔Japanese pairs from a Japanese corpus.
- **Published checkpoint**: accepted A75 interpolation checkpoint, SHA-256
  `97cd23ebb00582d461c04c0e8ed3ebd140395b298aa80f98f73cdfb5aa67b332`.
- **Special tokens**: `U+EE00` romaji-input start · `U+EE01` Japanese-output
  start · `U+EE02` reserved context · `<|endoftext|>` EOS/PAD. The model is
  prompted as `U+EE00 {romaji} U+EE01` and generates the Japanese continuation.

## Hybrid fast path (recommended for product use)

`infer_fast.py` routes each input through, in order:

1. input normalization (NFKC, lowercase, strip spaces / soft separators / zero-width)
2. learned choice feedback → exact lexicon → SQLite cache
3. exact lexicon segmentation
4. generic romaji/kana phrase fallback
5. whole-input fuzzy + fuzzy/weighted/dense segmentation routes
6. **general-phrase rescue** — a generic noisy-romaji recovery stage:
   bidirectional romaji *style* canonicalization (wapuro `syudan` ↔ Hepburn
   `shudan` both match the dictionary), IME small-tsu (`moxtute`→もって) and
   run-collapse (`syyyuu`→syuu) canonicalization, then an anchor-and-fill beam
   Viterbi over a general reading lexicon + reusable colloquial layer with a
   confidence gate. It runs *only* when every earlier route declines and
   strictly before the neural model, abstaining (→ neural model) on
   low-confidence / ambiguous input rather than emitting a wrong answer.
7. neural model fallback

## Verified accuracy (acceptance gate)

Practical Windows hybrid fast path — `python src/check_acceptance_98.py`,
**12/12 required gates pass**, evidence SHA-256 verified:

| Gate | Result |
|---|---:|
| fixed gold | 99/99 (100%) |
| typo stress | 1980/1980 (100%) |
| realworld noise | 594/594 (100%) |
| expanded realworld case/space/hyphen | 1188/1188 (100%) |
| full-v1 expanded case/space/hyphen | 9432/9432 (100%) |
| large realworld case/space/hyphen (95 gate) | 5675/5675 (100%) |
| large full-v1 case/space/hyphen (95 gate) | 45598/45598 (100%) |
| long composition | 800/800 (100%) |
| fresh case/space/hyphen | 2358/2358 (100%) |
| strict neural full-v1 (normalized) | 771/786 (98.09%) |
| strict neural fresh case/space/hyphen (normalized) | 2313/2358 (98.09%) |
| post-shared-normalization fresh rerun | 2358/2358 (100%) |

**Caveat (honest boundary)**: the 98% claim is for the practical hybrid path and
for *normalized* strict-neural input. Raw, un-normalized strict-neural
case/space/hyphen input is **not** a 95% claim (measured ~93.4%). Product input
should go through the hybrid path, which normalizes before routing. Full evidence
and per-file hashes are in `acceptance/summary.json`.

## Usage

Neural model directly (normalized input recommended):

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("limoXD/romaji2ja")
model = AutoModelForCausalLM.from_pretrained("limoXD/romaji2ja")

bos_in, bos_out = chr(0xEE00), chr(0xEE01)   # romaji-start, japanese-start
prompt = bos_in + "kyouhaiitenkidesu" + bos_out
ids = tok(prompt, return_tensors="pt", add_special_tokens=False)
ids.pop("token_type_ids", None)  # Qwen-style model does not consume this field
out = model.generate(**ids, max_new_tokens=64, do_sample=False)
print(tok.decode(out[0][ids.input_ids.shape[1]:], skip_special_tokens=True))
# 今日はいい天気です
```

Hybrid fast path (clone the repo for `code/` + `lexicon/`):

```bash
python code/infer_fast.py --model . --lexicon lexicon/romaji2ja_typo_95.json \
  --general-lexicon lexicon/general_reading_lexicon.json "syudan no hani nara"
```

## Repository layout

```
config.json, model.safetensors, tokenizer.json, ...   # the model (load at repo root)
code/        infer_fast.py, general_phrase.py, romaji_kana.py, normalization.py, infer.py, eval_general_phrase.py
lexicon/     romaji2ja*.json, general_reading_lexicon.json, candidate_feedback.jsonl, adversarial_piece_aliases.json
acceptance/  summary.json, gate_report.json   # 12/12 gate evidence with SHA-256 manifest
release_manifest.json                         # exact file hashes and checkpoint identity
```

Companion projects: [source code](https://github.com/dj-thank/romaji2ja) ·
[interactive Space](https://huggingface.co/spaces/limoXD/romaji2ja).

## Limitations

- Tuned for Windows-local use (CPU / DirectML); fp32 weights.
- Severe multi-error corruption or genuine homophone ambiguity is intentionally
  left to the neural fallback (or abstained) rather than guessed.
- Context window 1024 tokens.