sion_translate v1 Korean–Japanese Translation

KJ-X is a custom PyTorch encoder-decoder model for bidirectional Korean↔Japanese translation. The uploaded release contains inference exports and its SentencePiece tokenizer; training datasets and training checkpoints are not distributed.

v1 numeric inference update — runtime 0.1.2

Existing v1 users: reinstall the updated inference code. Existing verified v1 weights and tokenizer can be reused; no retraining or weight replacement is needed. The follow-up preserves spaced signs, repairs recognized measurement-unit suffixes (including compound units and powers), protects previously unrecognized numeric suffixes, and keeps ordinary uppercase acronyms visible during translation.

Actual v1 EMA execution now preserves ± 0.05, 250mL, 3,850km², 15mile, and 40mile in the previously failing cases. The five identified acronym regressions are also corrected. This remains a partial inference correction: arbitrary units, word-written numbers, arithmetic, and full numerical semantics are not guaranteed. Some numerical omissions still cause an error instead of a translation.

The focused follow-up attempted 238 inputs, returned 236 outputs, and blocked 2. It includes 72 authored numeric stress inputs, 16 existing diagnostics, and every affected FLORES input. Average chrF++ on the rerun no-digit subset increased in both directions; individual regressions are still possible. This was not a fresh full FLORES rerun. Related regression tests: 98 passed.

한국어 안내: 부호·단위·숫자 인식 누락·약어 회귀를 재학습 없이 보완했습니다. 기존 사용자는 추론 코드를 다시 받으세요. 기존 가중치는 재사용 가능합니다. 숫자 처리가 완벽하다는 의미는 아니며, 일부 값 누락은 오류로 차단됩니다.

Runtime commit: 45254b8cb84e.

See details and evidence.

Files

• model_ema.pt: recommended FP32 EMA inference export
• model_int8.pt: legacy pickled CPU-only dynamic INT8 export, kept for compatibility; prefer portable/model_int8_state_dict.pt
• portable/: FP32 safetensors and a plain INT8 state_dict with a SHA-256 manifest (see Portable artifacts)
• kjx.model, kjx.vocab, token_features.npz: tokenizer files

The model uses custom code from gaon12/sion_translate and is not a Transformers AutoModel checkpoint.

Quick start

python -m pip install --upgrade "sion-translate @ git+https://github.com/gaon12/sion_translate.git@45254b8cb84e7725aa7b58709f060317ff4e2ab6"
python -m pip install huggingface_hub
hf download gaon12/sion_translate \
  model_ema.pt kjx.model kjx.vocab token_features.npz inference_v1.py v1_numeric_hotfix.json \
  --local-dir models/sion_translate

from sion_translate.inference import Translator

translator = Translator(
    "models/sion_translate/model_ema.pt",
    "models/sion_translate/kjx.model",
    device="cuda",  # Use "cpu" when CUDA is unavailable.
)
print(translator.translate(["안녕하세요."], target_language="ja", num_beams=4)[0])
print(translator.translate(["こんにちは。"], target_language="ko", num_beams=4)[0])

Translator automatically finds token_features.npz beside the model/tokenizer. The model is custom code, so transformers.AutoModel and hosted Transformers inference are not supported.

The numeric update was verified with model_ema.pt on CPU. Use that EMA file for the safe weights-only path; the legacy pickled INT8 download is not covered by this validation.

You can also run python models/sion_translate/inference_v1.py --text "17시 30분" --target-language ja. The helper verifies the downloaded EMA, tokenizer, and feature-file hashes.

Reproducible smoke test

After downloading the files above, run this from a clean environment to exercise both directions:

python - <<'PY'
from sion_translate.inference import Translator

translator = Translator("models/sion_translate/model_ema.pt", "models/sion_translate/kjx.model")
for text, target in [("안녕하세요.", "ja"), ("こんにちは。", "ko")]:
    output = translator.translate([text], target_language=target, num_beams=1)[0]
    assert isinstance(output, str) and output.strip(), output
    print(f"{text} -> {output}")
PY

Portable artifacts (portable/)

portable/ holds plain-tensor copies of model_ema.pt that load with strict=True and without executing pickled classes. sion_portable.py checks the size and SHA-256 of every file listed in portable_manifest.json before it builds the model.

File	Contents	Size	SHA-256
model_fp32.safetensors	FP32 safetensors; every tensor bitwise-equal to model_ema.pt	0.80 GB	94d5fe4a6834c943aae2c98e64c716286f555cfa6151c3e182e8571e8168adb4
model_int8_state_dict.pt	CPU dynamic INT8 state_dict (torch.ao); loads with torch.load(weights_only=True), no pickled classes	0.31 GB	1757e67bb4ed611894d35fe15a96cf2b3f76f6b486a0e2b46eb3b82e8a1f0b20
kjx.model	Tokenizer	853 KB	c1b5447e08692bf7a7d2e485ba7b71b3ff5584c96844ecac07e25c9d1340db50
token_features.npz	Tokenizer features	69 KB	54951b569255bd0a2e1d333860db2294113f1368a2e25e4ae5bc2bcdb6976b4c
portable_manifest.json	Hashes, source checkpoint, runtime commit, architecture, quantization layout	12 KB	29044d46d94344b5eb4266f4550b2ce5749b24300a97930d4371e5dea3a86020

sion_portable.py (loader), translate_portable.py (CLI), and portable_validation.json (per-sentence outputs) are in the same folder.

• Source: gaon12/sion_translate revision 1f106d9caa22, model_ema.pt (SHA-256 829580d1db1693ec00ee641398ce6ba9549388e266f6259f85e5a19ea28bfc2e)
• Runtime: sion-translate commit 45254b8cb84e
• INT8 recipe: dynamic qint8 weights (per output channel) for 200 encoder/decoder nn.Linear modules, onednn engine, no calibration data, CPU only

python -m pip install "sion-translate @ git+https://github.com/gaon12/sion_translate.git@45254b8cb84e7725aa7b58709f060317ff4e2ab6" safetensors huggingface_hub
hf download gaon12/sion_translate --include "portable/*" --local-dir sion_translate
python sion_translate/portable/translate_portable.py --precision fp32 --target-language ja --text "회의는 17시 30분에 시작합니다."

import sys
sys.path.insert(0, "sion_translate/portable")
import sion_portable
from sion_translate.inference import Translator

sion_portable.install()  # Translator now accepts portable weight files
translator = Translator("sion_translate/portable/model_fp32.safetensors", "sion_translate/portable/kjx.model", device="cpu")

Validation (40-sentence diagnostic set, CPU, beam 4, torch 2.13.0+cpu):

Weights	Identical output	Blocked by numeric guard	chrF
model_ema.pt (source)	—	0	59.13
model_fp32.safetensors	40/40 vs source	0	59.13
model_int8_state_dict.pt	28/40 vs FP32	2	57.57

FP32 is the reference portable format. INT8 trades a small quality loss for size and CPU speed; prefer FP32 when numbers must be preserved. The INT8 file uses PyTorch's quantized tensor types, which PyTorch has deprecated; it was verified with torch 2.13.0+cpu only.

Evaluation

The GitHub repository includes a small, independently authored diagnostic JSONL covering both directions and provides a common scorer for KJ-X, LibreTranslate, Papago, Google Translation, DeepL, M2M100 418M, and NLLB-200. No universal quality claim is made from that small set.

Limitations

• Machine translations can omit, add, or reverse meaning.
• Proper nouns, short context-dependent utterances, specialist terminology, and long inputs need human review.
• Do not use unreviewed output for medical, legal, safety-critical, or certified translation.
• The custom architecture requires the repository code; hosted Transformers inference is not available out of the box.

AI-Hub dataset attribution for public v1

The AI-Hub portion of the public v1 (KJ-X) training data includes the datasets below. We acknowledge AI-Hub (aihub.or.kr) and the dataset creators. This table credits those AI-Hub sources; it is not a complete inventory of all training sources.

AI-Hub dataset ID	Official dataset name and source
127	한국어-일본어 번역 말뭉치
546	일상생활 및 구어체 한-중, 한-일 번역 병렬 말뭉치 데이터
71493	한국어-다국어(영어 제외) 번역 말뭉치(기술과학)
71496	한국어-다국어 번역 말뭉치(기초과학)
71498	한국어-다국어 번역 말뭉치 (인문학)
71646	생성형AI 다국어 번역 품질 평가 데이터
71735	생성형AI 한-중/한-일 특허 및 기술과학 분야 병렬 말뭉치 데이터
71782	생성형AI 금융 분야 다국어 병렬 말뭉치 데이터
71591	방송콘텐츠 한국어-아시아어 번역 말뭉치
71671	생성형AI 라이브 스트리밍 영상 일본어 통번역 데이터

AI-Hub's official FAQ permits commercial use and distribution of models developed using its data, with attribution to the dataset names and AI-Hub. Redistribution of original or merely edited source data is restricted. These terms apply to the AI-Hub sources listed here.

License

The original KJ-X code and uploaded model release are provided under the MIT License. This does not grant rights to user-provided inputs, generated service outputs, third-party software, or third-party models used only as comparison baselines.