license: apache-2.0
library_name: transformers
pipeline_tag: translation
language:
- ko
- en
- vi
DMTLLM Translation Research
Research Release
DMTLLM Translation Research is an open-weight multilingual language model developed by DMTLabs as part of the foundational research for DMTLLM.
The model is a compact ~50M parameter decoder-only Transformer based on a Llama-style architecture. It was trained from scratch with randomly initialized weights using Korean, English, and Vietnamese monolingual corpora.
No pretrained Llama model weights were used.
Following base language model pretraining, the model was further specialized for machine translation through full-parameter supervised fine-tuning (SFT) using Korean–English and Korean–Vietnamese parallel translation data.
This release represents an intermediate research milestone in the development of DMTLLM and is intended primarily for research and experimentation.
Model Overview
| Item | Description |
|---|---|
| Developer | DMTLabs |
| Model size | ~50M parameters |
| Model type | Decoder-only Transformer |
| Implementation | LlamaForCausalLM |
| Architecture | Llama-style Causal Language Model |
| Initialization | Random initialization |
| Base pretraining | Causal Language Modeling |
| Pretraining languages | Korean, English, Vietnamese |
| Post-training | Full-parameter Supervised Fine-Tuning |
| Translation data | Korean–English, Korean–Vietnamese |
| Specialization | Machine Translation |
| Model status | Research / Experimental |
| License | Apache License 2.0 |
Although the model implementation is based on the
LlamaForCausalLMarchitecture provided by Hugging Face Transformers, no pretrained Llama weights were used. All model parameters were initialized randomly and trained from scratch.
Model Architecture
The model uses a compact Llama-style decoder-only Transformer architecture designed at approximately the 50M parameter scale.
| Specification | Value |
|---|---|
| Parameters | ~50M |
| Vocabulary size | 48,000 |
| Hidden size | 512 |
| Transformer layers | 8 |
| Attention heads | 8 |
| Key-value heads | 4 |
| Attention type | Grouped Query Attention (GQA) |
| Attention head dimension | 64 |
| Intermediate size | 1,376 |
| Maximum context length | 1,024 tokens |
| Positional encoding | Rotary Position Embeddings (RoPE) |
| RoPE theta | 10,000 |
| Normalization | RMSNorm |
| RMSNorm epsilon | 1e-5 |
| Activation | SiLU / SwiGLU-style |
| Attention bias | No |
| MLP bias | No |
| Attention dropout | 0.0 |
| Input/output embeddings | Tied |
| Checkpoint parameter dtype | FP32 |
The architecture includes:
- Rotary Position Embeddings (RoPE)
- RMSNorm
- Grouped Query Attention (GQA)
- SwiGLU-style feed-forward layers
- tied input and output embeddings
- decoder-only causal self-attention
Training Pipeline
The model was developed in two stages:
Random Initialization
│
▼
Base Language Model Pretraining
(Korean / English / Vietnamese)
│
▼
Scratch-pretrained Base Model
│
▼
Full-parameter Translation SFT
(Korean–English / Korean–Vietnamese)
│
▼
Translation-specialized Research Model
Stage 1: Base Pretraining
The base language model was trained entirely from scratch.
No pretrained language model checkpoint was used. The model parameters were randomly initialized and optimized using the standard causal language modeling objective.
Conceptually, the model learns next-token prediction:
token₁ token₂ token₃ ... tokenₙ
↓ ↓ ↓ ↓
token₂ token₃ token₄ ... tokenₙ₊₁
The pretraining corpus consists of monolingual Korean, English, and Vietnamese text.
Training utilizes:
- Causal Language Modeling / Next Token Prediction
- BF16 mixed-precision computation
- AdamW optimization
- gradient accumulation
- gradient clipping
- linear learning-rate warmup
- cosine learning-rate decay
- optional gradient checkpointing
Base Pretraining Data
The monolingual corpus contains approximately 27.6 million training records after preprocessing and filtering.
| Language | Training | Validation | Test |
|---|---|---|---|
| Korean | 13,738,080 | 140,019 | 140,843 |
| English | 7,723,537 | 79,102 | 79,150 |
| Vietnamese | 6,169,183 | 62,694 | 62,914 |
| Total | 27,630,800 | 281,815 | 282,907 |
The records were tokenized and packed into fixed-length sequences for causal language model pretraining.
Monolingual Data Filtering
Duplicate and invalid samples were removed during preprocessing.
| Language | Removed Duplicates | Other Filtered Samples |
|---|---|---|
| Korean | 623,012 | 79 |
| English | 191,233 | 4 |
| Vietnamese | 274,093 | 123 |
| Total | 1,088,338 | 206 |
Stage 2: Translation Supervised Fine-Tuning
Following base pretraining, the entire model was further optimized for machine translation using full-parameter supervised fine-tuning.
No LoRA, adapter, or other parameter-efficient fine-tuning method was used.
The SFT corpus consists of Korean–English and Korean–Vietnamese parallel translation pairs.
Each training example contains:
[ Translation Prompt ] [ Target Translation ]
During SFT, the prompt portion is excluded from the language modeling loss.
[ Translation Prompt ] [ Target Translation ]
masked loss
Only target translation tokens contribute to the training objective, while all model parameters are updated.
Translation SFT Data
Approximately 14.1 million parallel sentence pairs were used for translation SFT.
| Language Pair | Training | Validation | Test |
|---|---|---|---|
| Korean–English | 7,907,787 | 80,562 | 81,229 |
| Korean–Vietnamese | 6,200,497 | 63,509 | 63,319 |
| Total | 14,108,284 | 144,071 | 144,548 |
Parallel Data Filtering
Parallel data preprocessing included duplicate removal, source–target length-ratio filtering, text validation, and identical-pair filtering where applicable.
| Language Pair | Raw Pairs | Duplicates | Ratio Filter | Text Filter | Identical Pair Filter |
|---|---|---|---|---|---|
| Korean–English | 8,073,026 | 1,636 | 1,769 | 43 | — |
| Korean–Vietnamese | 6,569,007 | 238,995 | 1,343 | 154 | 1,190 |
The remaining examples were divided into training, validation, and test sets.
Research Objectives
This model was developed as part of the foundational research for the DMTLLM project.
The primary objectives of this work are to investigate:
- training a decoder-only language model entirely from scratch;
- building an independent base model without relying on pretrained model weights;
- learning multilingual representations from Korean, English, and Vietnamese monolingual corpora;
- adapting a scratch-pretrained language model to machine translation;
- evaluating full-parameter SFT for translation specialization;
- investigating Korean–English and Korean–Vietnamese translation using a unified decoder-only architecture;
- establishing a reproducible research foundation for future DMTLLM models.
This model should therefore be considered a research artifact rather than a production-ready DMTLLM release.
Languages
The base pretraining stage includes:
- Korean (
ko) - English (
en) - Vietnamese (
vi)
The translation specialization stage uses parallel data for:
- Korean–English
- Korean–Vietnamese
Translation capabilities may vary depending on translation direction, domain, sentence complexity, and prompt format.
Evaluation
Comprehensive quantitative evaluation results are being prepared.
Planned evaluation includes:
- translation quality evaluation;
- language-pair-specific performance analysis;
- comparison between the scratch-pretrained base model and the translation-SFT model;
- analysis of translation specialization after full-parameter SFT;
- qualitative analysis of generated translations.
Evaluation results will be added in future updates.
Intended Use
This model is primarily intended for:
- machine translation research;
- multilingual language model research;
- research on language models trained from scratch;
- experiments involving translation-oriented supervised fine-tuning;
- decoder-only Transformer translation experiments;
- Korean–English and Korean–Vietnamese translation research;
- foundational research for future DMTLLM models.
Limitations
This is an experimental research model.
The model may generate:
- inaccurate translations;
- incomplete translations;
- hallucinated content;
- unexpected outputs;
- outputs that differ depending on prompt format or decoding configuration.
The model has not yet undergone comprehensive evaluation across languages, domains, safety scenarios, or production environments.
Its relatively compact model scale may also limit linguistic knowledge, reasoning ability, contextual understanding, and translation quality compared with substantially larger language models.
Performance may vary depending on:
- language pair;
- translation direction;
- input domain;
- sentence length;
- prompt format;
- context length;
- decoding parameters.
This model should not be used for safety-critical or other high-stakes applications without additional evaluation and validation.
License
The released model weights and accompanying materials are provided under the Apache License 2.0.