README.md

---
license: apache-2.0
library_name: transformers
pipeline_tag: translation
language:
  - ko
  - en
  - vi
---

# DMTLLM Translation Research

> **Research Release**

**DMTLLM Translation Research** is an open-weight multilingual language model developed by **DMTLabs** as part of the foundational research for DMTLLM.

The model is a compact **~50M parameter decoder-only Transformer** based on a Llama-style architecture. It was **trained from scratch with randomly initialized weights** using Korean, English, and Vietnamese monolingual corpora.

No pretrained Llama model weights were used.

Following base language model pretraining, the model was further specialized for machine translation through **full-parameter supervised fine-tuning (SFT)** using Korean–English and Korean–Vietnamese parallel translation data.

This release represents an intermediate research milestone in the development of DMTLLM and is intended primarily for research and experimentation.

---

## Model Overview

| Item | Description |
|---|---|
| Developer | DMTLabs |
| Model size | **~50M parameters** |
| Model type | Decoder-only Transformer |
| Implementation | `LlamaForCausalLM` |
| Architecture | Llama-style Causal Language Model |
| Initialization | Random initialization |
| Base pretraining | Causal Language Modeling |
| Pretraining languages | Korean, English, Vietnamese |
| Post-training | Full-parameter Supervised Fine-Tuning |
| Translation data | Korean–English, Korean–Vietnamese |
| Specialization | Machine Translation |
| Model status | Research / Experimental |
| License | Apache License 2.0 |

> Although the model implementation is based on the `LlamaForCausalLM` architecture provided by Hugging Face Transformers, **no pretrained Llama weights were used**. All model parameters were initialized randomly and trained from scratch.

---

## Model Architecture

The model uses a compact Llama-style decoder-only Transformer architecture designed at approximately the **50M parameter scale**.

| Specification | Value |
|---|---:|
| Parameters | ~50M |
| Vocabulary size | 48,000 |
| Hidden size | 512 |
| Transformer layers | 8 |
| Attention heads | 8 |
| Key-value heads | 4 |
| Attention type | Grouped Query Attention (GQA) |
| Attention head dimension | 64 |
| Intermediate size | 1,376 |
| Maximum context length | 1,024 tokens |
| Positional encoding | Rotary Position Embeddings (RoPE) |
| RoPE theta | 10,000 |
| Normalization | RMSNorm |
| RMSNorm epsilon | 1e-5 |
| Activation | SiLU / SwiGLU-style |
| Attention bias | No |
| MLP bias | No |
| Attention dropout | 0.0 |
| Input/output embeddings | Tied |
| Checkpoint parameter dtype | FP32 |

The architecture includes:

- Rotary Position Embeddings (RoPE)
- RMSNorm
- Grouped Query Attention (GQA)
- SwiGLU-style feed-forward layers
- tied input and output embeddings
- decoder-only causal self-attention

---

## Training Pipeline

The model was developed in two stages:

```text
Random Initialization
        │
        ▼
Base Language Model Pretraining
(Korean / English / Vietnamese)
        │
        ▼
Scratch-pretrained Base Model
        │
        ▼
Full-parameter Translation SFT
(Korean–English / Korean–Vietnamese)
        │
        ▼
Translation-specialized Research Model
```

### Stage 1: Base Pretraining

The base language model was trained entirely from scratch.

No pretrained language model checkpoint was used. The model parameters were randomly initialized and optimized using the standard causal language modeling objective.

Conceptually, the model learns next-token prediction:

```text
token₁ token₂ token₃ ... tokenₙ
   ↓      ↓      ↓           ↓
token₂ token₃ token₄ ... tokenₙ₊₁
```

The pretraining corpus consists of monolingual Korean, English, and Vietnamese text.

Training utilizes:

- Causal Language Modeling / Next Token Prediction
- BF16 mixed-precision computation
- AdamW optimization
- gradient accumulation
- gradient clipping
- linear learning-rate warmup
- cosine learning-rate decay
- optional gradient checkpointing

### Base Pretraining Data

The monolingual corpus contains approximately **27.6 million training records** after preprocessing and filtering.

| Language | Training | Validation | Test |
|---|---:|---:|---:|
| Korean | 13,738,080 | 140,019 | 140,843 |
| English | 7,723,537 | 79,102 | 79,150 |
| Vietnamese | 6,169,183 | 62,694 | 62,914 |
| **Total** | **27,630,800** | **281,815** | **282,907** |

The records were tokenized and packed into fixed-length sequences for causal language model pretraining.

### Monolingual Data Filtering

Duplicate and invalid samples were removed during preprocessing.

| Language | Removed Duplicates | Other Filtered Samples |
|---|---:|---:|
| Korean | 623,012 | 79 |
| English | 191,233 | 4 |
| Vietnamese | 274,093 | 123 |
| **Total** | **1,088,338** | **206** |

---

## Stage 2: Translation Supervised Fine-Tuning

Following base pretraining, the entire model was further optimized for machine translation using **full-parameter supervised fine-tuning**.

No LoRA, adapter, or other parameter-efficient fine-tuning method was used.

The SFT corpus consists of Korean–English and Korean–Vietnamese parallel translation pairs.

Each training example contains:

```text
[ Translation Prompt ] [ Target Translation ]
```

During SFT, the prompt portion is excluded from the language modeling loss.

```text
[ Translation Prompt ] [ Target Translation ]
        masked               loss
```

Only target translation tokens contribute to the training objective, while all model parameters are updated.

### Translation SFT Data

Approximately **14.1 million parallel sentence pairs** were used for translation SFT.

| Language Pair | Training | Validation | Test |
|---|---:|---:|---:|
| Korean–English | 7,907,787 | 80,562 | 81,229 |
| Korean–Vietnamese | 6,200,497 | 63,509 | 63,319 |
| **Total** | **14,108,284** | **144,071** | **144,548** |

### Parallel Data Filtering

Parallel data preprocessing included duplicate removal, source–target length-ratio filtering, text validation, and identical-pair filtering where applicable.

| Language Pair | Raw Pairs | Duplicates | Ratio Filter | Text Filter | Identical Pair Filter |
|---|---:|---:|---:|---:|---:|
| Korean–English | 8,073,026 | 1,636 | 1,769 | 43 | — |
| Korean–Vietnamese | 6,569,007 | 238,995 | 1,343 | 154 | 1,190 |

The remaining examples were divided into training, validation, and test sets.

---

## Research Objectives

This model was developed as part of the foundational research for the **DMTLLM** project.

The primary objectives of this work are to investigate:

- training a decoder-only language model entirely from scratch;
- building an independent base model without relying on pretrained model weights;
- learning multilingual representations from Korean, English, and Vietnamese monolingual corpora;
- adapting a scratch-pretrained language model to machine translation;
- evaluating full-parameter SFT for translation specialization;
- investigating Korean–English and Korean–Vietnamese translation using a unified decoder-only architecture;
- establishing a reproducible research foundation for future DMTLLM models.

This model should therefore be considered a **research artifact rather than a production-ready DMTLLM release**.

---

## Languages

The base pretraining stage includes:

- Korean (`ko`)
- English (`en`)
- Vietnamese (`vi`)

The translation specialization stage uses parallel data for:

- Korean–English
- Korean–Vietnamese

Translation capabilities may vary depending on translation direction, domain, sentence complexity, and prompt format.

---

## Evaluation

Comprehensive quantitative evaluation results are being prepared.

Planned evaluation includes:

- translation quality evaluation;
- language-pair-specific performance analysis;
- comparison between the scratch-pretrained base model and the translation-SFT model;
- analysis of translation specialization after full-parameter SFT;
- qualitative analysis of generated translations.

Evaluation results will be added in future updates.

---

## Intended Use

This model is primarily intended for:

- machine translation research;
- multilingual language model research;
- research on language models trained from scratch;
- experiments involving translation-oriented supervised fine-tuning;
- decoder-only Transformer translation experiments;
- Korean–English and Korean–Vietnamese translation research;
- foundational research for future DMTLLM models.

---

## Limitations

This is an experimental research model.

The model may generate:

- inaccurate translations;
- incomplete translations;
- hallucinated content;
- unexpected outputs;
- outputs that differ depending on prompt format or decoding configuration.

The model has not yet undergone comprehensive evaluation across languages, domains, safety scenarios, or production environments.

Its relatively compact model scale may also limit linguistic knowledge, reasoning ability, contextual understanding, and translation quality compared with substantially larger language models.

Performance may vary depending on:

- language pair;
- translation direction;
- input domain;
- sentence length;
- prompt format;
- context length;
- decoding parameters.

This model should not be used for safety-critical or other high-stakes applications without additional evaluation and validation.

---

## License

The released model weights and accompanying materials are provided under the **Apache License 2.0**.