Update README.md

README.md

@@ -1,3 +1,307 @@
 ---
 license: apache-2.0
+library_name: transformers
+pipeline_tag: translation
+language:
+  - ko
+  - en
+  - vi
 ---
+
+# DMTLLM Translation Research
+
+> **Research Release**
+
+**DMTLLM Translation Research** is an open-weight multilingual language model developed by **DMTLabs** as part of the foundational research for DMTLLM.
+
+The model is a compact **~50M parameter decoder-only Transformer** based on a Llama-style architecture. It was **trained from scratch with randomly initialized weights** using Korean, English, and Vietnamese monolingual corpora.
+
+No pretrained Llama model weights were used.
+
+Following base language model pretraining, the model was further specialized for machine translation through **full-parameter supervised fine-tuning (SFT)** using Korean–English and Korean–Vietnamese parallel translation data.
+
+This release represents an intermediate research milestone in the development of DMTLLM and is intended primarily for research and experimentation.
+
+---
+
+## Model Overview
+
+| Item | Description |
+|---|---|
+| Developer | DMTLabs |
+| Model size | **~50M parameters** |
+| Model type | Decoder-only Transformer |
+| Implementation | `LlamaForCausalLM` |
+| Architecture | Llama-style Causal Language Model |
+| Initialization | Random initialization |
+| Base pretraining | Causal Language Modeling |
+| Pretraining languages | Korean, English, Vietnamese |
+| Post-training | Full-parameter Supervised Fine-Tuning |
+| Translation data | Korean–English, Korean–Vietnamese |
+| Specialization | Machine Translation |
+| Model status | Research / Experimental |
+| License | Apache License 2.0 |
+
+> Although the model implementation is based on the `LlamaForCausalLM` architecture provided by Hugging Face Transformers, **no pretrained Llama weights were used**. All model parameters were initialized randomly and trained from scratch.
+
+---
+
+## Model Architecture
+
+The model uses a compact Llama-style decoder-only Transformer architecture designed at approximately the **50M parameter scale**.
+
+| Specification | Value |
+|---|---:|
+| Parameters | ~50M |
+| Vocabulary size | 48,000 |
+| Hidden size | 512 |
+| Transformer layers | 8 |
+| Attention heads | 8 |
+| Key-value heads | 4 |
+| Attention type | Grouped Query Attention (GQA) |
+| Attention head dimension | 64 |
+| Intermediate size | 1,376 |
+| Maximum context length | 1,024 tokens |
+| Positional encoding | Rotary Position Embeddings (RoPE) |
+| RoPE theta | 10,000 |
+| Normalization | RMSNorm |
+| RMSNorm epsilon | 1e-5 |
+| Activation | SiLU / SwiGLU-style |
+| Attention bias | No |
+| MLP bias | No |
+| Attention dropout | 0.0 |
+| Input/output embeddings | Tied |
+| Checkpoint parameter dtype | FP32 |
+
+The architecture includes:
+
+- Rotary Position Embeddings (RoPE)
+- RMSNorm
+- Grouped Query Attention (GQA)
+- SwiGLU-style feed-forward layers
+- tied input and output embeddings
+- decoder-only causal self-attention
+
+---
+
+## Training Pipeline
+
+The model was developed in two stages:
+
+```text
+Random Initialization
+        │
+        ▼
+Base Language Model Pretraining
+(Korean / English / Vietnamese)
+        │
+        ▼
+Scratch-pretrained Base Model
+        │
+        ▼
+Full-parameter Translation SFT
+(Korean–English / Korean–Vietnamese)
+        │
+        ▼
+Translation-specialized Research Model
+```
+
+### Stage 1: Base Pretraining
+
+The base language model was trained entirely from scratch.
+
+No pretrained language model checkpoint was used. The model parameters were randomly initialized and optimized using the standard causal language modeling objective.
+
+Conceptually, the model learns next-token prediction:
+
+```text
+token₁ token₂ token₃ ... tokenₙ
+   ↓      ↓      ↓           ↓
+token₂ token₃ token₄ ... tokenₙ₊₁
+```
+
+The pretraining corpus consists of monolingual Korean, English, and Vietnamese text.
+
+Training utilizes:
+
+- Causal Language Modeling / Next Token Prediction
+- BF16 mixed-precision computation
+- AdamW optimization
+- gradient accumulation
+- gradient clipping
+- linear learning-rate warmup
+- cosine learning-rate decay
+- optional gradient checkpointing
+
+### Base Pretraining Data
+
+The monolingual corpus contains approximately **27.6 million training records** after preprocessing and filtering.
+
+| Language | Training | Validation | Test |
+|---|---:|---:|---:|
+| Korean | 13,738,080 | 140,019 | 140,843 |
+| English | 7,723,537 | 79,102 | 79,150 |
+| Vietnamese | 6,169,183 | 62,694 | 62,914 |
+| **Total** | **27,630,800** | **281,815** | **282,907** |
+
+The records were tokenized and packed into fixed-length sequences for causal language model pretraining.
+
+### Monolingual Data Filtering
+
+Duplicate and invalid samples were removed during preprocessing.
+
+| Language | Removed Duplicates | Other Filtered Samples |
+|---|---:|---:|
+| Korean | 623,012 | 79 |
+| English | 191,233 | 4 |
+| Vietnamese | 274,093 | 123 |
+| **Total** | **1,088,338** | **206** |
+
+---
+
+## Stage 2: Translation Supervised Fine-Tuning
+
+Following base pretraining, the entire model was further optimized for machine translation using **full-parameter supervised fine-tuning**.
+
+No LoRA, adapter, or other parameter-efficient fine-tuning method was used.
+
+The SFT corpus consists of Korean–English and Korean–Vietnamese parallel translation pairs.
+
+Each training example contains:
+
+```text
+[ Translation Prompt ] [ Target Translation ]
+```
+
+During SFT, the prompt portion is excluded from the language modeling loss.
+
+```text
+[ Translation Prompt ] [ Target Translation ]
+        masked               loss
+```
+
+Only target translation tokens contribute to the training objective, while all model parameters are updated.
+
+### Translation SFT Data
+
+Approximately **14.1 million parallel sentence pairs** were used for translation SFT.
+
+| Language Pair | Training | Validation | Test |
+|---|---:|---:|---:|
+| Korean–English | 7,907,787 | 80,562 | 81,229 |
+| Korean–Vietnamese | 6,200,497 | 63,509 | 63,319 |
+| **Total** | **14,108,284** | **144,071** | **144,548** |
+
+### Parallel Data Filtering
+
+Parallel data preprocessing included duplicate removal, source–target length-ratio filtering, text validation, and identical-pair filtering where applicable.
+
+| Language Pair | Raw Pairs | Duplicates | Ratio Filter | Text Filter | Identical Pair Filter |
+|---|---:|---:|---:|---:|---:|
+| Korean–English | 8,073,026 | 1,636 | 1,769 | 43 | — |
+| Korean–Vietnamese | 6,569,007 | 238,995 | 1,343 | 154 | 1,190 |
+
+The remaining examples were divided into training, validation, and test sets.
+
+---
+
+## Research Objectives
+
+This model was developed as part of the foundational research for the **DMTLLM** project.
+
+The primary objectives of this work are to investigate:
+
+- training a decoder-only language model entirely from scratch;
+- building an independent base model without relying on pretrained model weights;
+- learning multilingual representations from Korean, English, and Vietnamese monolingual corpora;
+- adapting a scratch-pretrained language model to machine translation;
+- evaluating full-parameter SFT for translation specialization;
+- investigating Korean–English and Korean–Vietnamese translation using a unified decoder-only architecture;
+- establishing a reproducible research foundation for future DMTLLM models.
+
+This model should therefore be considered a **research artifact rather than a production-ready DMTLLM release**.
+
+---
+
+## Languages
+
+The base pretraining stage includes:
+
+- Korean (`ko`)
+- English (`en`)
+- Vietnamese (`vi`)
+
+The translation specialization stage uses parallel data for:
+
+- Korean–English
+- Korean–Vietnamese
+
+Translation capabilities may vary depending on translation direction, domain, sentence complexity, and prompt format.
+
+---
+
+## Evaluation
+
+Comprehensive quantitative evaluation results are being prepared.
+
+Planned evaluation includes:
+
+- translation quality evaluation;
+- language-pair-specific performance analysis;
+- comparison between the scratch-pretrained base model and the translation-SFT model;
+- analysis of translation specialization after full-parameter SFT;
+- qualitative analysis of generated translations.
+
+Evaluation results will be added in future updates.
+
+---
+
+## Intended Use
+
+This model is primarily intended for:
+
+- machine translation research;
+- multilingual language model research;
+- research on language models trained from scratch;
+- experiments involving translation-oriented supervised fine-tuning;
+- decoder-only Transformer translation experiments;
+- Korean–English and Korean–Vietnamese translation research;
+- foundational research for future DMTLLM models.
+
+---
+
+## Limitations
+
+This is an experimental research model.
+
+The model may generate:
+
+- inaccurate translations;
+- incomplete translations;
+- hallucinated content;
+- unexpected outputs;
+- outputs that differ depending on prompt format or decoding configuration.
+
+The model has not yet undergone comprehensive evaluation across languages, domains, safety scenarios, or production environments.
+
+Its relatively compact model scale may also limit linguistic knowledge, reasoning ability, contextual understanding, and translation quality compared with substantially larger language models.
+
+Performance may vary depending on:
+
+- language pair;
+- translation direction;
+- input domain;
+- sentence length;
+- prompt format;
+- context length;
+- decoding parameters.
+
+This model should not be used for safety-critical or other high-stakes applications without additional evaluation and validation.
+
+---
+
+## License
+
+The released model weights and accompanying materials are provided under the **Apache License 2.0**, unless otherwise noted.
+
+Users are responsible for ensuring that their use of the model complies with applicable laws, regulations, and third-party rights.