| | --- |
| | language: vi |
| | tags: |
| | - vietnamese |
| | - causal-lm |
| | - finetuning |
| | - viena |
| | library_name: transformers |
| | pipeline_tag: text-generation |
| | license: other |
| | base_model: vietrix/viena-60m-pretrain |
| | --- |
| | |
| | # Viena 60M (SFT) |
| |
|
| | ## Model details |
| | - Developed by: Vietrix |
| | - Model type: decoder-only causal LM (Llama-style) |
| | - Parameters: ~60M |
| | - Layers: 16 |
| | - Hidden size: 512 |
| | - Attention heads: 8 (KV heads: 4) |
| | - Max sequence length: 1024 |
| | - RoPE theta: 10000 |
| | - Normalization/MLP: RMSNorm + SwiGLU |
| | - Precision: BF16 training |
| |
|
| | ## Tokenizer |
| | - SentencePiece BPE |
| | - Target vocab in config: 32k |
| | - Actual vocab in tokenizer.model: 2105 (trained on a small corpus) |
| | - Note: embeddings are sized for 32k; only the first 2105 tokens are used by the tokenizer. |
| |
|
| | ## Training data |
| | - Internal synthetic Vietnamese instruction/chat data. |
| | - Train/val split: 2,000 / 200 JSONL records. |
| | - Format: messages with roles (system/user/assistant/tool). |
| | - PII: best-effort redaction applied during dataset preparation. |
| |
|
| | ## Fine-tuning procedure |
| | - Initialized from: `vietrix/viena-60m-pretrain`. |
| | - Objective: token-level cross-entropy, prompt loss disabled. |
| | - Sequence length: 1024. |
| | - Global batch size: 32 (batch 8 x grad_accum 4). |
| | - Optimizer: AdamW, lr 2e-4, weight decay 0.01, cosine decay with warmup. |
| | - Steps: 1,000. |
| | - Validation every 200 steps (10 batches). |
| | |
| | ## Intended use |
| | - Vietnamese chat/instruction-following use cases. |
| | - Research and prototyping; not a production-grade safety model. |
| | |
| | ## Limitations |
| | - Trained on a small synthetic corpus; may hallucinate or respond incorrectly. |
| | - Not safety-tuned for sensitive domains. |
| | - Tokenizer vocab is small; lexical coverage is limited. |
| | |
| | ## How to use |
| | ```python |
| | from transformers import AutoModelForCausalLM, AutoTokenizer |
| | |
| | model_id = "vietrix/viena-60m" |
| | tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=False) |
| | model = AutoModelForCausalLM.from_pretrained(model_id) |
| | ``` |
| | |
| | If `AutoTokenizer` fails, load the SentencePiece model explicitly: |
| | ```python |
| | from transformers import LlamaTokenizer |
| | |
| | tokenizer = LlamaTokenizer.from_pretrained(model_id, use_fast=False) |
| | ``` |
| | |
