| --- |
| language: |
| - lus |
| - en |
| license: mit |
| base_model: openai-community/gpt2 |
| tags: |
| - mizo |
| - mizgpt |
| - gpt2 |
| - text-generation |
| - causal-lm |
| - nlp |
| pipeline_tag: text-generation |
| metrics: |
| - accuracy |
| - perplexity |
| model-index: |
| - name: MizGPT |
| results: |
| - task: |
| name: Causal Language Modeling |
| type: text-generation |
| metrics: |
| - name: Accuracy |
| type: accuracy |
| value: 0.3918 |
| - name: Perplexity |
| type: perplexity |
| value: 24.99 |
| - name: Validation Loss |
| type: loss |
| value: 3.2185 |
| widget: |
| - text: "Kan khawvel hi" |
| example_title: "General prompt" |
| - text: "Thawnthu pakhat chu:" |
| example_title: "Story prompt" |
| - text: "Aizawl khawpui chu" |
| example_title: "City prompt" |
| - text: "Ka hming chu" |
| example_title: "Introduction" |
| --- |
| |
| # MizGPT |
|
|
| **MizGPT** is a causal language model fine-tuned for the **Mizo language** (Lushai). It is based on [openai-community/gpt2](https://huggingface.co/openai-community/gpt2) (~124M parameters) and trained on a curated corpus of cleaned and tokenized Mizo text. |
|
|
| Developed and maintained by **[Hillbyte](https://huggingface.co/hillbyte)**. |
|
|
| --- |
|
|
| ## Model Details |
|
|
| - **Model Name:** MizGPT |
| - **Model ID:** `hillbyte/MizGPT` |
| - **Architecture:** GPT-2 (Causal Language Model / `GPT2LMHeadModel`) |
| - **Base Model:** `openai-community/gpt2` |
| - **Parameters:** ~124 Million |
| - **Primary Language:** Mizo (`lus`) / English |
| - **License:** MIT |
| - **Training Corpus:** Curated Mizo text corpus (125,341 training samples) |
|
|
| --- |
|
|
| ## Evaluation & Performance |
|
|
| The model was evaluated on a dedicated validation split (7,403 samples) over 15 epochs (156,690 optimization steps): |
|
|
| | Metric | Score | |
| |---|---| |
| | **Perplexity** | **24.99** | |
| | **Validation Accuracy** | **39.18%** | |
| | **Validation Loss** | **3.2185** | |
| | **Training Loss** | **1.4742** | |
| | **Training Epochs** | 15.0 | |
| | **Total FLOPs** | 9.83e+17 | |
|
|
| --- |
|
|
| ## How to Use |
|
|
| ### 1. Using Hugging Face Pipeline |
|
|
| ```python |
| from transformers import pipeline |
| |
| generator = pipeline( |
| "text-generation", |
| model="hillbyte/MizGPT", |
| device=-1, # use 0 for CUDA |
| ) |
| |
| prompt = "Kan khawvel hi" |
| output = generator( |
| prompt, |
| max_new_tokens=50, |
| temperature=0.7, |
| top_p=0.9, |
| repetition_penalty=1.1, |
| do_sample=True, |
| ) |
| |
| print(output[0]["generated_text"]) |
| ``` |
|
|
| ### 2. Using PyTorch & AutoModelForCausalLM |
|
|
| ```python |
| import torch |
| from transformers import AutoTokenizer, AutoModelForCausalLM |
| |
| model_id = "hillbyte/MizGPT" |
| tokenizer = AutoTokenizer.from_pretrained(model_id) |
| model = AutoModelForCausalLM.from_pretrained(model_id) |
| |
| # Set padding token |
| if tokenizer.pad_token is None: |
| tokenizer.pad_token = tokenizer.eos_token |
| |
| prompt = "Thawnthu: " |
| inputs = tokenizer(prompt, return_tensors="pt") |
| |
| with torch.no_grad(): |
| output_tokens = model.generate( |
| **inputs, |
| max_new_tokens=60, |
| temperature=0.7, |
| top_p=0.9, |
| top_k=50, |
| repetition_penalty=1.1, |
| do_sample=True, |
| pad_token_id=tokenizer.pad_token_id, |
| eos_token_id=tokenizer.eos_token_id, |
| ) |
| |
| print(tokenizer.decode(output_tokens[0], skip_special_tokens=True)) |
| ``` |
|
|
| --- |
|
|
| ## Recommended Generation Parameters |
|
|
| Because GPT-2 is a 124M parameter model, setting a slight repetition penalty and top-p sampling yields natural and fluent Mizo completions: |
| - `temperature`: `0.7` |
| - `top_p`: `0.9` |
| - `top_k`: `50` |
| - `repetition_penalty`: `1.1` |
|
|
| --- |
|
|
| ## Intended Uses & Limitations |
|
|
| - **Intended Uses:** Mizo story generation, creative writing assistance, text completion, and linguistic NLP research in low-resource Indian languages. |
| - **Limitations:** Like all autoregressive models, generations should be reviewed for factual correctness and coherence over very long outputs. |
|
|
