---
language:
- lus
- en
license: mit
base_model: openai-community/gpt2
tags:
- mizo
- mizgpt
- gpt2
- text-generation
- causal-lm
- nlp
pipeline_tag: text-generation
metrics:
- accuracy
- perplexity
model-index:
- name: MizGPT
  results:
  - task:
      name: Causal Language Modeling
      type: text-generation
    metrics:
    - name: Accuracy
      type: accuracy
      value: 0.3918
    - name: Perplexity
      type: perplexity
      value: 24.99
    - name: Validation Loss
      type: loss
      value: 3.2185
widget:
- text: "Kan khawvel hi"
  example_title: "General prompt"
- text: "Thawnthu pakhat chu:"
  example_title: "Story prompt"
- text: "Aizawl khawpui chu"
  example_title: "City prompt"
- text: "Ka hming chu"
  example_title: "Introduction"
---

# MizGPT

**MizGPT** is a causal language model fine-tuned for the **Mizo language** (Lushai). It is based on [openai-community/gpt2](https://huggingface.co/openai-community/gpt2) (~124M parameters) and trained on a curated corpus of cleaned and tokenized Mizo text.

Developed and maintained by **[Hillbyte](https://huggingface.co/hillbyte)**.

---

## Model Details

- **Model Name:** MizGPT
- **Model ID:** `hillbyte/MizGPT`
- **Architecture:** GPT-2 (Causal Language Model / `GPT2LMHeadModel`)
- **Base Model:** `openai-community/gpt2`
- **Parameters:** ~124 Million
- **Primary Language:** Mizo (`lus`) / English
- **License:** MIT
- **Training Corpus:** Curated Mizo text corpus (125,341 training samples)

---

## Evaluation & Performance

The model was evaluated on a dedicated validation split (7,403 samples) over 15 epochs (156,690 optimization steps):

| Metric | Score |
|---|---|
| **Perplexity** | **24.99** |
| **Validation Accuracy** | **39.18%** |
| **Validation Loss** | **3.2185** |
| **Training Loss** | **1.4742** |
| **Training Epochs** | 15.0 |
| **Total FLOPs** | 9.83e+17 |

---

## How to Use

### 1. Using Hugging Face Pipeline

```python
from transformers import pipeline

generator = pipeline(
    "text-generation",
    model="hillbyte/MizGPT",
    device=-1, # use 0 for CUDA
)

prompt = "Kan khawvel hi"
output = generator(
    prompt,
    max_new_tokens=50,
    temperature=0.7,
    top_p=0.9,
    repetition_penalty=1.1,
    do_sample=True,
)

print(output[0]["generated_text"])
```

### 2. Using PyTorch & AutoModelForCausalLM

```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "hillbyte/MizGPT"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

# Set padding token
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

prompt = "Thawnthu: "
inputs = tokenizer(prompt, return_tensors="pt")

with torch.no_grad():
    output_tokens = model.generate(
        **inputs,
        max_new_tokens=60,
        temperature=0.7,
        top_p=0.9,
        top_k=50,
        repetition_penalty=1.1,
        do_sample=True,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output_tokens[0], skip_special_tokens=True))
```

---

## Recommended Generation Parameters

Because GPT-2 is a 124M parameter model, setting a slight repetition penalty and top-p sampling yields natural and fluent Mizo completions:
- `temperature`: `0.7`
- `top_p`: `0.9`
- `top_k`: `50`
- `repetition_penalty`: `1.1`

---

## Intended Uses & Limitations

- **Intended Uses:** Mizo story generation, creative writing assistance, text completion, and linguistic NLP research in low-resource Indian languages.
- **Limitations:** Like all autoregressive models, generations should be reviewed for factual correctness and coherence over very long outputs.