# 🚀 summerMC/TRM-textv3

`TRM-textv3` is an **ultra-lightweight (~84M parameters)** custom Transformer model, meticulously optimized through a pipeline of **Distillation**, **SFT**, and **DPO** to deliver efficient conversational intelligence.

## 🛠 Model Specifications

| Attribute | Detail |
| :--- | :--- |
| **Model Type** | Causal Language Model |
| **Architecture** | `trm_text_ism` (Single-layer Efficiency) |
| **Parameters** | 84,312,320 (~84M) |
| **Vocabulary Size** | 50,259 |
| **Sequence Length** | 512 tokens |
| **Precision** | `bfloat16` |

## ⚡ Training Paradigm

1.  **Distillation**: Knowledge extraction from high-capacity teacher models to define the foundation.
2.  **SFT (Supervised Fine-Tuning)**: Adapted using the `OpenHermes-2.5` dataset for refined chat-based interaction.
3.  **DPO (Direct Preference Optimization)**: Final alignment stage to enhance response coherence and mitigate hallucinations.

## 📖 Usage

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "summerMC/TRM-textv3"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)

prompt = "User: Hello! How can you help me?\nAssistant:"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=64, temperature=0.4)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## ⚠️ Limitations & Notes

- **Primary Language**: English. While it has some exposure to Japanese, complex multi-turn dialogue in Japanese is considered experimental.
- **Scale**: Due to its extreme 84M scale, it is best suited for narrative assistance, simple logic tasks, and edge-device deployment where latency is critical.
- **Safety**: Always apply a safety layer when deploying in production environments.