---
language:
- tl
tags:
- tagalog
- text-generation
- custom-architecture
- pytorch
license: mit
---

# Henyo-70M

**Henyo** is a custom Tagalog LLM trained on a subset of Wikipedia.

## Model Architecture
- **Parameter Count**: 70M
- **Architecture**: Decoder-only Transformer (Custom)
- **Features**:
  - **SwiGLU** Activation
  - **Grouped Query Attention (GQA)**
  - **Rotary Positional Embeddings (RoPE)**
  - **RMSNorm**

## Usage

This model uses a custom architecture code. You can load it using the `AutoModel` class with `trust_remote_code=True` (if code is uploaded) or by defining the class manually.

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("marcuscedricridia/Henyo-70M")
# Note: Since this is a custom model, you may need the inference script provided in the repo.