# Sage

Sage is a custom-built transformer language model designed for text generation tasks. This model demonstrates the full lifecycle of building and publishing a custom AI model to Hugging Face.

## Model Overview

- **Model Type**: Transformer-based language model
- **Architecture**: Decoder-only transformer
- **Vocabulary Size**: 40 characters
- **Hidden Size**: 256
- **Number of Layers**: 4
- **Number of Attention Heads**: 8
- **Feedforward Size**: 1024
- **Max Sequence Length**: 64
- **Parameters**: ~3,195,944
- **Training Framework**: PyTorch
- **License**: MIT

## Training Data

Sage was trained on a curated dataset of example sentences covering:
- Conversational phrases and greetings
- Weather and environmental descriptions
- Machine learning and AI concepts
- Deep learning architectures (transformers, neural networks)
- Natural language processing applications
- Model development and deployment practices

The dataset consists of 10 examples designed to teach the model patterns in technical and conversational English.

## Technical Specifications

### Model Architecture
```
TransformerLM(
  (embedding): Embedding(40, 256)
  (pos_embedding): Embedding(64, 256)
  (transformer_encoder): TransformerEncoder(
    (layers): ModuleList(
      (0-3): TransformerEncoderLayer(
        (self_attn): MultiheadAttention(embed_dim=256, num_heads=8)
        (linear1): Linear(256, 1024)
        (linear2): Linear(1024, 256)
        (norm1): LayerNorm(256)
        (norm2): LayerNorm(256)
        (dropout): Dropout(p=0.1)
      )
    )
  )
  (output_layer): Linear(256, 40)
)
```

### Tokenization
Sage uses a character-level tokenizer with:
- Vocabulary: 40 unique characters including special tokens
- Special tokens: `<PAD>` (0), `<UNK>` (1)
- Encoding: UTF-8 character mapping
- Maximum sequence length: 64 tokens

## Usage

### With Transformers Library
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_name = "itriedcoding/Sage"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

def generate_text(prompt, max_length=50, temperature=0.8):
    inputs = tokenizer.encode(prompt, return_tensors="pt")
    with torch.no_grad():
        outputs = model.generate(
            inputs,
            max_length=max_length,
            temperature=temperature,
            do_sample=True,
            pad_token_id=tokenizer.eos_token_id
        )
    return tokenizer.decode(outputs[0], skip_special_tokens=True)

print(generate_text("Hello"))
print(generate_text("Deep learning"))
```

### Direct PyTorch Usage
```python
import torch
from modeling_transformer_lm import TransformerLM

model = TransformerLM.from_pretrained("itriedcoding/Sage")
```

## Model Card Metadata

```
library_name: transformers
license: MIT
base_model: custom-built
tags:
- text-generation
- transformer
- character-level
- custom-model
- educational
pipeline_tag: text-generation
```

## Hugging Face Spaces Deployment

You can run Sage in the dedicated Hugging Face Space:
https://huggingface.co/spaces/itriedcoding/sage-space

### Gradio Space
The Space at `itriedcoding/sage-space` provides a Gradio interface for text generation.
Create a new Space with `app.py`:
```python
import gradio as gr
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_name = "itriedcoding/Sage"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

def generate_text(prompt, max_length, temperature):
    inputs = tokenizer.encode(prompt, return_tensors="pt")
    with torch.no_grad():
        outputs = model.generate(
            inputs,
            max_length=int(max_length),
            temperature=temperature,
            do_sample=True,
            pad_token_id=tokenizer.eos_token_id
        )
    return tokenizer.decode(outputs[0], skip_special_tokens=True)

demo = gr.Interface(
    fn=generate_text,
    inputs=[
        gr.Textbox(label="Prompt", value="Hello"),
        gr.Slider(minimum=10, maximum=100, value=30, label="Max Length"),
        gr.Slider(minimum=0.1, maximum=2.0, value=0.8, label="Temperature")
    ],
    outputs=gr.Textbox(label="Generated Text"),
    title="Sage Text Generator",
    description="Custom character-level language model"
)

if __name__ == "__main__":
    demo.launch()
```

## GGUF Format

Sage is available in GGUF format as `sage-f16.gguf`.

### Compatibility Warning
Sage uses a custom `transformer_lm` architecture that is NOT supported by standard llama.cpp or llama-cpp-python. The GGUF file is provided as a reference format and for custom inference implementations that can match Sage's architecture.

### File Details
- **File**: `sage-f16.gguf` (12.7 MB)
- **Format**: GGUF (GGML Universal Format)
- **Precision**: Float16
- **Tensors**: 52 layers
- **Architecture**: `transformer_lm` (custom)

### Using with Custom Inference
To use this GGUF file, you need a GGUF loader that supports Sage's custom architecture:
```python
import gguf
import torch
import numpy as np

# Load GGUF file
reader = gguf.GGUFReader("sage-f16.gguf")
tensors = {t.name: torch.from_numpy(t.data) for t in reader.tensors}

# Map tensor names back to Sage architecture
# See gguf_convert.py for the tensor name mapping
```

### GGUF Conversion
The conversion script `gguf_convert.py` is included in this repository. It uses the `gguf` Python library to convert the PyTorch checkpoint to GGUF format.

## Performance & Limitations

### Intended Use
- Educational demonstrations of transformer architectures
- Character-level language modeling experiments
- Prototyping and testing custom model pipelines
- Learning about model deployment on Hugging Face

### Limitations
- Character-level tokenization limits coherence
- Small training dataset (10 examples)
- Small model size (3.2M parameters)
- Not suitable for production NLP applications
- Best for short text generation (<50 tokens)

## Citation

```bibtex
@misc{sage_model_2026,
  author = {itriedcoding},
  title = {Sage: Custom Character-Level Language Model},
  year = {2026},
  publisher = {Hugging Face},
  journal = {Hugging Face Model Hub},
  url = {https://huggingface.co/itriedcoding/Sage}
}
```

## Training Reproducibility

To reproduce this model:
1. Clone the repository
2. Install requirements: `pip install torch pandas`
3. Run training: The model was trained using the script in `train_model.py`
4. The trained checkpoint is saved as a PyTorch .pth file

## Contact

- Hugging Face: https://huggingface.co/itriedcoding
- Model Space: https://huggingface.co/spaces/itriedcoding/sage-space
- Issues: Use the "Issues" tab on this model page