---
language:
- en
license: mit
pipeline_tag: text-generation
tags:
- code
- python
- javascript
- html
- css
- causal-lm
- safetensors
---

# Ather 1 (22.1M Parameters)

**Ather 1** is a compact, high-efficiency causal language model trained for code synthesis, algorithm reasoning, and software engineering. Developed by Atherium Labs for Milo.

## Model Summary

- **Parameters**: 22,146,048
- **Architecture**: Decoder-only Transformer with Rotary Position Embeddings (RoPE)
- **Attention**: Grouped-Query Attention (GQA) with 8 query heads and 2 key-value heads
- **Feed-Forward**: SwiGLU Gated Multi-Layer Perceptron
- **Sequence Length**: 512 tokens
- **Vocabulary**: 576 byte-level BPE tokens (guarantees 0% out-of-vocabulary rate)
- **Trained With**: PyTorch, AdamW, Cosine Learning Rate Schedule, Target Masking SFT

## Architecture Specifications

| Hyperparameter | Value |
| :--- | :--- |
| Embedding Dimension (`dim`) | 512 |
| Hidden Dimension (`hidden_dim`) | 1,360 |
| Transformer Layers (`n_layers`) | 8 |
| Query Heads (`n_heads`) | 8 |
| Key-Value Heads (`n_kv_heads`) | 2 |
| Position Embeddings | RoPE (base = 10,000) |
| Normalization | RMSNorm (eps = 1e-5) |
| Tied Embeddings | Yes |

## Inference Example

You can run Ather 1 directly using the included standalone scripts:

```python
import torch
import json
from safetensors.torch import load_file
from standalone_model import Ather1Model
from standalone_tokenizer import StandaloneAtherTokenizer

# 1. Load configuration and weights
with open("config.json", "r") as f:
    config = json.load(f)

tokenizer = StandaloneAtherTokenizer("vocab.json")
model = Ather1Model(config)
model.load_state_dict(load_file("model.safetensors"))
model.eval()

# 2. Encode prompt
prompt = "<|user|>\nWrite a Python function to reverse a string\n<|assistant|>\n"
prompt_tokens = tokenizer.encode(prompt, add_bos=True)

# 3. Generate completion
out_tokens = model.generate(
    prompt_tokens=prompt_tokens,
    max_new_tokens=256,
    temperature=0.7,
    eos_token_id=tokenizer.eos_token_id,
)

response = tokenizer.decode(out_tokens[len(prompt_tokens):], skip_special_tokens=True)
print(response)
```

## API Access

Ather 1 is hosted live on Cloudflare Pages and through the Atherium AI platform:
- Web Application: [https://atheriumai.pages.dev](https://atheriumai.pages.dev)
- Model ID: `ather-1`