| --- |
| language: |
| - en |
| license: mit |
| pipeline_tag: text-generation |
| tags: |
| - code |
| - python |
| - javascript |
| - html |
| - css |
| - causal-lm |
| - safetensors |
| --- |
| |
| # Ather 1 (22.1M Parameters) |
|
|
| **Ather 1** is a compact, high-efficiency causal language model trained for code synthesis, algorithm reasoning, and software engineering. Developed by Atherium Labs for Milo. |
|
|
| ## Model Summary |
|
|
| - **Parameters**: 22,146,048 |
| - **Architecture**: Decoder-only Transformer with Rotary Position Embeddings (RoPE) |
| - **Attention**: Grouped-Query Attention (GQA) with 8 query heads and 2 key-value heads |
| - **Feed-Forward**: SwiGLU Gated Multi-Layer Perceptron |
| - **Sequence Length**: 512 tokens |
| - **Vocabulary**: 576 byte-level BPE tokens (guarantees 0% out-of-vocabulary rate) |
| - **Trained With**: PyTorch, AdamW, Cosine Learning Rate Schedule, Target Masking SFT |
|
|
| ## Architecture Specifications |
|
|
| | Hyperparameter | Value | |
| | :--- | :--- | |
| | Embedding Dimension (`dim`) | 512 | |
| | Hidden Dimension (`hidden_dim`) | 1,360 | |
| | Transformer Layers (`n_layers`) | 8 | |
| | Query Heads (`n_heads`) | 8 | |
| | Key-Value Heads (`n_kv_heads`) | 2 | |
| | Position Embeddings | RoPE (base = 10,000) | |
| | Normalization | RMSNorm (eps = 1e-5) | |
| | Tied Embeddings | Yes | |
|
|
| ## Inference Example |
|
|
| You can run Ather 1 directly using the included standalone scripts: |
|
|
| ```python |
| import torch |
| import json |
| from safetensors.torch import load_file |
| from standalone_model import Ather1Model |
| from standalone_tokenizer import StandaloneAtherTokenizer |
| |
| # 1. Load configuration and weights |
| with open("config.json", "r") as f: |
| config = json.load(f) |
| |
| tokenizer = StandaloneAtherTokenizer("vocab.json") |
| model = Ather1Model(config) |
| model.load_state_dict(load_file("model.safetensors")) |
| model.eval() |
| |
| # 2. Encode prompt |
| prompt = "<|user|>\nWrite a Python function to reverse a string\n<|assistant|>\n" |
| prompt_tokens = tokenizer.encode(prompt, add_bos=True) |
| |
| # 3. Generate completion |
| out_tokens = model.generate( |
| prompt_tokens=prompt_tokens, |
| max_new_tokens=256, |
| temperature=0.7, |
| eos_token_id=tokenizer.eos_token_id, |
| ) |
| |
| response = tokenizer.decode(out_tokens[len(prompt_tokens):], skip_special_tokens=True) |
| print(response) |
| ``` |
|
|
| ## API Access |
|
|
| Ather 1 is hosted live on Cloudflare Pages and through the Atherium AI platform: |
| - Web Application: [https://atheriumai.pages.dev](https://atheriumai.pages.dev) |
| - Model ID: `ather-1` |
|
|
