ARCHITECTURE.md

Ares architecture contract

Token flow

text → tokenizer IDs → token embeddings → RoPE → causal GQA → RMSNorm → SwiGLU → residual stream → final RMSNorm → unembedding → logits → temperature/top-p sampler → token output

Every stage must have a unit test and a shape assertion. The first checkpoint should be small enough to overfit a tiny corpus; that proves the implementation before scaling.

Initial research configuration

• context: 2,048
• vocabulary: choose after tokenizer measurement; do not assume 128K
• layers: 8
• model width: 512
• attention heads: 8 query / 2 KV (GQA)
• FFN: SwiGLU, hidden size approximately 4× width
• normalization: RMSNorm, pre-norm
• positions: RoPE
• objective: causal next-token cross entropy
• optimizer: AdamW, warmup + cosine decay, gradient clipping

This is an experiment configuration, not a frontier configuration. It is approximately tens of millions of parameters depending on vocabulary and FFN size.

Memory and tools

RAG is not model memory: retrieve records, score them, add bounded excerpts to the prompt, and retain provenance. Tools must use an allowlist, explicit user-visible calls, timeouts, resource limits, and sanitized outputs. Code execution must happen outside a static page in a sandbox.