ARCHITECTURE.md

# Ares architecture contract

## Token flow

`text → tokenizer IDs → token embeddings → RoPE → causal GQA → RMSNorm → SwiGLU → residual stream → final RMSNorm → unembedding → logits → temperature/top-p sampler → token output`

Every stage must have a unit test and a shape assertion. The first checkpoint should be small enough to overfit a tiny corpus; that proves the implementation before scaling.

## Initial research configuration

- context: 2,048
- vocabulary: choose after tokenizer measurement; do not assume 128K
- layers: 8
- model width: 512
- attention heads: 8 query / 2 KV (GQA)
- FFN: SwiGLU, hidden size approximately 4× width
- normalization: RMSNorm, pre-norm
- positions: RoPE
- objective: causal next-token cross entropy
- optimizer: AdamW, warmup + cosine decay, gradient clipping

This is an experiment configuration, not a frontier configuration. It is approximately tens of millions of parameters depending on vocabulary and FFN size.

## Memory and tools

RAG is not model memory: retrieve records, score them, add bounded excerpts to the prompt, and retain provenance. Tools must use an allowlist, explicit user-visible calls, timeouts, resource limits, and sanitized outputs. Code execution must happen outside a static page in a sandbox.