README.md

---
language:
- en
license: apache-2.0
tags:
- recursive-language-model
- multimodal
- self-automated
- pytorch
- safetensors
- ancient-ai
model_type: ancient_ai
pipeline_tag: text-generation
---

# Ancient.AI.V — Recursive Language Model

**Architecture:** Recursive Language Model (RLM)
**Not a Large Language Model** — a fundamentally different architecture built from scratch.

| Property | Value |
|---|---|
| Parameters | 1.147B |
| Context Window | 64,000 tokens |
| Layers | 24 |
| Hidden Size | 2,048 |
| Attention Heads | 16 (GQA, 8 KV heads) |
| FFN Dimension | 8,192 |
| Vocab Size | 64,000 |
| Activation | SwiGLU |
| Position Encoding | YaRN-extended RoPE (base 500k, scale 8×) |
| Weight Format | safetensors |
| Precision | bfloat16 (fine-tune target) |

---

## What Makes It Different From an LLM

Standard LLMs run one forward pass: input → output.

Ancient.AI.V runs a **Recursive Outer Loop**: the model refines its own
output `recursion_depth` times per call, with a learned halting gate that
stops early when confident. This is the core of the Recursive Language Model paradigm.

---

## Integrated Self-Automated (SA) Modules

All 17 SA modules operate **simultaneously** within each decoder layer as
parallel residual paths — not sequential post-processing steps.

| Module | Implementation |
|---|---|
| SA Meta-Learning | Per-sample fast-weight delta generation (learned MAML inner loop) |
| SA Reinforcement Learning | Per-token value estimation + policy gate (actor-critic in forward pass) |
| SA Continual Learning | EWC-inspired importance weighting from initial representations |
| SA Adaptive Learning | Learned depth-gating; tokens can exit processing early |
| SA Rewriting | Cross-attention from current → earlier hidden states (in-context revision) |
| SA NLP | Bigram/trigram convolutions + semantic role projection |
| SA Problem Solving | Multi-step latent chain-of-thought scratchpad (3 internal steps) |
| SA Innovation | Novelty-promoting repulsion in embedding space |
| SA Debugging | Anomaly detection + learned correction on hidden state norms |
| SA Long/Short-Term Memory | 512 persistent learnable memory slots with read/write gating |
| SA Recursive Seed Learning | Compress → refine → expand self-representation cycle |
| SA Self-Evaluation & Reward | Per-token reward MLP; plugs directly into PPO/GRPO fine-tuning |
| SA Goal & Constraint Engine | Learned goal embedding cross-attends to steer generation |
| SA Memory Consolidation | Bidirectional GRU trace encoder with hippocampal replay |
| SA Introspection Interface | Uncertainty + confidence mapping over hidden states |
| SA Recursive Outer Loop | Post-stack self-refinement with learned halting |
| SA Conversational Intelligence | Dialogue state tracker (turn, topic shift, emotion, formality) |

---

## Multimodal Support

Native encoders for all four modalities, fused before the decoder stack:

- **Text** — BPE tokenizer, 64k vocab
- **Image** — ViT-style patch encoder (16×16 patches, up to 224×224)
- **Audio** — Whisper-style mel-spectrogram encoder (80 mel bins)
- **Video** — Frame-by-frame ViT + temporal self-attention

---

## Training / Fine-Tuning

This checkpoint contains **randomly initialized weights** — it is an
architecture shell ready for fine-tuning.

Recommended fine-tuning approaches:
- **SFT** (Supervised Fine-Tuning) with causal LM loss
- **RLHF/PPO** — plug training reward into the `SASelfEvaluation` reward head
- **GRPO** — the `sa_eval` reward signal is already shaped for group-relative optimization
- **LoRA / QLoRA** — compatible with standard PEFT adapters

Training the self-reward head jointly with SFT gives Ancient.AI.V
self-improvement capability without a separate reward model.

---

## Usage

```python
# AutoTokenizer available after fine-tuning with a trained tokenizer
from ancient_ai import AncientConfig, AncientAIV  # after registering custom class
import torch

cfg   = AncientConfig()
model = AncientAIV(cfg)
# Load weights:
# model = AncientAIV.from_pretrained("GODsStrongestSoldier/Ancient.AI.V")

tokenizer = AutoTokenizer.from_pretrained("GODsStrongestSoldier/Ancient.AI.V")
input_ids = tokenizer("Hello Ancient.AI", return_tensors="pt").input_ids

generated = model.generate_text(input_ids, max_new=200, temperature=0.8)
print(tokenizer.decode(generated[0]))
```

---

## Architecture Citation

```
Ancient.AI.V — Recursive Language Model (RLM)
Author: GODsStrongestSoldier
Year: 2025
Architecture: Custom RLM with 17 integrated SA modules
Repo: https://huggingface.co/GODsStrongestSoldier/Ancient.AI.V
```

---

## License

Apache 2.0 — free for research and commercial fine-tuning.