README.md

---
license: other
language: en
library_name: transformers
tags:
- transformer
- language-models
- long-context
- memory-augmented-transformers
- eidosformer
- causal-language-modeling
- ai-research
- neural-architecture
- episodic-memory
- semantic-memory
- kNN-inference
- compressive-transformer
- llama-family
- PyTorch
model-index:
- name: EidosFormer 1B
  config:
    model_type: eidosformer
    num_hidden_layers: 24
    hidden_size: 2048
    num_attention_heads: 16
    intermediate_size: 5632
    memory_store: FAISS
    max_position_embeddings: 8192
  results: []
widget:
  - text: "The EidosFormer model represents a significant advancement in"
---

# EidosFormer — Memory-Native Transformer Architecture

<div align="center">

**A novel transformer architecture that unifies episodic, semantic, and working memory systems for long-context understanding.**

[🤗 Hugging Face](https://huggingface.co/Himan-de/EidosFormer) • [📄 Model Card](https://huggingface.co/Himan-de/EidosFormer)

</div>

---

## 📋 Overview

**EidosFormer** (pronounced *eye-dos-former*) is a memory-augmented language model that integrates four independently validated research mechanisms into a unified, end-to-end trainable architecture. It addresses the long-context problem by maintaining **three parallel memory subsystems**:

| Memory System | Capacity | Update Rule | Retrieval Mechanism |
|---|---|---|---|
| **Episodic** | Dynamic (FAISS-backed) | STE kNN append | Cosine similarity retrieval |
| **Semantic** | Fixed centroid slots | Soft-gated EMA consolidation | Attention-based recall |
| **Working** | Window-size bounded | Standard transformer attention | Direct positional access |

### Key Innovations

1. **Differentiable Consolidation**: Episodic memories automatically compress into semantic representations using dual objectives (reconstruction loss + attention-preservation loss)
2. **Ebbinghaus-Style Recall Reinforcement**: Frequently retrieved items receive memory boost, preventing catastrophic forgetting in long sequences
3. **Multi-Cross-Attention Gating**: First-layer cross-attention connects all memory streams; subsequent layers operate on unified representations
4. **Modern Architecture Stack**: Grouped Query Attention (GQA), RoPE with NTK/Instruct scaling, RMSNorm, SwiGLU feedforward — proven Llama-family components

## 🏗️ Architecture

```
Input Tokens
    │
    ├─► Token Embedding + Temporal Encoding (wall-clock aware)
    │
    ▼
┌───────────────────────┐
│  Episodic Memory Store │ ← FAISS ANN index, STE kNN retrieval
└───────────────────────┘
         │ retrieved_kv
         ▼
┌───────────────────────┐
│   Semantic Patterns    │ ← EMA-updated centroids (consolidated from episodic)
└───────────────────────┘
         │ merged with kv
         ▼
┌───────────────────────┐
│  Memory Cross-Attn     │ ← Layer 0: connects all memory streams
└───────────────────────┘
         │ unified representation
         ▼
┌───────────────────────────┐
│  EidosBlock × N layers    │ ← CausalSelfAttention + SwiGLU blocks
└───────────────────────────┘
         │
         ▼
┌───────────────────────┐
│  MemoryWriteHead        │ ← gated write-back to episodic store
└───────────────────────┘

Side channels (trainer-driven, no-grad):
  - model.append_episodic(x)     → FAISS vector store
  - model.write_to_semantic()    → soft-gated EMA update
  - model.apply_consolidation()  → weakest-slot compression
  - model.resize_semantic(n)     → memory curriculum (grow/shrink)
```

## 🚀 Quick Start

### Installation

```bash
pip install transformers torch faiss-cpu accelerate sentencepiece
```

### Loading the Model

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Himan-de/EidosFormer",
    trust_remote_code=True,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(
    "Himan-de/EidosFormer",
    trust_remote_code=True
)

# Generate text
inputs = tokenizer("The future of AI is driven by memory systems that enable:", return_tensors="pt")
outputs = model.generate(**inputs.to(model.device), max_new_tokens=256, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

### Using with the Hugging Face Hub

```bash
# Access via huggingface-cli
huggingface-cli login  # Enter your token when prompted
huggingface-cli download Himan-de/EidosFormer --local-dir ./eidosformer
```

## 📊 Model Specifications

| Parameter | Value |
|---|---|
| **Parameters** | ~1 Billion (1B) |
| **Architecture** | Custom Transformer with Memory Modules |
| **Hidden Size** | 2048 |
| **Layers** | 24 EidosBlocks |
| **Attention Heads** | 16 (Grouped Query Attention) |
| **Intermediate FFN** | 5632 (SwiGLU) |
| **Position Encoding** | RoPE with NTK/Instruct Scaling + Temporal Tokens |
| **Max Context Length** | 8192 tokens |
| **Embedding Dim** | 2048 (tied to LM head) |
| **Memory Stores** | Episodic (FAISS) + Semantic (EMA centroids) |

## 🧪 Training Details

- **Framework**: PyTorch with custom memory backends
- **Optimizer**: AdamW with cosine annealing and warmup
- **Checkpoint Steps**: Trained from step 1000 through ~98,000 steps
- **Final Checkpoint**: `1b_final.pt` (recommended for inference)
- **Intermediate Checkpoints**: Available (`1b_ckpt_*pt`) for reproducibility

## 🔬 Research Foundations

EidosFormer synthesizes mechanisms from:

1. **[Memorizing Transformers](https://arxiv.org/abs/2003.07862)** (Wu et al., 2022) — differentiable memory with kNN retrieval
2. **[Compressive Transformers](https://arxiv.org/abs/1907.01014)** (Rae et al., 2019) — multi-resolution episodic-to-semantic consolidation
3. **[MemoryBank](https://arxiv.org/abs/2306.xxxx)** (Zhong et al., 2023) — recency-and-frequency-based recall dynamics

## 📝 Citation

If you use EidosFormer in your research:

```bibtex
@misc{eidosformer2026,
  title={EidosFormer: Memory-Native Transformers with Multi-Store Consolidation},
  author={Dixit, Himanshu},
  year={2026},
  publisher={Hugging Face},
  url={https://huggingface.co/Himan-de/EidosFormer}
}
```

## 📄 License

**Proprietary — Contact Author for Commercial Use**

This model is released under a custom license. For commercial usage, licensing inquiries, or collaboration opportunities, please reach out via the discussions tab on Hugging Face.

For academic research use, please request access through the gated repo link above.

## ⚠️ Known Limitations

- Episodic memory retrieval quality depends on FAISS index construction (flat L2 for best accuracy, HNSW for speed)
- Semantic consolidation is gradient-free during inference; quality depends on training-time consolidation ratio
- Memory store capacity grows with sequence length; consider pruning strategies for production deployments

## 🙏 Acknowledgments

- Built upon the excellent Transformers library by Hugging Face
- FAISS vector search library (Meta AI) for efficient kNN retrieval
- Inspired by: Wu et al. (Memorizing Transformers), Rae et al. (Compressive Transformers), Zhong et al. (MemoryBank)

## 📬 Contact

For questions, collaboration, or licensing inquiries:

- Open a [Discussion](https://huggingface.co/Himan-de/EidosFormer/discussions)
- Visit the [GitHub repo](https://github.com/himan/eidosformer)

---

<p align="center">
  <i>Memory is the foundation of intelligence.</i><br>
  — EidosFormer Research Team, 2026
</p>