README.md

---
license: apache-2.0
datasets:
- HuggingFaceFW/fineweb-edu
- eliceai/korean-fineweb-edu-demo
language:
- en
- ko
base_model:
- RetentionLabs/TTT-Linear-1.3B-Base-Pile-8k
- Qwen/Qwen3-4B-Thinking-2507
base_model_relation: merge
pipeline_tag: text-generation
library_name: transformers
tags:
- Test-time Training
- Memory-augemted Transformer
---

# TTTPilot-Q-5B-Thinking-MAC

**A Pilot Implementation of Titans MAC Architecture**

Combining [TTT-Linear-1.3B-Base-Pile-8k](https://huggingface.co/test-time-training/TTT-Linear-1.3B-Base-Pile-8k) and [Qwen3-4B-Thinking-2507](https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507) using the **Memory as Context (MAC)** architecture pattern.

---

## 🎯 Pilot Experiment Overview

This is an **experimental pilot** exploring how test-time training (TTT) memory layers can be combined with standard transformer cores in a modular architecture. The MAC pattern separates:

- **Memory Layers**: TTT-Linear's self-adaptation mechanism for dynamic context learning
- **Core Layers**: Qwen3's transformer decoder for reasoning and generation

### Key Idea: MAC Processing Flow

```
Input Sequence (Processed in Segments)
    ↓
┌─────────────────────────────────────┐
│  1. Read-Only Retrieval (R-Mode)   │  ← Memory layers (Q-only projection)
│     Generate memory queries          │    Enables parallel computation
└─────────────────────────────────────┘
    ↓
┌─────────────────────────────────────┐
│  2. Core Processing                 │  ← Qwen3 transformer layers
│     [Fixed Memory + Query + Input]  │    Standard attention + MLP
└─────────────────────────────────────┘
    ↓
┌─────────────────────────────────────┐
│  3. Memory Update (W-Mode)          │  ← Memory layers (full QKV)
│     Update context representations  │    Test-time adaptation
└─────────────────────────────────────┘
    ↓
Final Output (Join updated context with core output)
```

### Architecture Benefits

1. **Parallel Memory Retrieval**: Q-only projection in R-mode enables efficient segment processing
2. **Weight Tying**: `retriever.q` shares weights with `memory.q` for efficiency
3. **Modular Design**: Memory and core can be independently scaled/fine-tuned
4. **Hybrid Capabilities**: Combines TTT's adaptive learning with transformer's proven performance

---

## 📊 Model Statistics

| Component | Source Model | Layers | Parameters | Intermediate Size |
|-----------|-------------|--------|------------|-------------------|
| **Embedding & LM Head** | Qwen3-4B-Thinking | - | ~310M (vocab: 151,936) | - |
| **Memory Layers** | TTT-Linear-1.3B | 24 | ~1.3B | 5,504 |
| **Core Layers** | Qwen3-4B-Thinking | 36 | ~4B | 9,728 |
| **Total** | Combined MAC | **60 layers** | **~5.6B** | Mixed |

**Hidden Size**: 2048 (from TTT-Linear)  
**Attention Heads**: 32 (memory), 32/8 GQA (core)  
**Context Length**: Up to 262K tokens (Qwen3's max)  
**Precision**: BFloat16

---

## 🏗️ Architecture Details


### Memory Module (TTT-Linear)

- **Purpose**: Dynamic context adaptation through test-time training
- **Key Components**:
  - Self-adaptation layers with learnable neural memory
  - Momentum-based learning rate gates
  - Q/K/V projections with RoPE
  - Mini-batch processing (chunk_size=16)
- **Special Features**:
  - Weight tying between retriever and full memory
  - Shared Q/K projections with separate conv layers
  - Learnable token-wise learning rates

### Core Module (Qwen3)

- **Purpose**: High-capacity reasoning and generation
- **Key Components**:
  - Multi-head attention with Grouped Query Attention (GQA)
  - SwiGLU MLP activations
  - RMSNorm for layer normalization
  - RoPE with theta=5,000,000 for long context
- **Special Features**:
  - 8 KV heads for efficient inference
  - No attention bias
  - Sliding window attention support

### Fixed Persistent Memory

- **Size**: 64 tokens × 2048 dimensions
- **Purpose**: Store global context/knowledge across segments
- **Initialization**: Zeros (trainable parameter)

---

## 🚀 Usage

### Installation

```bash
# Install dependencies
pip install torch transformers safetensors accelerate
```

### Quick Start

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load model
model = AutoModelForCausalLM.from_pretrained(
    "./TTTPilot-Q-5B-Thinking-MAC",
    trust_remote_code=True,
    torch_dtype="bfloat16",
    device_map="auto"
)

tokenizer = AutoTokenizer.from_pretrained("./TTTPilot-Q-5B-Thinking-MAC")

# Generate text
prompt = "The future of AI is"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100, do_sample=True, temperature=0.7)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

### Advanced: Segment Processing

```python
# MAC processes inputs in segments for memory efficiency
# Segment size controlled by mini_batch_size (default: 16)

# For long sequences, the model automatically:
# 1. Chunks input into mini-batches
# 2. Processes each chunk through R-mode → Core → W-mode
# 3. Accumulates context updates across chunks

long_text = "..." * 1000  # Very long input
inputs = tokenizer(long_text, return_tensors="pt", truncation=False)
outputs = model.generate(**inputs, max_new_tokens=200)
```

---

## 🔧 Weight Conversion

To recreate this model from source checkpoints:

```bash
python convert_weights.py
```

The script:
1. Loads TTT-Linear-1.3B and Qwen3-4B-Thinking
2. Maps TTT weights → memory layers (preserving exact key names)
3. Maps Qwen3 weights → core layers (preserving exact key names)
4. Uses Qwen3's embedding & lm_head (for vocab compatibility)
5. Copies tokenizer files from Qwen3
6. Saves combined model in HuggingFace format

---

## ⚙️ Configuration

Key hyperparameters in `config.json`:

```json
{
  "model_type": "tttpilot_mac",
  "vocab_size": 151936,          // Qwen3
  "hidden_size": 2048,            // TTT-Linear
  "num_memory_layers": 24,        // TTT-Linear
  "num_core_layers": 36,          // Qwen3
  "memory_intermediate_size": 5504,   // TTT MLP
  "core_intermediate_size": 9728,     // Qwen3 MLP
  "num_attention_heads": 32,
  "num_key_value_heads": 8,       // GQA in cores
  "mini_batch_size": 16,          // TTT chunk size
  "ttt_base_lr": 1.0,             // TTT learning rate
  "fixed_memory_size": 64,        // Persistent memory tokens
  "rope_theta": 5000000,          // Long context RoPE
  "max_position_embeddings": 262144   // Max sequence length
}
```

---

## 🧪 Pilot Experiment Status

### What Works
✅ Model architecture defined  
✅ Weight conversion pipeline  
✅ Configuration files generated  
✅ Tokenizer compatibility (Qwen3)  
✅ Basic forward pass structure  

### What's Experimental
⚠️ **MAC segment processing**: Simplified in pilot, needs full TTT integration  
⚠️ **Retriever implementation**: Placeholder, requires Q-only inference mode  
⚠️ **Weight tying**: Defined but not verified in practice  
⚠️ **Memory update logic**: Full TTT adaptation step needs integration  

### Known Limitations
- Memory layers use placeholder identity functions (need full TTT code)
- No actual segment-based MAC flow yet (processes like standard transformer)
- TTT cache and Qwen3 KV cache not properly integrated
- No training/fine-tuning tested
- Generation quality not benchmarked

---

## 🎓 Research Context

This pilot implements concepts from:

1. **Test-Time Training (TTT)**: Self-supervised adaptation during inference
   - Paper: [Learning to (Learn at Test Time)](https://arxiv.org/abs/2407.04620)
   - Code: [TTT-Linear](https://github.com/test-time-training/ttt-lm-pytorch)

2. **Titans Architecture**: Modular memory-augmented patterns
   - Inspiration: Memory-as-X design patterns (MAC, MAE, MAL, etc.)
   - Idea: Separate stateful memory from stateless reasoning

3. **Grouped Query Attention**: Efficient multi-head attention
   - From: Qwen3 and modern LLMs
   - Benefit: Faster inference with minimal quality loss

---

## 📝 Citation

If you use this pilot or build upon it:

```bibtex
@misc{tttpilot-mac-2026,
  title={TTTPilot-MAC: A Pilot Implementation of Memory-Augmented-Core Architecture},
  author={Your Name},
  year={2026},
  note={Pilot experiment combining TTT-Linear and Qwen3},
  howpublished={\url{https://github.com/...}}
}
```

**Source Models**:
- TTT-Linear: [test-time-training/TTT-Linear-1.3B-Base-Pile-8k](https://huggingface.co/test-time-training/TTT-Linear-1.3B-Base-Pile-8k)
- Qwen3: [Qwen/Qwen3-4B-Thinking-2507](https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507)

---

## 📄 License

This project combines:
- **TTT-Linear** (MIT License)
- **Qwen3** (Apache 2.0 License)

Final license: **Apache 2.0** (compatible with both)

See [LICENSE](./LICENSE) for details.

---

## 🤝 Contributing

This is a **pilot experiment** for research exploration. Contributions welcome:

1. Full MAC segment processing implementation
2. TTT-Linear integration (replace placeholders)
3. Retriever Q-only mode
4. Training scripts
5. Benchmark evaluations
6. Documentation improvements

---

## 🐛 Issues & Feedback

Found a bug or have suggestions? Open an issue!

**Important Notes**:
- This is NOT a production-ready model
- Use for research/experimentation only
- Performance not guaranteed
- May require significant compute resources (5.6B parameters)

---

## 🌟 Acknowledgments

- **TTT Team** for the test-time training paradigm
- **Qwen Team** for Qwen3-Thinking model
- **Titans Architecture** inspiration from modular design patterns

---

**Status**: 🚧 Experimental Pilot - Use with caution!