🧠 Athena Modern Hopfield Network SLM

Athena MHN SLM is an experimental small language model built using a Modern Hopfield Network-inspired attention mechanism combined with a recurrent Transformer-style architecture.

This project explores how associative memory (Hopfield networks) can be used to improve reasoning and pattern retrieval in language models.

---

⚙️ Architecture Overview

The model is a hybrid of:

• 🔷 Modern Hopfield Network-style attention
• 🔷 Transformer embedding + positional encoding
• 🔷 Recurrent layer execution (multi-loop depth)
• 🔷 Adaptive computation halting

Key Components

• Embedding size: 512
• Layers: 4 MHN blocks
• Attention type: Hopfield-style energy-based retrieval (softmax attention form)
• Recurrent loops per layer: up to 3
• Context length: 1024 tokens
• Adaptive halting mechanism for dynamic computation depth

---

🧠 Modern Hopfield Mechanism

Each layer performs associative memory retrieval:

• Queries represent current token states
• Keys/Values are derived from the same hidden states
• Attention is computed as energy-based similarity
• Output is retrieved memory patterns from the state space

This makes the model behave like a continuous associative memory system.

---

📚 Training Setup

• Dataset: tatsu-lab/alpaca
• Training samples: 15,000 instruction-response pairs
• Format:
  • Instruction
  • Input (if any)
  • Output response

---

🔄 Recurrent Depth Mechanism

Instead of stacking many layers, each MHN block is executed multiple times:

• Improves reasoning depth
• Enables iterative refinement of representations
• Reduces need for very deep architectures

---

⏱️ Adaptive Halting

The model includes a halting head that dynamically stops computation when:

• Confidence threshold is reached (> 0.96)
• Reduces unnecessary computation
• Improves efficiency during inference

---

🧪 Status

⚠️ Experimental Research Model

• Architecture is actively evolving
• Not yet optimized for production use
• Intended for research and prototyping only

---

🚀 Intended Use Cases

• Research on Modern Hopfield Networks
• Associative memory-based language modeling
• Recurrent depth transformer experiments
• Instruction-following prototype systems

---

📦 Version History

v2

• Introduced True Modern Hopfield-style attention block
• Improved associative memory behavior
• Stabilized recurrent loop architecture
• Added adaptive halting mechanism

---

🔗 Model Repository

https://huggingface.co/sasindumalhara/Athena

---

license: apache-2.0