---
language:
- en
- es
tags:
- pytorch
- custom-code
- text-generation
- conversational
- moire-attention
- biological-ai
license: mit
---

# MoireFormer (104.9M Proof-of-Concept)

This repository hosts the PyTorch weights (`moire_phase2_weights_final.pt`) for **MoireFormer**,
a fundamentally new neural network architecture that replaces standard scalar dot-product attention with **Moiré phase-interference wave mechanics**.

Instead of computing attention via `Q · K^T`, this model splits token embeddings into amplitude and phase
(`q_amp`, `q_phase`) and computes attention through geometric wave resonance (`q_real * k_real + q_imag * k_imag`). 
This proves that artificial intelligence can be trained using the continuous, biological wave-geometry observed
in human EEGs.

🔗 **GitHub Repository (Code & Inference):** [anttiluode/MoireFormer](https://github.com/anttiluode/MoireFormer)
🔗 **Theory & Clinical Proof:** [anttiluode/Geometric-Neuron](https://github.com/anttiluode/Geometric-Neuron)

## Model Details
* **Architecture:** MoireGPT (Custom Transformer Bolt-on)
* **Size:** 104.9M Parameters
* **Structure:** 8 Layers, 8 Heads, 768 Embedding Dimension
* **Capabilities:** Coherent bilingual (English/Spanish) grammar, persona adoption (Assistant), structural instruction following.
* **Disclaimer:** At ~100M parameters, this is a proof-of-substrate, not a knowledge oracle. It demonstrates that wave
fields can learn discrete human syntax, but it will hallucinate factual data due to its small parameter count.

## ⚠️ How to Use (Read Before Downloading)
Because this is a novel mathematical architecture, **you cannot load this model using the standard Hugging Face `AutoModel` pipeline.**

To run inference, you must download these weights and run them through the custom Moiré architecture provided in the
GitHub repository.

### Step-by-Step Instructions:

**1. Clone the GitHub Repository:**
```bash
git clone [https://github.com/anttiluode/MoireFormer.git](https://github.com/anttiluode/MoireFormer.git)
cd MoireFormer
2. Download the Weights:
Download moire_phase2_weights_final.pt from the Files and versions tab of this Hugging Face repository and place
it in your cloned MoireFormer folder.

3. Run the Chat Interface:

pip install torch transformers datasets

python moire_chat.py --weights moire_phase2_weights_final.pt --size large

Training Curriculum

The model was trained in two continuous phases to demonstrate that wave-fields avoid catastrophic forgetting via

phase-locking (destructive and constructive interference):

Phase 1 (Base Geometry): 15 Epochs on a mixed dataset of Databricks Dolly-15k, WikiText-2, and OpenAssistant.

This established the foundational phase-space for English and conversational structure.

Phase 2 (Phase-Space Expansion): 5 Epochs finetuning on the Guanaco dataset to refine logical geometry

and instruction-following, organically expanding the model's topological complexity without overwriting previous data.
(Perhaps?)