| --- |
| language: |
| - en |
| - es |
| tags: |
| - pytorch |
| - custom-code |
| - text-generation |
| - conversational |
| - moire-attention |
| - biological-ai |
| license: mit |
| --- |
| |
| # MoireFormer (104.9M Proof-of-Concept) |
|
|
| This repository hosts the PyTorch weights (`moire_phase2_weights_final.pt`) for **MoireFormer**, |
| a fundamentally new neural network architecture that replaces standard scalar dot-product attention with **Moiré phase-interference wave mechanics**. |
|
|
| Instead of computing attention via `Q · K^T`, this model splits token embeddings into amplitude and phase |
| (`q_amp`, `q_phase`) and computes attention through geometric wave resonance (`q_real * k_real + q_imag * k_imag`). |
| This proves that artificial intelligence can be trained using the continuous, biological wave-geometry observed |
| in human EEGs. |
|
|
| 🔗 **GitHub Repository (Code & Inference):** [anttiluode/MoireFormer](https://github.com/anttiluode/MoireFormer) |
| 🔗 **Theory & Clinical Proof:** [anttiluode/Geometric-Neuron](https://github.com/anttiluode/Geometric-Neuron) |
|
|
| ## Model Details |
| * **Architecture:** MoireGPT (Custom Transformer Bolt-on) |
| * **Size:** 104.9M Parameters |
| * **Structure:** 8 Layers, 8 Heads, 768 Embedding Dimension |
| * **Capabilities:** Coherent bilingual (English/Spanish) grammar, persona adoption (Assistant), structural instruction following. |
| * **Disclaimer:** At ~100M parameters, this is a proof-of-substrate, not a knowledge oracle. It demonstrates that wave |
| fields can learn discrete human syntax, but it will hallucinate factual data due to its small parameter count. |
|
|
| ## ⚠️ How to Use (Read Before Downloading) |
| Because this is a novel mathematical architecture, **you cannot load this model using the standard Hugging Face `AutoModel` pipeline.** |
|
|
| To run inference, you must download these weights and run them through the custom Moiré architecture provided in the |
| GitHub repository. |
|
|
| ### Step-by-Step Instructions: |
|
|
| **1. Clone the GitHub Repository:** |
| ```bash |
| git clone [https://github.com/anttiluode/MoireFormer.git](https://github.com/anttiluode/MoireFormer.git) |
| cd MoireFormer |
| 2. Download the Weights: |
| Download moire_phase2_weights_final.pt from the Files and versions tab of this Hugging Face repository and place |
| it in your cloned MoireFormer folder. |
| |
| 3. Run the Chat Interface: |
| |
| pip install torch transformers datasets |
| |
| python moire_chat.py --weights moire_phase2_weights_final.pt --size large |
| |
| Training Curriculum |
| |
| The model was trained in two continuous phases to demonstrate that wave-fields avoid catastrophic forgetting via |
| |
| phase-locking (destructive and constructive interference): |
| |
| Phase 1 (Base Geometry): 15 Epochs on a mixed dataset of Databricks Dolly-15k, WikiText-2, and OpenAssistant. |
| |
| This established the foundational phase-space for English and conversational structure. |
| |
| Phase 2 (Phase-Space Expansion): 5 Epochs finetuning on the Guanaco dataset to refine logical geometry |
| |
| and instruction-following, organically expanding the model's topological complexity without overwriting previous data. |
| (Perhaps?) |
