File size: 3,004 Bytes
1a630f0 b8ba86d 1a630f0 b8ba86d 1a630f0 b8ba86d 1a630f0 b8ba86d 1a630f0 b8ba86d 1a630f0 b8ba86d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 | ---
language:
- en
- es
tags:
- pytorch
- custom-code
- text-generation
- conversational
- moire-attention
- biological-ai
license: mit
---
# MoireFormer (104.9M Proof-of-Concept)
This repository hosts the PyTorch weights (`moire_phase2_weights_final.pt`) for **MoireFormer**,
a fundamentally new neural network architecture that replaces standard scalar dot-product attention with **Moiré phase-interference wave mechanics**.
Instead of computing attention via `Q · K^T`, this model splits token embeddings into amplitude and phase
(`q_amp`, `q_phase`) and computes attention through geometric wave resonance (`q_real * k_real + q_imag * k_imag`).
This proves that artificial intelligence can be trained using the continuous, biological wave-geometry observed
in human EEGs.
🔗 **GitHub Repository (Code & Inference):** [anttiluode/MoireFormer](https://github.com/anttiluode/MoireFormer)
🔗 **Theory & Clinical Proof:** [anttiluode/Geometric-Neuron](https://github.com/anttiluode/Geometric-Neuron)
## Model Details
* **Architecture:** MoireGPT (Custom Transformer Bolt-on)
* **Size:** 104.9M Parameters
* **Structure:** 8 Layers, 8 Heads, 768 Embedding Dimension
* **Capabilities:** Coherent bilingual (English/Spanish) grammar, persona adoption (Assistant), structural instruction following.
* **Disclaimer:** At ~100M parameters, this is a proof-of-substrate, not a knowledge oracle. It demonstrates that wave
fields can learn discrete human syntax, but it will hallucinate factual data due to its small parameter count.
## ⚠️ How to Use (Read Before Downloading)
Because this is a novel mathematical architecture, **you cannot load this model using the standard Hugging Face `AutoModel` pipeline.**
To run inference, you must download these weights and run them through the custom Moiré architecture provided in the
GitHub repository.
### Step-by-Step Instructions:
**1. Clone the GitHub Repository:**
```bash
git clone [https://github.com/anttiluode/MoireFormer.git](https://github.com/anttiluode/MoireFormer.git)
cd MoireFormer
2. Download the Weights:
Download moire_phase2_weights_final.pt from the Files and versions tab of this Hugging Face repository and place
it in your cloned MoireFormer folder.
3. Run the Chat Interface:
pip install torch transformers datasets
python moire_chat.py --weights moire_phase2_weights_final.pt --size large
Training Curriculum
The model was trained in two continuous phases to demonstrate that wave-fields avoid catastrophic forgetting via
phase-locking (destructive and constructive interference):
Phase 1 (Base Geometry): 15 Epochs on a mixed dataset of Databricks Dolly-15k, WikiText-2, and OpenAssistant.
This established the foundational phase-space for English and conversational structure.
Phase 2 (Phase-Space Expansion): 5 Epochs finetuning on the Guanaco dataset to refine logical geometry
and instruction-following, organically expanding the model's topological complexity without overwriting previous data.
(Perhaps?) |