| # BERT-Thetis: Geometric BERT Models | |
| This repository contains BERT-Thetis models with deterministic crystal embeddings. | |
| I don't like what raw geo-simplex did to Bert without full cantor-stairs control. | |
| I'm currently working out a way to negate the need for backprop by integrating elements of David, | |
| but the process isn't immediate. David WORKS because of feature compatability, so enabling education with this | |
| compatability into other systems is paramount to rapid learning. | |
| Eliminating full backprop will be a very time consuming and systems rigorous refactoring of each mathematical element into flow-geometric diffusion. | |
| David makes this possible, but the possibility requires many steps between here and a full realized restructuring. | |
| This will enable a new realm of experimentation and present it's own optimization issues, while simultaneously eliminating a large experimental | |
| overhead that backprop requires due to the hierarchical climb and return structure. | |
| This version was using the older geometric vocabulary system with standard backprop as a preliminary test and it didn't do very well. | |
| The next will feature a fully robust cantor stairing system with the vit-beatrix cohesion and full learning k-simplex cantor stairway embeddings, | |
| this variation will still have backprop but there will be an additional head and complexity analysis tool for geometric stability testing with divergent pathways. | |
| Likely the followup variation will utilize a full David-inspired shunt network that coalesces multiple Bert variants together while simultaneously acting as tiny | |
| experts in a form of MOE that should enable at least 5 variants of alternative bert models to intercommunicate opinions. | |
| Even without my own version of Bert, this can already happen with David I just haven't set it up. | |
| Backprop is both a glue and a burden to independent research, so I'll do my best to both mitigate it and keep solid cohesive responses to my bert variants. | |
| Some will work, some will not. | |
| This one, didn't work very well. However, it was not a completely useless experiment. | |
| ## 📁 Repository Structure | |
| ``` | |
| AbstractPhil/bert-thetis-tiny-wikitext103/ | |
| ├── bert-thetis-tiny-wikitext103/ | |
| │ └── YYYY-MM-DD_HH-MM-SS/ (training run timestamp) | |
| │ ├── best/ (best validation checkpoint) | |
| │ ├── final/ (final checkpoint) | |
| │ └── step-N/ (intermediate checkpoints) | |
| ``` | |
| ## 🌊 What is BERT-Thetis? | |
| BERT-Thetis replaces traditional learned embeddings with **deterministic crystal structures**: | |
| - **Beatrix Staircase Encodings**: Zero-parameter positional structure | |
| - **Character Composition**: Learnable semantic bridge | |
| - **Crystal Inflation**: Deterministic 5-vertex simplex generation | |
| This reduces vocabulary parameters by ~95% while maintaining performance. | |
| ## 🚀 Quick Start | |
| ```python | |
| from geovocab2.train.model.core.bert_thetis import ThetisConfig, ThetisForMaskedLM | |
| # Load model | |
| config = ThetisConfig.from_pretrained("AbstractPhil/bert-thetis-tiny-wikitext103") | |
| model = ThetisForMaskedLM(config) | |
| ``` | |
| ## 📚 Resources | |
| - **Repository:** [github.com/AbstractEyes/lattice_vocabulary](https://github.com/AbstractEyes/lattice_vocabulary) | |
| - **Author:** AbstractPhil | |
| --- | |
| **Latest Run:** 2025-10-13_20-09-33 | |
| **Model Variant:** bert-thetis-tiny-wikitext103 | |