AbstractPhila PRO

Self-distillation has shown improvement. I think most importantly I've discovered a core component that can be utilized as a geometric attention, the quaternion MHA. The constellation produces all the necessary information to allow the quaternion MHA to benefit from the information in a directly utilizable fashion.

The quaternion MHA is quite the vessel. It's bulky, has multiple MHA structures, and is shockingly effective in the process. I'll be refining this head in the coming days as a composite Procrustes alignment tool.

Geometric structure has a very high amount of informational accumulation potential, so a multi-series of MHA can capture a great amount of informational processing from those elements, if the elements are curated correctly and within the specifications.

I've taken the benchmarks of the model from 50% to 86-93% spearman utilizing a quaternion-oriented attention head.

This is getting dangerously close to 99.9% mutation detection accuracy, with a model deemed 50% accurate - all by extracting geometric features from the constellation and training the ensemble head with the correct rules.

These are spearman result logits. These are in fact detecting the results.

This is the power of what I'm doing. From 50% to 90% in 48 hours with a single GPU.

Training your own alignment only requires a piece of the dataset you wish to run and about 8 hours or so. Run it, fall asleep, check on it in the morning. It'll be ready. Extract features, train your head in minutes. The spearman will be nearly perfect.

I'm currently preparing what I consider to be the final head that will need to be created. The quaternion head, which will be specifically predictive based on an ensemble of four divergent-methodology heads, each specifically tasked to solve the SVD in conjunction with the features. This system should extract any little bit of differentiation that exists. The imaginary head is the most crucial. Explaining this requires an entire paper of it's own.

I call this imaginary head the "Cletus" head, as it's inherently lesser accuracy in relation to the others. However, without it the combination does not coalesce correctly. Without the Cletus, the model does not reach full cohesion. This head is the most crucial, because it has the hardest job. It's actually the one who returned from the battlefield with the blueprint to describe everything it saw.

I expect the sheer geometric alignment alone to yield a new form of Adam tuning specific to introspective analytical alignment and with that a new format of optimizer dedicated to geometric preservation in conjunction with informational data accumulation. I also expect a new methodology for larger-buffer data movement kernel-wise, a structural boundary for SVD limitations within full spectrum, a substructure measured collapse state of SVD when projected, and multiple other models that will have hiccups and growing pains.

These tools are all building to the end-state format, which will express everything simultaneously in order to combine the necessary data from many many forms of models together, without requiring direct tooling to each model simultaneously.

Such finalized tools will include a reusable pretrained geometric patchwork that exhibits all the necessary traits of a geometric structure in it's frozen state, capable of being finetuned quickly into any other state, or simply utilized as a lookup beacon with the correct geometric transformer alignment.

The geometric transformer, which is specifically a revamped format for the transformer intentionally designed with the structural preservation of the overarching structure in mind, rather than falling directly to the naturalistic entropy of immediate solution over larger-scale contribution. This system will not replace rope, it will contribute to the concept of long-concept preservation and work hand-in-hand with systems like rope, attention, and original transformers simultaneously. ROPE based models will benefit most from this structure, as they are already trained intrinsically with alignment and rotation at their cores.

The geometric transformer by design takes nth inputs in as variant states, and those are transformed internally. Utilizing this by it's default state will yield by design, but it will require tuning and curation for specific use cases no matter which case. This is conceptually familiar to those who use transformers, and simultaneously intimidating to those who understand what I'm describing I'd think. I myself am a little intimidated that I'm this close as-is.

There are multiple other prototypes at work all leading to the geometric transformer, which will be both an empirically superior utility to any of the utilities I currently use, and embody the very essence of the geometric structure that I'm currently working with as a full trainable data mutation operation - meant to directly attenuate the structure of the observation, to the expectation of the autograd and gradients.

Getting pretty close to a few pieces, but not there yet.

I've spent the better part of the day refactoring the geolip-core github code so it would be better inline with the actual findings, and I'm currently having claude build the models using the geofractal router system.

With that I'll enable a dummy-clause structure, so the component code files can be snipped out and work standalone as necessary. Due to the geofractal router's rigidity as a structure, geovocab's problematic multi-layer formatting for formulas that often have strange hardware quirks, and a few of the more reusable systems requiring modularity - I'll just build it like this to allow for just... USE MINE INSTEAD mentality.

You'll be able to just snip them out and use them, like many of the representations within the "models" experiment folders. They are simply standalone that may or may not snap onto pieces of the larger wholes.

geolip-core will be built specifically with the geofractal router structure in mind, inheriting it's strengths, weaknesses, and hardware control - while simultaneously having a wrapper that simply says: use standard pytorch instead.

Using standard pytorch will disable much of the functionality, but the components in their standalone forms WILL WORK. The pipelines are another story.

Keep my attribution and naming in the comments please, this is a testament to a very long series of research that resulted in solutions to problems rather than trying to introduce more problems. Attribution is all I wish, and you can make your fortune from that. I believe many of the great minds of the past would agree; Nikola Tesla I believe would agree, accreditation is all I want, but not for my name - for them and my humble contribution. The greats who put the pieces together and solved the biggest problems.

As we grow, the shadows of giants are cast upon the surfaces of life and stone. Work hard, progress steadily, expand your mind, build your skills, and by the time you have any time to look down... you will be casting a shadow of your own. As we grow old and our shadows grow, others are born and see the cast shadows. Encouraging them through the same process is all I know how to do.

Wrote a triton kernel to approximate SVD at around 15000x on blackwell architecture, while the standard torch.linalg.svd basically sits in a swamp of slow.

It only tackles small kernel sizes for now, 3x3, which is the current experiment's encoder paradigm. Standard SVD causes death by a thousand cuts when using small matrixes. Smaller matrixes provide a much more robust access to certain elemental linkages on many spectrum.

The formula isn't perfect. It's absolutely lightning quick though. The svd_trison.py file has a profiler.

https://huggingface.co/AbstractPhil/geolip-hypersphere-experiments/blob/main/spectral/notebooks/experiment_2_manifold_structures.ipynb

https://huggingface.co/AbstractPhil/geolip-core/blob/main/svd_triton.py

Incoming the geofractal router structure.

It will provide the necessary hardware and software implications to create much larger structures and curate the code in a much more effective way to hardware control than simply leaving it up to random chance or ai.

As I expand the system I will be heavily testing to include systems like Ulysses, accelerate, and more to encompass a larger array of learning. This will be crucial to building a proper bert trainer as well.

Might get lost down this scattering point system, it seems highly responsive and deterministic. I might make some PRs to the kymatio version for speed as well.

I'm seeing a large series of discrepancies between what Claude THOUGHT was correct, and what the experiments that Claude is yielding are producing.

This will take a large series of micro managing and consultation with multiple AIs to get working formulas for some of these more complex geometric substrates.

The calculations are exceedingly complex, so I will be adopting the https://github.com/kymatio/kymatio repo directly into the geofractal repo soon.

geolip-captionbert's captions are still cooking, it's going to need more days.

Until then I'm restoring an old prototype named vit-zana and reforming her into geolip-zana. This vit was built on the old pentachoron vocabulary which only contained 5 anchors of frozen utility, this specific version houses the full nth anchor structural hypersphere, which we'll test for behavior within the new spectrum of utility.

I've been working out something akin to a liars paradox solver. This should help the berts form their own internal adjudication system for determining knot potential and the utility of solution. This should allow the bert soup to be more cohesive.

This will only be a fraction of the complexity of something like an LLM has, but it would endow the bert with a bit more information curation.

As of today they are still too agreeable, so I'll need to run hard negatives to ensure captions are correct as a processing substrate for the 180m caption train.

That's not the case. I'm training adjacent versions with different datasets with R1 100%, the objective is just too easy.

Weight decay corrupts the geometry it cannot be used.

Early stopping isn't necessary.

Everything you mentioned is very basic, but thank you for your effort.

Dino losses appear to align on many fronts to my research, which means I may be able to adopt some and adjust others.

It's possible the Dino team figured out a solution that I'm currently architecting, and it might lead directly to where I need to be, while enabling many of my geometric research benefits in the process.

Anchor alignment is analogous with the Dino iBOT loss spectrum, but they are two sides of the same coin. Combining the two may provide a much more robust geometric objective. The paper collapsed it, but... I think it was missing the anchoring, I'll run some tests.

KoLeo loss from DinoV2 may be considerably more valuable, I'll need to research V3 as well.

Screw it, I'll analyze it anyway. I might find an easy route.

It won't be 34 though, I'll prune a bunch and find the most aligned clips. Probably 3 will do for a proper prototype.

The x34 crossover was excessive noise, the continuity is very difficult to rationalize and form a cohesive anchoring for.

Predominantly the boundaries. I ran multiple scans on the COCO data and found huge deviances of composite differentiation, which means the manifold overall would be massive.

Something like trying to represent 800b params in a single 84m param space, which even the best of alignment geometric would require considerably more refined mathematics to even channel. The full process will require a multi-stage relational conjecture and interpolation between alignment positive, negative, and including a symbolic relational architecture just to make sense of the noise itself.

It could be done, and I can say for certain it could be done. It's just a matter of what the finished product between such a complex relational structure would look like, and how much information would be required to actually train it. The math lines up, the architecture lines up, but it does not have a reasonable horizon for the manifold.

Simply put, because of the boundaries, even if constricted, training 34 experts at runtime is beyond my scope. It would require full patch extraction, not just attenuation like CaptionBERT. CaptionBERT is a different form of differentiation that allows rapid pooled learning between multiple from the same family, while the x34 would require multiple pooled learning from adjacent families. Each family requires it's own patchwork size, it's own formulas for alignment based on that, and it's own specific attenuation principality based on the adjacent differentiation.

Possible yes, very possible. A true challenge and test for the architecture with a team and a series of experts, but I am but one researcher. I would be here for days figuring out how to attenuate patch14 to patch16 differences, just to yield little information based on the hypersphere alignment.

I've noticed some definite data overlap in the system - a percentage of the validation data has been trained, the R1 constraints aren't necessarily perfect in the current state. Even without the overlap, the early stage model can be directly trained to be R1@100% within 3 epochs, so this isn't the crucial fail point. The fail point is in the large scheme tests having potential overlap, and these cannot have overlap for the big train.

This overlap happened when I switched from the 200k set to the 500k set, so the full 12 million set will need a new validation target other than itself. I only ran it twice, but both times likely bled 20k images of mixed origin from the other, which likely is less than 8% or so bleedover but it's enough to taint the outcome.

I'll require another dataset to validate, something completely removed from the attribution and completely differentiated. I'm going to likely use my own dataset as validation, which is essentially a billion trash prompts that cannot simply be solved, and often make zero sense.

Even a small percentage of the validation data having been trained is enough for me to resort to extreme measures, not to mention the damn thing is reporting R1 100% all the time which is annoying me. I want to see a legitimate series of impossible combinations that cannot be represented, essentially garbage noise mixed with pure captions that the model has never learned from.

The model cannot easily solve these, which will give a perfect measure. I'd say maybe a million of these will be the best possible impossible goal.