Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
AbstractPhil 
posted an update 16 days ago
Post
107
nano-castor and nano-pollux - (4 variants total) are training complete for the story versions. AbstractPhil/alephllm-nano-castor-pollux

Trained with an identical dataset until framing as the mini-beatrix-2s variants, and finetuned for context and framing as an expected behavior.

The context windows and framing show the models can capture meaningful storage, read, copy, and recall capacity at 4096 byte depth. They aren't particularly smart, but they are functional, and even capable at driving Anima sliders in a meaningfully similar way as the mini-beatrix-3 variant can.

AbstractPhil/alephllm-nano-castor-pollux

Anima sliders often do not have a bi-directional capacity, so the multi-slider arrays were created. Thus allowing the model to rebound naturally as expected when necessary, but this still isn't strong enough.

Up next is the full frame trained variant of the champion; nano-pollux-dual-hub and nano-castor. Both of them are the remaining champions. Neither canon models showed similar capacity as the dual-hub's capacity yielded, so we have our champions.

Before continuing to the micro variant, these two will be retrained and issued a full education in art. Everything from color theory, to structural awareness, to semantic relations, techniques, and everything between.

Our little nano twins here, will be a cooperative pair meant for the first diffusion edit experiments.

There were quite a few experiments, many yielded interesting potentials and information. The most powerful variant being the beatrix 3 as a conditioner, yet the nano variants showed a similarly powerful control.

AbstractPhil/geolip-beatrix-anima

Even just untrained regularization training on Anima shows yield. The model is pretty underfitted, which is both helpful and not helpful for this train.

The frame-centric finetune is in progress, and it broke the readme for the repo so I'll fix it momentarily.

image

image

As it stands the full frame training is holding pretty effectively.

image

The nano framed variants are heavily dependent on the banks, which is both good and bad. The lack of shared structure is a bit of an efficiency problem, as it takes longer to learn and solidify the depth content, but since the model is learning framed depth by structure it's holding either way. The training process is working so far.

The construction begins on a specific chain of thought dataset built around conditioning for image generation and VLM classification chain of thought. The dataset will be selectively designed for image curation, editing, associative reasoning, classification, and more.

Being text-specific the dataset will contain multiple direct conversions of known popular dataset caption sets into CoT formatting utilizing symbolic associative reasoning. If images do exist, the repo will be directly linked in the readmes, or the image pointed directly at a URL depending the ruling.

The entire purpose of this dataset is to provide a specific set of framed curation datasets, built specifically around populating framed portions of context window information. Think about it like curated thought pathology meant to encourage stable thought structure specific to images. In short, the dataset is being built around multi-parallel thinking, but will function as standard CoT when trained on other models.

There will be multiple prototype trains based on this topic with the entire goal meant to provide strong image control editing and capacity for editing using runtime conditioning, rather than requiring a multi-tuned retrain of models post finetune and post training.

The goal is to enable image editing guidance capacity to every applicable CoT model with the power of a single chain of thought dataset. Image training directly on diffusion models within specific parameters can enable this editing to be participated in with minimal supervision, even when models use entirely different embedding, encoding, lookup, or position systems.

There are no guarantees this will work, and the experimentation will be ongoing while curating and designing. The dataset will be simple at first, then expanding outward heavily as time progresses.

The research shows that stable and curated CoT can be stabilized and made reasonably learnable with simple mitigated finetuning, and the experiments based on this dataset will yield those results publicly.

nano-pollux-dual-v1.2 is done, full frame train is in.

image

Model eats them up. Control is the nano-castor model having never seen frames.

image

Experimental thought dataset should be pretty beefy now. Not sure how well it'll work unless I train a vit with the information using cross-contrast, so I'll run some finetunes and benchmarks. The nano models don't understand thinking, so I'm not really sure what'll happen if I just raw feed it. I assume overfitting from just the data if I run multi-epoch due to the lack of CoT training, so the model's CoT will likely need to be conditioned ahead of time to be generic CoT as per my assumption. This isn't a guaranteed requirement, it may very well be the opposite - where CoT can be taught using symbolic objective representation rather than needing some sort of CoT dataset to baby-step into the system.

I'll run some experiments and tests, then figure out some good methodologies for the trie structure.

Castor learned clean chain of thought.

image

Got a bit overzealous on my fable, opus, and sonnet usage - it's pretty much gone. I'll be setting up something with Opus but the usage is limited so nows a good a time as any to take a break until tomorrow.

Pollux should finish soon, then the dual thinking nanos should be functional.

image

Pollux finished.

image

image

image

image

image

So... yeah the little models seem to learn most text related tasks on this scale. Individual structures with training runs seem to have very little problems learning them.

9 million params isn't very much, but it's more than enough for this task apparently. I'll be finetuning something much stronger with the dataset sonnet, opus, and fable engineered for the image thinking task. I'm thinking a powerful qwen or something like that. Something that knows images already.

That should be an appropriate target. Something with more brains than an atom.

In this post