TenStrip

/

10Eros-Max

like 107

Test3 can overlap with a long base style prompt, as it should since it's additive and the same weights outside of some attn change. Make sure you're actually looking at literal A/B outputs v.s. the base pruned instead of just by feel. I've been carefully over-restoring them to preserve audio in the test versions. I added both the Krea2 and Wan elements as loras to put on top as well to boost it.

But you're aiming to create a NSFW model, aren't you? I mean your finetune is not more NSFW than the H3 base model. Certain "nsfw body parts" still are heavily deformed (same as in original H3).

That would need actual training and a large finetune. There is a huge problem faced though that any adverse training fine tune or lora will unwind the model's quality. It will always get worse at prompt response, lose a lot of it's dynamic character and details. I'm currently exploring what elements in the model pull weight, pull change, and what effect you get from what. Also I'm not gonna keep stating this: if you read the minimax license, I can not make the model do nsfw or claim it does nsfw period.

Thanks for making that clear. It was just that when I read 10Eros it made me think of LTX. But that was based on Sulphur. MiniMax is a completely different model.

You can always attempt to use the model a certain way and find it does things better v.s. the base model though, nothing is guaranteed or intended beyond experiments. It is exploratory since I'm grafting matching elements from previous models into all of the current training targets like attn,mlp,text.

There is a huge problem faced though that any adverse training fine tune or lora will unwind the model's quality. It will always get worse at prompt response, lose a lot of it's dynamic character and details.

I can see that people are having a hard time training adapters for H3, even turbo ones. Could you explain why fine-tuning H3 will be a problem say compared to LTX or WAN2.2? Is it attributed to the fact that it is a distilled checkpoint or is there something fundamentally different in the model design?

There is a huge problem faced though that any adverse training fine tune or lora will unwind the model's quality. It will always get worse at prompt response, lose a lot of it's dynamic character and details.

I can see that people are having a hard time training adapters for H3, even turbo ones. Could you explain why fine-tuning H3 will be a problem say compared to LTX or WAN2.2? Is it attributed to the fact that it is a distilled checkpoint or is there something fundamentally different in the model design?

Main issue as I see it is fused attn triplets and mainly inside the tokenizer, but also main block attention. LTX has attn_q through v unfused, each can be modulated independently. Not sure exactly how training on LTX interacts with them, but LTX also has gate logic, and H3 doesn't. My current theory is H3 is actually fully wild and it's constraint/reinforcement is purely through attn_k token relations since it's unified. When you train on to fused triplet attn qkv you're training each at 1.0. I think you want to train only the v band first, and maybe q. I have A/B sampling results from messing with attn_v v.s. messing with attn_k at high strength and attn_k manipulation fully ruins audio and degrades the sensibility of the model heavily. But I'm doing massive shifts, lora tuning is extremely subtle shift but tbh I don't see any reason to shift attn_k from where it is. I posted this in Ostiris discord and he just didn't see any reason to really unfuse them. But you get plenty of representation through q and v alone. The model's been honed on their massive dataset and those token relations are fine without being shifted by a set of 30 images or clips v.s. their reinforcement learning. But that's all just my current theory and what we need is the full actual architectural report/paper they said they'd release.

Main issue as I see it is fused attn triplets and mainly inside the tokenizer, but also main block attention. LTX has attn_q through v unfused, each can be modulated independently. Not sure exactly how training on LTX interacts with them, but LTX also has gate logic, and H3 doesn't. My current theory is H3 is actually fully wild and it's constraint/reinforcement is purely through attn_k token relations since it's unified. When you train on to fused triplet attn qkv you're training each at 1.0. I think you want to train only the v band first, and maybe q. I have A/B sampling results from messing with attn_v v.s. messing with attn_k at high strength and attn_k manipulation fully ruins audio and degrades the sensibility of the model heavily. But I'm doing massive shifts, lora tuning is extremely subtle shift but tbh I don't see any reason to shift attn_k from where it is. I posted this in Ostiris discord and he just didn't see any reason to really unfuse them. But you get plenty of representation through q and v alone. The model's been honed on their massive dataset and those token relations are fine without being shifted by a set of 30 images or clips v.s. their reinforcement learning. But that's all just my current theory and what we need is the full actual architectural report/paper they said they'd release.

Upon reading more on it, it seems as if fused qkv along with baked cfg due to distillation make the model rigid to finetuning. Gemma did also mention that fusing means that they are a single qkv matrix instead of three for each q,k and v as in LTX. That would mean unfusing is not possible and the only way to go about it is to retrain the model from scratch with unfused qkv.
Also, the anatomy issues of LTX can be largely due to unfused qkv along with lack of relevant data in training set