It's surprising it works so well using a much smaller dataset, 512 x 2048, so 1 million tokens, and the loss being minimized is not KL divergence on the text generation, but layer by layer reconstruction error inside the model.
The way they learn to scale and shift blocks of parameters seems conceptually similar, but rather than then applying hard quants, backpropping steadily more aggressive soft ternerizaition using layer reconstruction loss, that seems to be the novel bit.
Do you think it's cost effective for you to fine tune weights in this way to lower perplexity / KLD further?
Thank you! My instinct is that direction will have issues with catastrophic forgetting outside the calibration dataset because it involves finetuning. I think that in order to be more faithful to the original model's post training, you need to blindly quantize the weights of the model. That way the calibration dataset fit is an emergent byproduct instead of a direct fit which more specifically directs outcomes. Dynamic quantization is always bound to overfit to some degree so I think making the least decisions about the model as possible will lead to stronger generalization.
I didn't really see it as fine tuning, because the backprop signal is not coming from the text. I was thinking it was more like just using sample data to create a diverse set of N reference forward passes, so you can ensure each layers behaviour hasn't drifted too far by aggressive quantisation.
However, if it is as aggressive and impactful on KL as you suggest, is there a softer incarnation, so during your normal quantisation process you effectively have a choice for each weight, up or down, and rather than a simple absolute difference measure, you could use layer by layer reconstruction loss (because it's cheaper than checking the full text generation KL divergence on a massive dataset) to decide on a weight by weight basis to round up or down? And in theory better layer by layer reconstruction will reduce KL divergence overall?
My take is as good as yours since I haven't studied their work in detail, its just my instinct as I said. I'm curious if you're right.
So as I mentioned my belief is that any decision made that is not data agnostic is bound to overfit. The representations created by MLPs are very durable as we see here, but fundamentally the precision is eventually needed for some critical unknown weights. We can't reveal what those weights are with any amount of economical analysis up to a full retrain because the function of gradient descent is the accumulation of all steps. This is of course obvious, but what I am saying is that any training is essentially changing that function to bias toward the chosen dataset, which will lead to it erasing valuable representations except in very sparse circumstances. So you are right that this direction will improve results, but the question is which results?
I've studied LLMs at very small constrained sizes considerably which led to my work here. This study has changed my perspective on how representations form and change over training fundamentally compared to established opinions. You can check out some of my previous work:
