Is this abliterated or derestricted?
Is this vanilla abliterate or you have also applied norm-preservation and biprojection update?
The latter result in better quality usually
vanilla but it still has issues , are u able to infer ? ortho weight direction done
My plan was to extract a LoRA from the difference of this model and the vanilla through SVD decomposition of the weight differences (example: mergekit LoRA extraction).
This way it is possible to launch it coupled with unsloth dynamic 2bit quants in llama.cpp as LoRAs can be converted in gguf files. The problem is the huge disk space for the difference, and I cannot rent a large disk space server or delete a half of my SSD.
Hmm, technically, if the weights in the shards perfectly correspond to the other shards, this extraction can be done in streaming fashion!
Download shard 1 -> download shard 1* -> substract all shard 1 weights from the shard 1* weights -> extract LoRA for each weight in the difference through SVD -> discard the downloaded shards -> proceed to downloading shards until all are processed -> save the LoRA -> convert the LoRA to .gguf -> launch 2bit unsloth quant with LoRA -> test the model
Yeah, seems like a solid plan. Though may need some debugging and reliable failsafe coding 🥴