Quantizing on CPU vs GPU
Something I just came across playing with these new formats. With w4a4 my CPU and GPU will create checkpoints that create different images. If I dequantize them and compare the weights they have a tiny discrepancy between the two ~1e-4 relative error due to I assume floating point differences. Might be something worth checking out on your end too, quants made on my GPU were colder. one was warmer the other colder, the prompt included instructions for a sign on the fence that read "SUNNY DAYS". one focused on the word "SUNNY" one focused on "DAYS". one image the subject is angled inward left, the other angled right. the background shading on one draws your eyes left and the other the opposite side. it's not exactly like a mirror it's more like there was a wave on the baseline 0 and one used the crest and one used the trough