Add direct native int8 convrot quantization of v1 test2 with mseclip

#8
by LokkenJP - opened

This PR adds a direct native int8 convrot quantization of v1 test2 with mseclip scales instead of using absmax. Very small quality increase (around 2.2% overall error reconstruction improvement over native kitchen int8 convrot with absmax).

Labeled HQ due to selective protection of edges in source bf16 precision (blocks 0,1,47-49), preserving all other tensors in their original source precision.

My usual smaller activation aware quantizations for lower memory requirements and lower-end GPUs with improved quality will come next.

Actual VRAM usage depends on the workflow and other loaded models.

A full model card and standalone releases will follow only once gating for S3 is lifted.

TenStrip changed pull request status to merged

Sign up or log in to comment