/home/henry/.cache/uv/archive-v0/fRk9qAw51B-A84KDUWQdY/lib/python3.12/site-packages/transformers/quantizers/auto.py:239: UserWarning: You passed `quantization_config` or equivalent parameters to `from_pretrained` but the model you're loading already has a `quantization_config` attribute. The `quantization_config` from the model will be used.However, loading attributes (e.g. ['run_compressed']) will be overwritten with the one you passed to `from_pretrained`. The rest will be ignored.
  warnings.warn(warning_msg)

Compressing model:   0%|          | 0/3136 [00:00<?, ?it/s]
Compressing model:   2%|▏         | 59/3136 [00:00<00:05, 584.78it/s]
Compressing model:   4%|▍         | 119/3136 [00:00<00:05, 592.65it/s]
Compressing model:   6%|▌         | 180/3136 [00:00<00:04, 596.28it/s]
Compressing model:   8%|▊         | 241/3136 [00:00<00:04, 598.01it/s]
Compressing model:  10%|▉         | 301/3136 [00:00<00:04, 598.35it/s]
Compressing model:  12%|█▏        | 362/3136 [00:00<00:04, 599.03it/s]
Compressing model:  13%|█▎        | 423/3136 [00:00<00:04, 599.69it/s]
Compressing model:  15%|█▌        | 483/3136 [00:00<00:04, 599.77it/s]
Compressing model:  17%|█▋        | 544/3136 [00:00<00:04, 600.03it/s]
Compressing model:  19%|█▉        | 605/3136 [00:01<00:04, 600.10it/s]
Compressing model:  21%|██        | 666/3136 [00:01<00:04, 599.95it/s]
Compressing model:  23%|██▎       | 727/3136 [00:01<00:04, 600.29it/s]
Compressing model:  25%|██▌       | 788/3136 [00:01<00:03, 600.52it/s]
Compressing model:  27%|██▋       | 849/3136 [00:01<00:03, 600.27it/s]
Compressing model:  29%|██▉       | 910/3136 [00:01<00:03, 600.15it/s]
Compressing model:  31%|███       | 971/3136 [00:01<00:03, 600.16it/s]
Compressing model:  33%|███▎      | 1032/3136 [00:01<00:03, 600.15it/s]
Compressing model:  35%|███▍      | 1093/3136 [00:01<00:03, 600.38it/s]
Compressing model:  37%|███▋      | 1154/3136 [00:01<00:03, 600.38it/s]
Compressing model:  39%|███▊      | 1215/3136 [00:02<00:03, 600.25it/s]
Compressing model:  41%|████      | 1276/3136 [00:02<00:03, 599.93it/s]
Compressing model:  43%|████▎     | 1337/3136 [00:02<00:02, 600.04it/s]
Compressing model:  45%|████▍     | 1398/3136 [00:02<00:02, 600.33it/s]
Compressing model:  47%|████▋     | 1459/3136 [00:02<00:02, 600.59it/s]
Compressing model:  48%|████▊     | 1520/3136 [00:02<00:02, 600.17it/s]
Compressing model:  50%|█████     | 1581/3136 [00:02<00:02, 600.16it/s]
Compressing model:  52%|█████▏    | 1642/3136 [00:02<00:02, 600.14it/s]
Compressing model:  54%|█████▍    | 1703/3136 [00:02<00:02, 600.18it/s]
Compressing model:  56%|█████▋    | 1764/3136 [00:02<00:02, 600.34it/s]
Compressing model:  58%|█████▊    | 1825/3136 [00:03<00:02, 600.22it/s]
Compressing model:  60%|██████    | 1886/3136 [00:03<00:02, 600.01it/s]
Compressing model:  62%|██████▏   | 1947/3136 [00:03<00:01, 600.29it/s]
Compressing model:  64%|██████▍   | 2008/3136 [00:03<00:01, 600.39it/s]
Compressing model:  66%|██████▌   | 2069/3136 [00:03<00:01, 600.67it/s]
Compressing model:  68%|██████▊   | 2130/3136 [00:03<00:01, 600.71it/s]
Compressing model:  70%|██████▉   | 2191/3136 [00:03<00:01, 600.64it/s]
Compressing model:  72%|███████▏  | 2252/3136 [00:03<00:01, 600.67it/s]
Compressing model:  74%|███████▍  | 2313/3136 [00:03<00:01, 600.70it/s]
Compressing model:  76%|███████▌  | 2374/3136 [00:03<00:01, 600.79it/s]
Compressing model:  78%|███████▊  | 2435/3136 [00:04<00:01, 600.66it/s]
Compressing model:  80%|███████▉  | 2496/3136 [00:04<00:01, 600.69it/s]
Compressing model:  82%|████████▏ | 2557/3136 [00:04<00:00, 600.51it/s]
Compressing model:  83%|████████▎ | 2618/3136 [00:04<00:00, 600.44it/s]
Compressing model:  85%|████████▌ | 2679/3136 [00:04<00:00, 600.47it/s]
Compressing model:  87%|████████▋ | 2740/3136 [00:04<00:00, 600.48it/s]
Compressing model:  89%|████████▉ | 2801/3136 [00:04<00:00, 600.23it/s]
Compressing model:  91%|█████████▏| 2862/3136 [00:04<00:00, 600.40it/s]
Compressing model:  93%|█████████▎| 2923/3136 [00:04<00:00, 600.52it/s]
Compressing model:  95%|█████████▌| 2984/3136 [00:04<00:00, 600.62it/s]
Compressing model:  97%|█████████▋| 3045/3136 [00:05<00:00, 600.72it/s]
Compressing model:  99%|█████████▉| 3106/3136 [00:05<00:00, 600.57it/s]
Compressing model: 100%|██████████| 3136/3136 [00:05<00:00, 600.03it/s]

Decompressing model:   0%|          | 0/3136 [00:00<?, ?it/s]
Decompressing model:   3%|▎         | 82/3136 [00:00<00:03, 818.61it/s]
Decompressing model:   8%|▊         | 259/3136 [00:00<00:02, 1373.70it/s]
Decompressing model:  14%|█▎        | 428/3136 [00:00<00:01, 1515.23it/s]
Decompressing model:  19%|█▉        | 594/3136 [00:00<00:01, 1571.89it/s]
Decompressing model:  24%|██▍       | 761/3136 [00:00<00:01, 1604.97it/s]
Decompressing model:  30%|██▉       | 927/3136 [00:00<00:01, 1623.26it/s]
Decompressing model:  35%|███▍      | 1093/3136 [00:00<00:01, 1633.99it/s]
Decompressing model:  40%|████      | 1259/3136 [00:00<00:01, 1639.46it/s]
Decompressing model:  45%|████▌     | 1425/3136 [00:00<00:01, 1644.19it/s]
Decompressing model:  51%|█████     | 1591/3136 [00:01<00:00, 1647.21it/s]
Decompressing model:  56%|█████▌    | 1757/3136 [00:01<00:00, 1649.88it/s]
Decompressing model:  61%|██████▏   | 1922/3136 [00:01<00:00, 1649.44it/s]
Decompressing model:  67%|██████▋   | 2087/3136 [00:01<00:00, 1648.88it/s]
Decompressing model:  72%|███████▏  | 2252/3136 [00:01<00:00, 1648.31it/s]
Decompressing model:  77%|███████▋  | 2417/3136 [00:01<00:00, 1646.31it/s]
Decompressing model:  82%|████████▏ | 2582/3136 [00:01<00:00, 1645.62it/s]
Decompressing model:  88%|████████▊ | 2747/3136 [00:01<00:00, 1643.51it/s]
Decompressing model:  93%|█████████▎| 2912/3136 [00:01<00:00, 1644.80it/s]
Decompressing model:  98%|█████████▊| 3077/3136 [00:01<00:00, 1642.02it/s]
Decompressing model: 100%|██████████| 3136/3136 [00:01<00:00, 1614.72it/s]
shard shard_00000.pt: cumulative 73031 tokens
shard shard_00001.pt: cumulative 142706 tokens
[W720 13:34:51.273718740 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 618659840 bytes (free: 165085184, total: 25292898304).
shard shard_00002.pt: cumulative 221592 tokens
{
  "quant": "w4a16",
  "shards": 3,
  "tokens": 221592,
  "kl_mean_nats": 0.03984629407661169,
  "kl_median_nats": 0.00012320266978349537,
  "kl_p95_nats": 0.18111330270767212,
  "kl_max_nats": 15.414010047912598,
  "top128_jaccard_mean": 0.7149058585433935,
  "top1_agreement": 0.9574262608758439,
  "captured_mass_bf16": 0.9997188871511677,
  "captured_mass_quant_at_bf16_support": 0.9995829765622533
}