| /home/henry/.cache/uv/archive-v0/fRk9qAw51B-A84KDUWQdY/lib/python3.12/site-packages/transformers/quantizers/auto.py:239: UserWarning: You passed `quantization_config` or equivalent parameters to `from_pretrained` but the model you're loading already has a `quantization_config` attribute. The `quantization_config` from the model will be used.However, loading attributes (e.g. ['run_compressed']) will be overwritten with the one you passed to `from_pretrained`. The rest will be ignored. |
| warnings.warn(warning_msg) |
|
Compressing model: 0%| | 0/3136 [00:00<?, ?it/s]
Compressing model: 2%|▏ | 59/3136 [00:00<00:05, 584.78it/s]
Compressing model: 4%|▍ | 119/3136 [00:00<00:05, 592.65it/s]
Compressing model: 6%|▌ | 180/3136 [00:00<00:04, 596.28it/s]
Compressing model: 8%|▊ | 241/3136 [00:00<00:04, 598.01it/s]
Compressing model: 10%|▉ | 301/3136 [00:00<00:04, 598.35it/s]
Compressing model: 12%|█▏ | 362/3136 [00:00<00:04, 599.03it/s]
Compressing model: 13%|█▎ | 423/3136 [00:00<00:04, 599.69it/s]
Compressing model: 15%|█▌ | 483/3136 [00:00<00:04, 599.77it/s]
Compressing model: 17%|█▋ | 544/3136 [00:00<00:04, 600.03it/s]
Compressing model: 19%|█▉ | 605/3136 [00:01<00:04, 600.10it/s]
Compressing model: 21%|██ | 666/3136 [00:01<00:04, 599.95it/s]
Compressing model: 23%|██▎ | 727/3136 [00:01<00:04, 600.29it/s]
Compressing model: 25%|██▌ | 788/3136 [00:01<00:03, 600.52it/s]
Compressing model: 27%|██▋ | 849/3136 [00:01<00:03, 600.27it/s]
Compressing model: 29%|██▉ | 910/3136 [00:01<00:03, 600.15it/s]
Compressing model: 31%|███ | 971/3136 [00:01<00:03, 600.16it/s]
Compressing model: 33%|███▎ | 1032/3136 [00:01<00:03, 600.15it/s]
Compressing model: 35%|███▍ | 1093/3136 [00:01<00:03, 600.38it/s]
Compressing model: 37%|███▋ | 1154/3136 [00:01<00:03, 600.38it/s]
Compressing model: 39%|███▊ | 1215/3136 [00:02<00:03, 600.25it/s]
Compressing model: 41%|████ | 1276/3136 [00:02<00:03, 599.93it/s]
Compressing model: 43%|████▎ | 1337/3136 [00:02<00:02, 600.04it/s]
Compressing model: 45%|████▍ | 1398/3136 [00:02<00:02, 600.33it/s]
Compressing model: 47%|████▋ | 1459/3136 [00:02<00:02, 600.59it/s]
Compressing model: 48%|████▊ | 1520/3136 [00:02<00:02, 600.17it/s]
Compressing model: 50%|█████ | 1581/3136 [00:02<00:02, 600.16it/s]
Compressing model: 52%|█████▏ | 1642/3136 [00:02<00:02, 600.14it/s]
Compressing model: 54%|█████▍ | 1703/3136 [00:02<00:02, 600.18it/s]
Compressing model: 56%|█████▋ | 1764/3136 [00:02<00:02, 600.34it/s]
Compressing model: 58%|█████▊ | 1825/3136 [00:03<00:02, 600.22it/s]
Compressing model: 60%|██████ | 1886/3136 [00:03<00:02, 600.01it/s]
Compressing model: 62%|██████▏ | 1947/3136 [00:03<00:01, 600.29it/s]
Compressing model: 64%|██████▍ | 2008/3136 [00:03<00:01, 600.39it/s]
Compressing model: 66%|██████▌ | 2069/3136 [00:03<00:01, 600.67it/s]
Compressing model: 68%|██████▊ | 2130/3136 [00:03<00:01, 600.71it/s]
Compressing model: 70%|██████▉ | 2191/3136 [00:03<00:01, 600.64it/s]
Compressing model: 72%|███████▏ | 2252/3136 [00:03<00:01, 600.67it/s]
Compressing model: 74%|███████▍ | 2313/3136 [00:03<00:01, 600.70it/s]
Compressing model: 76%|███████▌ | 2374/3136 [00:03<00:01, 600.79it/s]
Compressing model: 78%|███████▊ | 2435/3136 [00:04<00:01, 600.66it/s]
Compressing model: 80%|███████▉ | 2496/3136 [00:04<00:01, 600.69it/s]
Compressing model: 82%|████████▏ | 2557/3136 [00:04<00:00, 600.51it/s]
Compressing model: 83%|████████▎ | 2618/3136 [00:04<00:00, 600.44it/s]
Compressing model: 85%|████████▌ | 2679/3136 [00:04<00:00, 600.47it/s]
Compressing model: 87%|████████▋ | 2740/3136 [00:04<00:00, 600.48it/s]
Compressing model: 89%|████████▉ | 2801/3136 [00:04<00:00, 600.23it/s]
Compressing model: 91%|█████████▏| 2862/3136 [00:04<00:00, 600.40it/s]
Compressing model: 93%|█████████▎| 2923/3136 [00:04<00:00, 600.52it/s]
Compressing model: 95%|█████████▌| 2984/3136 [00:04<00:00, 600.62it/s]
Compressing model: 97%|█████████▋| 3045/3136 [00:05<00:00, 600.72it/s]
Compressing model: 99%|█████████▉| 3106/3136 [00:05<00:00, 600.57it/s]
Compressing model: 100%|██████████| 3136/3136 [00:05<00:00, 600.03it/s] |
|
Decompressing model: 0%| | 0/3136 [00:00<?, ?it/s]
Decompressing model: 3%|▎ | 82/3136 [00:00<00:03, 818.61it/s]
Decompressing model: 8%|▊ | 259/3136 [00:00<00:02, 1373.70it/s]
Decompressing model: 14%|█▎ | 428/3136 [00:00<00:01, 1515.23it/s]
Decompressing model: 19%|█▉ | 594/3136 [00:00<00:01, 1571.89it/s]
Decompressing model: 24%|██▍ | 761/3136 [00:00<00:01, 1604.97it/s]
Decompressing model: 30%|██▉ | 927/3136 [00:00<00:01, 1623.26it/s]
Decompressing model: 35%|███▍ | 1093/3136 [00:00<00:01, 1633.99it/s]
Decompressing model: 40%|████ | 1259/3136 [00:00<00:01, 1639.46it/s]
Decompressing model: 45%|████▌ | 1425/3136 [00:00<00:01, 1644.19it/s]
Decompressing model: 51%|█████ | 1591/3136 [00:01<00:00, 1647.21it/s]
Decompressing model: 56%|█████▌ | 1757/3136 [00:01<00:00, 1649.88it/s]
Decompressing model: 61%|██████▏ | 1922/3136 [00:01<00:00, 1649.44it/s]
Decompressing model: 67%|██████▋ | 2087/3136 [00:01<00:00, 1648.88it/s]
Decompressing model: 72%|███████▏ | 2252/3136 [00:01<00:00, 1648.31it/s]
Decompressing model: 77%|███████▋ | 2417/3136 [00:01<00:00, 1646.31it/s]
Decompressing model: 82%|████████▏ | 2582/3136 [00:01<00:00, 1645.62it/s]
Decompressing model: 88%|████████▊ | 2747/3136 [00:01<00:00, 1643.51it/s]
Decompressing model: 93%|█████████▎| 2912/3136 [00:01<00:00, 1644.80it/s]
Decompressing model: 98%|█████████▊| 3077/3136 [00:01<00:00, 1642.02it/s]
Decompressing model: 100%|██████████| 3136/3136 [00:01<00:00, 1614.72it/s] |
| shard shard_00000.pt: cumulative 73031 tokens |
| shard shard_00001.pt: cumulative 142706 tokens |
| [W720 13:34:51.273718740 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 618659840 bytes (free: 165085184, total: 25292898304). |
| shard shard_00002.pt: cumulative 221592 tokens |
| { |
| "quant": "w4a16", |
| "shards": 3, |
| "tokens": 221592, |
| "kl_mean_nats": 0.03984629407661169, |
| "kl_median_nats": 0.00012320266978349537, |
| "kl_p95_nats": 0.18111330270767212, |
| "kl_max_nats": 15.414010047912598, |
| "top128_jaccard_mean": 0.7149058585433935, |
| "top1_agreement": 0.9574262608758439, |
| "captured_mass_bf16": 0.9997188871511677, |
| "captured_mass_quant_at_bf16_support": 0.9995829765622533 |
| } |
| |
