| === [Sun May 24 13:19:04 UTC 2026] AA-SVD H200 MODE=smoke === |
| `torch_dtype` is deprecated! Use `dtype` instead! |
| The fast path is not available because on of `(selective_state_update, causal_conv1d_fn, causal_conv1d_update)` is None. Falling back to the naive implementation. To install follow https: |
| === AA-SVD H200 run === |
| Config(base_model_id='/home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16', tinker_adapter_path='/home/lakesenberg/nemotron_lora_work/adapter_v2/final', aligned_output_path='/home/lakesenberg/aa_svd_work/nemotron-adapter-aligned', plain_output_path='/home/lakesenberg/aa_svd_work/nemotron-adapter-plain', covariance_path='/home/lakesenberg/aa_svd_work/covariances.pt', submission_zip_path='/home/lakesenberg/aa_svd_work/submission', use_precomputed_covariance=False, calibration_prompts_path=None, num_calibration_samples=8, max_calibration_seq_len=1024, calibration_batch_size=1, base_model_load_in_4bit=False, base_model_device_map='auto', forced_fused_rank=32, svd_damping=1e-05, use_double_precision_svd=True, fallback_to_plain_svd=True, enable_activation_aware_for=('in_proj', 'qkv_proj', 'gate_up_proj'), device='cuda', seed=0, skip_plain_build=True, skip_smoke_gen=True) |
| Using built-in default calibration prompts. |
| Calibration prompts ready: 8 |
| Loading base model /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16 (4bit=False) ... |
|
Loading weights: 0%| | 0/6243 [00:00<?, ?it/s]
Loading weights: 2%|▏ | 145/6243 [00:00<00:04, 1433.31it/s]
Loading weights: 5%|▍ | 289/6243 [00:00<00:06, 895.67it/s]
Loading weights: 6%|▌ | 390/6243 [00:00<00:07, 810.48it/s]
Loading weights: 8%|▊ | 477/6243 [00:00<00:07, 768.20it/s]
Loading weights: 9%|▉ | 557/6243 [00:00<00:07, 746.38it/s]
Loading weights: 10%|█ | 634/6243 [00:00<00:07, 732.02it/s]
Loading weights: 11%|█▏ | 708/6243 [00:00<00:07, 719.44it/s]
Loading weights: 13%|█▎ | 781/6243 [00:01<00:07, 710.96it/s]
Loading weights: 14%|█▎ | 853/6243 [00:01<00:07, 712.99it/s]
Loading weights: 15%|█▍ | 925/6243 [00:01<00:07, 713.15it/s]
Loading weights: 16%|█▌ | 997/6243 [00:01<00:07, 706.47it/s]
Loading weights: 17%|█▋ | 1068/6243 [00:01<00:07, 704.83it/s]
Loading weights: 18%|█▊ | 1141/6243 [00:01<00:07, 710.75it/s]
Loading weights: 19%|█▉ | 1213/6243 [00:01<00:07, 708.94it/s]
Loading weights: 21%|██ | 1284/6243 [00:01<00:06, 708.86it/s]
Loading weights: 22%|██▏ | 1360/6243 [00:01<00:06, 723.83it/s]
Loading weights: 23%|██▎ | 1433/6243 [00:01<00:06, 715.93it/s]
Loading weights: 24%|██▍ | 1506/6243 [00:02<00:06, 718.22it/s]
Loading weights: 25%|██▌ | 1578/6243 [00:02<00:06, 718.62it/s]
Loading weights: 26%|██▋ | 1652/6243 [00:02<00:06, 723.41it/s]
Loading weights: 28%|██▊ | 1725/6243 [00:02<00:06, 719.52it/s]
Loading weights: 29%|██▉ | 1797/6243 [00:02<00:06, 718.86it/s]
Loading weights: 30%|██▉ | 1869/6243 [00:02<00:06, 711.78it/s]
Loading weights: 31%|███ | 1942/6243 [00:02<00:06, 715.16it/s]
Loading weights: 32%|███▏ | 2014/6243 [00:02<00:05, 708.23it/s]
Loading weights: 33%|███▎ | 2085/6243 [00:02<00:05, 705.57it/s]
Loading weights: 35%|███▍ | 2157/6243 [00:02<00:05, 708.35it/s]
Loading weights: 36%|███▌ | 2231/6243 [00:03<00:05, 715.03it/s]
Loading weights: 37%|███▋ | 2303/6243 [00:03<00:05, 714.90it/s]
Loading weights: 38%|███▊ | 2375/6243 [00:03<00:05, 712.11it/s]
Loading weights: 39%|███▉ | 2447/6243 [00:03<00:05, 707.68it/s]
Loading weights: 40%|████ | 2518/6243 [00:03<00:05, 700.18it/s]
Loading weights: 41%|████▏ | 2590/6243 [00:03<00:05, 704.79it/s]
Loading weights: 43%|████▎ | 2662/6243 [00:03<00:05, 708.61it/s]
Loading weights: 44%|████▍ | 2734/6243 [00:03<00:04, 710.25it/s]
Loading weights: 45%|████▍ | 2806/6243 [00:03<00:04, 707.55it/s]
Loading weights: 46%|████▌ | 2877/6243 [00:03<00:04, 706.22it/s]
Loading weights: 47%|████▋ | 2948/6243 [00:04<00:04, 703.99it/s]
Loading weights: 48%|████▊ | 3021/6243 [00:04<00:04, 710.50it/s]
Loading weights: 50%|████▉ | 3093/6243 [00:04<00:04, 712.25it/s]
Loading weights: 51%|█████ | 3165/6243 [00:04<00:04, 707.55it/s]
Loading weights: 52%|█████▏ | 3236/6243 [00:04<00:04, 698.37it/s]
Loading weights: 53%|█████▎ | 3308/6243 [00:04<00:04, 702.46it/s]
Loading weights: 54%|█████▍ | 3380/6243 [00:04<00:04, 706.20it/s]
Loading weights: 55%|█████▌ | 3451/6243 [00:04<00:03, 704.62it/s]
Loading weights: 56%|█████▋ | 3522/6243 [00:04<00:03, 700.17it/s]
Loading weights: 58%|█████▊ | 3596/6243 [00:04<00:03, 711.16it/s]
Loading weights: 59%|█████▉ | 3668/6243 [00:05<00:03, 713.39it/s]
Loading weights: 60%|█████▉ | 3740/6243 [00:05<00:03, 712.19it/s]
Loading weights: 61%|██████ | 3813/6243 [00:05<00:03, 711.88it/s]
Loading weights: 62%|██████▏ | 3886/6243 [00:05<00:03, 714.52it/s]
Loading weights: 63%|██████▎ | 3958/6243 [00:05<00:03, 707.93it/s]
Loading weights: 65%|██████▍ | 4029/6243 [00:05<00:03, 706.25it/s]
Loading weights: 66%|██████▌ | 4104/6243 [00:05<00:02, 716.69it/s]
Loading weights: 67%|██████▋ | 4177/6243 [00:05<00:02, 720.11it/s]
Loading weights: 68%|██████▊ | 4250/6243 [00:05<00:02, 712.96it/s]
Loading weights: 69%|██████▉ | 4322/6243 [00:05<00:02, 714.47it/s]
Loading weights: 70%|███████ | 4396/6243 [00:06<00:02, 720.07it/s]
Loading weights: 72%|███████▏ | 4469/6243 [00:06<00:02, 718.65it/s]
Loading weights: 73%|███████▎ | 4542/6243 [00:06<00:02, 720.74it/s]
Loading weights: 74%|███████▍ | 4615/6243 [00:06<00:02, 721.88it/s]
Loading weights: 75%|███████▌ | 4690/6243 [00:06<00:02, 728.86it/s]
Loading weights: 76%|███████▋ | 4764/6243 [00:06<00:02, 729.47it/s]
Loading weights: 77%|███████▋ | 4837/6243 [00:06<00:01, 723.23it/s]
Loading weights: 79%|███████▊ | 4912/6243 [00:06<00:01, 728.93it/s]
Loading weights: 80%|███████▉ | 4985/6243 [00:06<00:01, 725.87it/s]
Loading weights: 81%|████████ | 5058/6243 [00:07<00:01, 722.52it/s]
Loading weights: 82%|████████▏ | 5131/6243 [00:07<00:01, 718.03it/s]
Loading weights: 83%|████████▎ | 5205/6243 [00:07<00:01, 722.04it/s]
Loading weights: 85%|████████▍ | 5278/6243 [00:07<00:01, 717.62it/s]
Loading weights: 86%|████████▌ | 5350/6243 [00:07<00:01, 716.87it/s]
Loading weights: 87%|████████▋ | 5422/6243 [00:07<00:01, 711.97it/s]
Loading weights: 88%|████████▊ | 5495/6243 [00:07<00:01, 716.43it/s]
Loading weights: 89%|████████▉ | 5567/6243 [00:07<00:00, 712.51it/s]
Loading weights: 90%|█████████ | 5640/6243 [00:07<00:00, 715.64it/s]
Loading weights: 92%|█████████▏| 5714/6243 [00:07<00:00, 722.68it/s]
Loading weights: 93%|█████████▎| 5787/6243 [00:08<00:00, 718.80it/s]
Loading weights: 94%|█████████▍| 5859/6243 [00:08<00:00, 712.69it/s]
Loading weights: 95%|█████████▌| 5931/6243 [00:08<00:00, 706.88it/s]
Loading weights: 96%|█████████▌| 6003/6243 [00:08<00:00, 710.52it/s]
Loading weights: 97%|█████████▋| 6075/6243 [00:08<00:00, 708.21it/s]
Loading weights: 98%|█████████▊| 6146/6243 [00:08<00:00, 707.73it/s]
Loading weights: 100%|█████████▉| 6219/6243 [00:08<00:00, 712.56it/s]
Loading weights: 100%|██████████| 6243/6243 [00:08<00:00, 712.30it/s] |
| after base load GPU mem: used=58.8 GiB, peak=58.8 GiB |
| Traceback (most recent call last): |
| File "/home/lakesenberg/code/nemotron_lora_train/run_aa_svd_h200.py", line 600, in <module> |
| main() |
| File "/home/lakesenberg/code/nemotron_lora_train/run_aa_svd_h200.py", line 548, in main |
| covariances = collect_activation_covariance(cfg, calib) |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| File "/home/lakesenberg/code/nemotron_lora_train/run_aa_svd_h200.py", line 327, in collect_activation_covariance |
| accums[weight_key] = _CovAccumulator(module.in_features, device=device) |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| File "/home/lakesenberg/code/nemotron_lora_train/run_aa_svd_h200.py", line 278, in __init__ |
| self.cov = torch.zeros(in_dim, in_dim, dtype=torch.float32, device=device) |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 28.00 MiB. GPU 0 has a total capacity of 139.72 GiB of which 4.69 MiB is free. Including non-PyTorch memory, this process has 139.71 GiB memory in use. Of the allocated memory 139.12 GiB is allocated by PyTorch, and 146.50 KiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) |
| === [Sun May 24 13:19:36 UTC 2026] AA-SVD H200 MODE=smoke === |
| `torch_dtype` is deprecated! Use `dtype` instead! |
| The fast path is not available because on of `(selective_state_update, causal_conv1d_fn, causal_conv1d_update)` is None. Falling back to the naive implementation. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1d |
| === AA-SVD H200 run === |
| Config(base_model_id='/home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16', tinker_adapter_path='/home/lakesenberg/nemotron_lora_work/adapter_v2/final', aligned_output_path='/home/lakesenberg/aa_svd_work/nemotron-adapter-aligned', plain_output_path='/home/lakesenberg/aa_svd_work/nemotron-adapter-plain', covariance_path='/home/lakesenberg/aa_svd_work/covariances.pt', submission_zip_path='/home/lakesenberg/aa_svd_work/submission', use_precomputed_covariance=False, calibration_prompts_path=None, num_calibration_samples=8, max_calibration_seq_len=1024, calibration_batch_size=1, base_model_load_in_4bit=False, base_model_device_map='auto', forced_fused_rank=32, svd_damping=1e-05, use_double_precision_svd=True, fallback_to_plain_svd=True, enable_activation_aware_for=('in_proj', 'qkv_proj', 'gate_up_proj'), device='cuda', seed=0, skip_plain_build=True, skip_smoke_gen=True) |
| Using built-in default calibration prompts. |
| Calibration prompts ready: 8 |
| Loading base model /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16 (4bit=False) ... |
|
Loading weights: 0%| | 0/6243 [00:00<?, ?it/s]
Loading weights: 2%|▏ | 130/6243 [00:00<00:04, 1294.30it/s]
Loading weights: 4%|▍ | 260/6243 [00:00<00:06, 874.24it/s]
Loading weights: 6%|▌ | 356/6243 [00:00<00:07, 801.03it/s]
Loading weights: 7%|▋ | 440/6243 [00:00<00:07, 768.56it/s]
Loading weights: 8%|▊ | 519/6243 [00:00<00:07, 736.18it/s]
Loading weights: 10%|▉ | 594/6243 [00:00<00:07, 730.31it/s]
Loading weights: 11%|█ | 668/6243 [00:00<00:07, 716.59it/s]
Loading weights: 12%|█▏ | 740/6243 [00:00<00:07, 711.02it/s]
Loading weights: 13%|█▎ | 812/6243 [00:01<00:07, 708.64it/s]
Loading weights: 14%|█▍ | 885/6243 [00:01<00:07, 712.88it/s]
Loading weights: 15%|█▌ | 957/6243 [00:01<00:07, 708.58it/s]
Loading weights: 16%|█▋ | 1028/6243 [00:01<00:07, 708.72it/s]
Loading weights: 18%|█▊ | 1101/6243 [00:01<00:07, 713.05it/s]
Loading weights: 19%|█▉ | 1173/6243 [00:01<00:07, 705.90it/s]
Loading weights: 20%|█▉ | 1245/6243 [00:01<00:07, 708.25it/s]
Loading weights: 21%|██ | 1316/6243 [00:01<00:06, 706.78it/s]
Loading weights: 22%|██▏ | 1390/6243 [00:01<00:06, 716.03it/s]
Loading weights: 23%|██▎ | 1462/6243 [00:01<00:06, 713.55it/s]
Loading weights: 25%|██▍ | 1534/6243 [00:02<00:06, 713.49it/s]
Loading weights: 26%|██▌ | 1606/6243 [00:02<00:06, 712.11it/s]
Loading weights: 27%|██▋ | 1678/6243 [00:02<00:06, 697.10it/s]
Loading weights: 28%|██▊ | 1748/6243 [00:02<00:06, 696.26it/s]
Loading weights: 29%|██▉ | 1818/6243 [00:02<00:06, 694.81it/s]
Loading weights: 30%|███ | 1888/6243 [00:02<00:06, 693.01it/s]
Loading weights: 31%|███▏ | 1958/6243 [00:02<00:06, 692.63it/s]
Loading weights: 33%|███▎ | 2030/6243 [00:02<00:06, 698.28it/s]
Loading weights: 34%|███▎ | 2100/6243 [00:02<00:05, 694.96it/s]
Loading weights: 35%|███▍ | 2175/6243 [00:03<00:05, 694.25it/s]
Loading weights: 36%|███▌ | 2250/6243 [00:03<00:05, 707.55it/s]
Loading weights: 37%|███▋ | 2321/6243 [00:03<00:05, 706.07it/s]
Loading weights: 38%|███▊ | 2392/6243 [00:03<00:05, 700.59it/s]
Loading weights: 39%|███▉ | 2464/6243 [00:03<00:05, 703.90it/s]
Loading weights: 41%|████ | 2535/6243 [00:03<00:05, 702.62it/s]
Loading weights: 42%|████▏ | 2606/6243 [00:03<00:05, 703.32it/s]
Loading weights: 43%|████▎ | 2677/6243 [00:03<00:05, 701.43it/s]
Loading weights: 44%|████▍ | 2748/6243 [00:03<00:04, 702.17it/s]
Loading weights: 45%|████▌ | 2819/6243 [00:03<00:04, 701.18it/s]
Loading weights: 46%|████▋ | 2890/6243 [00:04<00:04, 690.35it/s]
Loading weights: 47%|████▋ | 2960/6243 [00:04<00:04, 689.87it/s]
Loading weights: 49%|████▊ | 3033/6243 [00:04<00:04, 701.50it/s]
Loading weights: 50%|████▉ | 3104/6243 [00:04<00:04, 699.14it/s]
Loading weights: 51%|█████ | 3174/6243 [00:04<00:04, 696.77it/s]
Loading weights: 52%|█████▏ | 3244/6243 [00:04<00:04, 695.90it/s]
Loading weights: 53%|█████▎ | 3316/6243 [00:04<00:04, 701.08it/s]
Loading weights: 54%|█████▍ | 3387/6243 [00:04<00:04, 701.84it/s]
Loading weights: 55%|█████▌ | 3458/6243 [00:04<00:04, 693.76it/s]
Loading weights: 57%|█████▋ | 3529/6243 [00:04<00:03, 697.12it/s]
Loading weights: 58%|█████▊ | 3600/6243 [00:05<00:03, 698.93it/s]
Loading weights: 59%|█████▉ | 3671/6243 [00:05<00:03, 700.69it/s]
Loading weights: 60%|█████▉ | 3742/6243 [00:05<00:03, 698.32it/s]
Loading weights: 61%|██████ | 3813/6243 [00:05<00:03, 698.46it/s]
Loading weights: 62%|██████▏ | 3885/6243 [00:05<00:03, 702.45it/s]
Loading weights: 63%|██████▎ | 3956/6243 [00:05<00:03, 696.95it/s]
Loading weights: 64%|██████▍ | 4026/6243 [00:05<00:03, 694.49it/s]
Loading weights: 66%|██████▌ | 4100/6243 [00:05<00:03, 705.26it/s]
Loading weights: 67%|██████▋ | 4171/6243 [00:05<00:02, 706.57it/s]
Loading weights: 68%|██████▊ | 4242/6243 [00:05<00:02, 705.82it/s]
Loading weights: 69%|██████▉ | 4313/6243 [00:06<00:02, 700.07it/s]
Loading weights: 70%|███████ | 4386/6243 [00:06<00:02, 706.45it/s]
Loading weights: 71%|███████▏ | 4457/6243 [00:06<00:02, 704.77it/s]
Loading weights: 73%|███████▎ | 4528/6243 [00:06<00:02, 705.97it/s]
Loading weights: 74%|███████▎ | 4600/6243 [00:06<00:02, 708.55it/s]
Loading weights: 75%|███████▍ | 4673/6243 [00:06<00:02, 714.57it/s]
Loading weights: 76%|███████▌ | 4745/6243 [00:06<00:02, 712.86it/s]
Loading weights: 77%|███████▋ | 4817/6243 [00:06<00:02, 701.41it/s]
Loading weights: 78%|███████▊ | 4892/6243 [00:06<00:01, 698.33it/s]
Loading weights: 80%|███████▉ | 4966/6243 [00:06<00:01, 708.62it/s]
Loading weights: 81%|████████ | 5037/6243 [00:07<00:01, 708.73it/s]
Loading weights: 82%|████████▏ | 5108/6243 [00:07<00:01, 707.93it/s]
Loading weights: 83%|████████▎ | 5182/6243 [00:07<00:01, 715.26it/s]
Loading weights: 84%|████████▍ | 5254/6243 [00:07<00:01, 707.79it/s]
Loading weights: 85%|████████▌ | 5326/6243 [00:07<00:01, 710.40it/s]
Loading weights: 86%|████████▋ | 5398/6243 [00:07<00:01, 709.74it/s]
Loading weights: 88%|████████▊ | 5471/6243 [00:07<00:01, 713.74it/s]
Loading weights: 89%|████████▉ | 5543/6243 [00:07<00:00, 706.14it/s]
Loading weights: 90%|████████▉ | 5615/6243 [00:07<00:00, 707.73it/s]
Loading weights: 91%|█████████ | 5686/6243 [00:08<00:00, 707.61it/s]
Loading weights: 92%|█████████▏| 5757/6243 [00:08<00:00, 703.72it/s]
Loading weights: 93%|█████████▎| 5828/6243 [00:08<00:00, 690.93it/s]
Loading weights: 94%|█████████▍| 5898/6243 [00:08<00:00, 688.72it/s]
Loading weights: 96%|█████████▌| 5970/6243 [00:08<00:00, 693.29it/s]
Loading weights: 97%|█████████▋| 6043/6243 [00:08<00:00, 701.44it/s]
Loading weights: 98%|█████████▊| 6114/6243 [00:08<00:00, 701.68it/s]
Loading weights: 99%|█████████▉| 6185/6243 [00:08<00:00, 702.20it/s]
Loading weights: 100%|██████████| 6243/6243 [00:08<00:00, 701.75it/s] |
| `use_return_dict` is deprecated! Use `return_dict` instead! |
| MoE expert LoRA serving for nemotron models is experimental in vLLM and not yet supported in SGLang. The adapter will be produced but may not work with all serving configurations. |
| after base load GPU mem: used=58.8 GiB, peak=58.8 GiB |
| Registered hooks on 3008 fused-target modules. |
| step 0: GPU mem: used=58.9 GiB, peak=59.7 GiB |
| [covariance collection] took 54.07s |
| Saved 81.0 GiB -> /home/lakesenberg/aa_svd_work/covariances.pt |
| Have 3008 covariance matrices. |
| patched: patched_merge_fused_projections |
| Traceback (most recent call last): |
| File "/home/lakesenberg/code/nemotron_lora_train/run_aa_svd_h200.py", line 601, in <module> |
| main() |
| File "/home/lakesenberg/code/nemotron_lora_train/run_aa_svd_h200.py", line 557, in main |
| build_adapter(cfg, cfg.aligned_output_path, "activation-aware") |
| File "/home/lakesenberg/code/nemotron_lora_train/run_aa_svd_h200.py", line 445, in build_adapter |
| weights.build_lora_adapter( |
| File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/tinker_cookbook/weights/_adapter.py", line 150, in build_lora_adapter |
| peft_weights, target_modules, rank_overrides = _convert_adapter( |
| ^^^^^^^^^^^^^^^^^ |
| File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/tinker_cookbook/weights/_adapter.py", line 293, in _convert_adapter |
| _expand_expert_weights( |
| File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/tinker_cookbook/weights/_adapter.py", line 506, in _expand_expert_weights |
| raise WeightsAdapterError( |
| tinker_cookbook.exceptions.WeightsAdapterError: Expert LoRA weights must be 3D, got lora_A: torch.Size([32, 1856]), lora_B: torch.Size([2688, 32]) for key targeting 'backbone.layers.1.mixer.experts.0.down_proj.weight' |
| === [Sun May 24 13:24:53 UTC 2026] AA-SVD H200 MODE=smoke === |
| MoE expert LoRA serving for nemotron models is experimental in vLLM and not yet supported in SGLang. The adapter will be produced but may not work with all serving configurations. |
| === AA-SVD H200 run === |
| Config(base_model_id='/home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16', tinker_adapter_path='/home/lakesenberg/nemotron_lora_work/adapter_v2/final', aligned_output_path='/home/lakesenberg/aa_svd_work/nemotron-adapter-aligned', plain_output_path='/home/lakesenberg/aa_svd_work/nemotron-adapter-plain', covariance_path='/home/lakesenberg/aa_svd_work/covariances.pt', submission_zip_path='/home/lakesenberg/aa_svd_work/submission', use_precomputed_covariance=True, calibration_prompts_path=None, num_calibration_samples=8, max_calibration_seq_len=1024, calibration_batch_size=1, base_model_load_in_4bit=False, base_model_device_map='auto', forced_fused_rank=32, svd_damping=1e-05, use_double_precision_svd=True, fallback_to_plain_svd=True, enable_activation_aware_for=('in_proj', 'qkv_proj', 'gate_up_proj'), device='cuda', seed=0, skip_plain_build=True, skip_smoke_gen=True) |
| Using built-in default calibration prompts. |
| Calibration prompts ready: 8 |
| Loading precomputed covariances from /home/lakesenberg/aa_svd_work/covariances.pt |
| Have 3008 covariance matrices. |
| patched: patched_merge_fused_projections |
| [activation-aware adapter build] took 6.29s |
| activation-aware build wrote 2 files -> /home/lakesenberg/aa_svd_work/nemotron-adapter-aligned |
| |
| === DONE === |
| Aligned adapter : /home/lakesenberg/aa_svd_work/nemotron-adapter-aligned |
| Submission zip : /home/lakesenberg/aa_svd_work/submission.zip (3.0 GiB) |
| |
| Kaggle tips (RTX Pro 6000): |
| - set CFG.base_model_load_in_4bit = True |
| - use this notebook with adapter uploaded as Kaggle Model dataset |
| === [Sun May 24 13:28:09 UTC 2026] AA-SVD finished === |
| |
