base 892 | lin 446 | sna intouches 446
Loading the LM will be faster if you build a binary file.
Reading /scratch/lm/lin_5g.arpa
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
****************************************************************************************************
Found entries of length > 1 in alphabet. This is unusual unless style is BPE, but the alphabet was not recognized as BPE type. Is this correct?
beam principal : 446 clips

Loading weights:   0%|          | 0/1096 [00:00<?, ?it/s]
Loading weights:  47%|████▋     | 514/1096 [00:00<00:00, 5116.05it/s]
Loading weights: 100%|██████████| 1096/1096 [00:00<00:00, 6483.42it/s]
[W809 07:42:50.801488078 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 22413312, total: 47665709056).
[W809 07:42:50.801625166 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 22413312, total: 47665709056).
Traceback (most recent call last):
  File "/root/crosspool_lin.py", line 249, in <module>
    main()
  File "/root/crosspool_lin.py", line 129, in main
    dproc, DL = compute_logits(DONOR, files)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/gen_sna_rescore.py", line 53, in compute_logits
    m = AutoModelForCTC.from_pretrained(model_dir, dtype=torch.float32).cuda().eval()
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/venv/lib/python3.12/site-packages/transformers/modeling_utils.py", line 3788, in cuda
    return super().cuda(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1096, in cuda
    return self._apply(lambda t: t.cuda(device))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 933, in _apply
    module._apply(fn)
  File "/root/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 933, in _apply
    module._apply(fn)
  File "/root/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 933, in _apply
    module._apply(fn)
  [Previous line repeated 3 more times]
  File "/root/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 964, in _apply
    param_applied = fn(param)
                    ^^^^^^^^^
  File "/root/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1096, in <lambda>
    return self._apply(lambda t: t.cuda(device))
                                 ^^^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacity of 44.39 GiB of which 21.38 MiB is free. Process 3981785 has 23.76 GiB memory in use. Process 3983476 has 11.85 GiB memory in use. Process 3983486 has 7.69 GiB memory in use. Including non-PyTorch memory, this process has 1.04 GiB memory in use. Of the allocated memory 621.74 MiB is allocated by PyTorch, and 14.26 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)