# Uncaught tokenize.TokenError in np.load() via crafted NPY header

**Target:** numpy/numpy
**Platform:** huntr.com
**Severity:** High
**CWE:** CWE-248 (Uncaught Exception)
**Affected versions:** numpy 2.3.5 (confirmed), main branch commit b6e8ad8 (confirmed)
**Python versions affected:** 3.12+ (where C tokenizer enforces nesting limit)

---

## Summary

Loading a crafted NPY file causes numpy.load() to raise tokenize.TokenError instead of the documented ValueError. The exception is not a subclass of ValueError and is not caught anywhere in the numpy loading path. Any application that handles numpy loading errors by catching ValueError will have the TokenError crash it unexpectedly.

---

## Vulnerability Details

**Type:** Uncaught Exception / Improper Exception Handling
**CWE:** CWE-248
**Affected file:** numpy/lib/_format_impl.py
**Affected function:** _read_array_header() (line 659-666) and _filter_header() (line 586-618)
**Affected NPY versions:** v1.0 and v2.0 format files (not v3.0)

---

## Root Cause

In _read_array_header(), when ast.literal_eval() fails with SyntaxError parsing the header, a Python 2 backwards-compatibility fallback calls _filter_header():

```python
# numpy/lib/_format_impl.py, line 656-666
try:
    d = ast.literal_eval(header)
except SyntaxError as e:
    if version <= (2, 0):
        header = _filter_header(header)   # <-- raises TokenError
        try:
            d = ast.literal_eval(header)
        except SyntaxError as e2:
            raise ValueError(...) from e2
```

_filter_header() calls tokenize.generate_tokens() (line 608). Python 3.12+ enforces a nesting depth limit in its C tokenizer. When a header has 100+ nested parentheses, tokenize.generate_tokens() raises tokenize.TokenError with the message "too many nested parentheses".

TokenError is NOT a subclass of ValueError. It is not caught by the except SyntaxError handler above, nor by any other handler in the numpy call stack. It propagates all the way out of np.load() as an undocumented exception.

**The full exception hierarchy:**
```
tokenize.TokenError
  -> Exception
  -> BaseException
  
NOT related to:
  ValueError (what np.load() documentation says to catch)
  OSError
```

---

## Trigger Construction

A crafted NPY v1.0 file with a dtype descriptor having 100+ nesting levels.

The header must:
1. Have a descr field with >= 100 nested list-tuple structures
2. Be <= 10,000 characters (the max_header_size limit)

Example descriptor at depth=100 (fits in 1,147 chars, well under 10,000 limit):

```
[('f99', [('f98', [('f97', ... [('f0', '<f4')] ... )])])]
```

At depth=99: ast.literal_eval succeeds, numpy loads without error (safe behavior).
At depth=100: ast.literal_eval raises SyntaxError, _filter_header raises TokenError (bug).

---

## Steps to Reproduce

1. Clone repository: git clone https://github.com/numpy/numpy
2. Install numpy: pip install numpy
3. Run PoC: python3 poc_numpy.py

Expected (incorrect) behavior: TokenError propagates from np.load()
Expected (correct) behavior: ValueError raised with descriptive message

---

## PoC Script

See poc_numpy.py in this directory.

Key output:
```
[+] CONFIRMED: tokenize.TokenError raised!
    Message : ('too many nested parentheses', (1, 894))
    Type MRO: ['TokenError', 'Exception', 'BaseException', 'object']

[+] CONFIRMED: TokenError ESCAPED ValueError handler!
    Is ValueError: False
    Is Exception : True

depth=99: ValueError raised (safe)
depth=100: TokenError raised (VULNERABLE)
```

---

## Impact

Loading a 1,214-byte crafted NPY file causes np.load() to raise tokenize.TokenError rather than the documented ValueError.

Real-world attack scenario: A web service or ML pipeline accepts user-provided model files in NPY format, loads them with np.load(), and wraps the call in "except ValueError" to handle malformed inputs. The attacker uploads a crafted NPY file. The service fails with an unexpected TokenError that bypasses the error handler. If not caught at a higher level, this crashes the request handler or the worker process.

Affected pattern (extremely common in ML applications):
```python
try:
    arr = np.load(user_uploaded_file)  # crashes with TokenError
except ValueError as e:               # never reached
    return error_response(e)
```

Any service that:
1. Accepts user-provided NPY files
2. Calls np.load() on them
3. Handles errors with except ValueError

...can be crashed by a single 1.2KB crafted file with no authentication or privileges required.

---

## Suggested Fix

In _read_array_header(), catch TokenError from the _filter_header() call and re-raise as ValueError:

```python
from tokenize import TokenError

try:
    d = ast.literal_eval(header)
except SyntaxError as e:
    if version <= (2, 0):
        try:
            header = _filter_header(header)
        except TokenError as te:
            raise ValueError(
                f"Cannot parse header (tokenizer error): {header!r}"
            ) from te
        try:
            d = ast.literal_eval(header)
        except SyntaxError as e2:
            raise ValueError(f"Cannot parse header: {header!r}") from e2
```

---

## CVSS 3.1

AV:N/AC:L/PR:N/UI:R/S:U/C:N/I:N/A:L

Score: 4.3 (Medium)

Justification:
- AV:N — attacker uploads a file to a network-accessible service
- AC:L — no special conditions, just upload a 1.2KB file
- PR:N — no privileges needed
- UI:R — a user/service must load the file (implicit in model loading services)
- C:N/I:N — no data leaked or modified
- A:L — crashes the request handler / individual worker process

---

## References

- numpy _format_impl.py _read_array_header: https://github.com/numpy/numpy/blob/main/numpy/lib/_format_impl.py#L659
- numpy _filter_header: https://github.com/numpy/numpy/blob/main/numpy/lib/_format_impl.py#L586
- Python tokenize.TokenError: https://docs.python.org/3/library/tokenize.html#tokenize.TokenError
- Python 3.12 C tokenizer nesting limit: https://docs.python.org/3/whatsnew/3.12.html