README.md

# Uncontrolled memory pre-allocation (CWE-789) in msgpack-python's C-extension Unpacker

**Status:** Independently verified via actual execution against the real, unmodified, compiled
`msgpack._cmsgpack` C extension. This is a PRIVATE, gated staging repo for a huntr submission — not a
public advisory.

## Target

- Project: [msgpack-python](https://github.com/msgpack/msgpack-python) (PyPI package `msgpack`)
- Version confirmed vulnerable: **1.2.1** (`msgpack/__init__.py`: `version = (1, 2, 1)`)
- Commit used for line citations below: `c6b4a481ed574b8b785c4487d7ef9b592c8b4f2b`
  (`git clone https://github.com/msgpack/msgpack-python.git`)
- Not to be confused with:
  - `msgpack-numpy`'s `pickle.loads()` RCE (already filed separately by this researcher —
    `EnigmaConsultant/huntr-poc-msgpack-numpy-rllib-pickle-rce`).
  - The sibling **msgpack-c** C library, which has an architecturally-related but distinct,
    separately-tracked (unfiled) finding in its own vendored `unpack_template.h` /
    `unpack_container_header.h` / `unpack_define.h`. This report is specifically about the
    **msgpack-python** package's own C extension, confirmed against the real compiled `.so`.

## Root cause

In `msgpack/unpack.h`, `unpack_callback_array()` (and `unpack_callback_map()` when
`object_pairs_hook` is used) does:

```c
static inline int unpack_callback_array(unpack_user* u, unsigned int n, msgpack_unpack_object* o)
{
    if (n > u->max_array_len) {
        PyErr_Format(PyExc_ValueError, "%u exceeds max_array_len(%zd)", n, u->max_array_len);
        return -1;
    }
    PyObject *p = u->use_list ? PyList_New(n) : PyTuple_New(n);
    ...
```

(`msgpack/unpack.h` lines 126-132, confirmed against commit `c6b4a481`.)

This allocation fires **immediately upon parsing the array/map header** — i.e. as soon as the
attacker-controlled element count `n` has been read from the wire format (e.g. an `array32` header:
1 tag byte + 4-byte big-endian count) — and **before a single one of the `n` claimed elements has
actually been received or validated**.

The only gate is `if (n > u->max_array_len)`. For the streaming `Unpacker` class — the documented,
recommended API for "streaming deserialize from socket" — `max_array_len` **defaults to
`max_buffer_size`** (default 100 MiB; see `msgpack/_unpacker.pyx` lines 364-365:
`if max_array_len == -1: max_array_len = max_buffer_size`). That silently sets an
**element-count limit equal to a byte-count budget**. Since each pre-allocated list/tuple slot costs
8 bytes (a `PyObject*` pointer) regardless of how many actual input bytes encode that element, this
gives an inherent **~8x memory amplification per container level** versus the developer's
configured/default budget.

This compounds across nesting: `unpack_template.h`'s `start_container` machinery fires this same
allocation at **every nesting level independently**, bounded only by `MSGPACK_EMBED_STACK_SIZE` = 32
— which correctly bounds nesting *depth* but does nothing to bound the *per-level element count
claimed*. So 32 nested array headers (~160 bytes of input) each independently pass the same
`n <= max_array_len` check and each trigger their own `PyList_New(n)`, **stacking the 8x
amplification 32-fold**.

Critically, this defeats the library's own documented security guidance. The docstring for
`max_buffer_size` explicitly states:

> "You should set this parameter when unpacking data from untrusted source."

Yet a developer who follows that exact advice (e.g. explicitly setting `max_buffer_size=10*1024*1024`)
still sees ~1.9-2.7 GB of virtual memory reserved from a **160-byte** crafted payload — an **180x+
bypass** of their configured budget — because `max_array_len` silently inherited the byte-budget as
an element-count budget, and per-level compounding is not accounted for at all.

## Negative control (confirms the one-shot `unpackb()` path is NOT vulnerable)

`msgpack/_unpacker.pyx` lines 181-182: for `unpackb()`, when `max_array_len == -1` it defaults to
`buf_len` — the length of the *actual buffer supplied*, not a configurable budget — so a header
claiming more elements than bytes present is correctly rejected:

```
$ ./venv/bin/python poc1.py
payload len: 5
Exception: <class 'ValueError'> 100000000 exceeds max_array_len(5)
```

This is a useful control: it isolates the bug to the streaming `Unpacker` class specifically (the
class the docs recommend for untrusted/socket input), not to the msgpack wire-format parser in
general.

## Proof of Concept

Built a venv, installed real `msgpack==1.2.1` from PyPI (unmodified, official wheel — real compiled
`msgpack._cmsgpack` C extension). Four escalating PoCs, each printing `/proc/self/status`
`VmPeak`/`VmSize`/`VmRSS` before and after. All scripts included verbatim in this repo.

### PoC 2 — single streaming header, no payload following (`poc4.py`)

A **5-byte** `array32` header (tag `0xdd` + count = 100,000,000) fed to a plain `Unpacker()`
(`max_buffer_size` default 100 MiB → `max_array_len` defaults to the same 100 MiB, so `n` is
"legal"):

```
$ ./venv/bin/python poc4.py
[start] VmPeak: 17728 kB | VmSize: 17628 kB | VmRSS: 10544 kB
header bytes: 5 claimed elements: 100000000 -> naive PyList_New(n) would need 800.0 MB just for pointer array
[after feed (before iterate)] VmPeak: 18656 kB | VmSize: 18656 kB | VmRSS: 10816 kB
[after iterate attempt] VmPeak: 799908 kB | VmSize: 799908 kB | VmRSS: 10956 kB
got_exc: None
```

VmPeak jumps from ~18 MB to ~800 MB — matching the predicted `100,000,000 * 8 bytes/pointer` exactly
— from **5 bytes of input**, with **zero exception raised**. VmRSS stays ~11 MB, confirming this is a
virtual/reserved-not-committed allocation (demand-paged mmap/calloc), the same pattern documented in
the related msgpack-c finding.

### PoC 3 — nested-container compounding (`poc6_capped.py`)

4 nested `array32` headers (20 bytes total), each claiming `n=40,000,000`.
`RLIMIT_AS` capped at 1.5 GB purely as a **test-harness safety net** for the shared host — not part
of the bug:

```
$ ./venv/bin/python poc6_capped.py
[start (RLIMIT_AS capped at 1.5GB for safety)] VmPeak=17736 kB VmSize=17660 kB VmRSS=10552 kB
total payload bytes: 20 -> naive 4-level total: 1.28 GB
elapsed: 0.0 s
[after 20-byte (4x5) nested-header feed] VmPeak=1268704 kB VmSize=1268704 kB VmRSS=11216 kB
```

VmPeak lands at exactly the predicted `4 * 40,000,000 * 8 bytes = 1.269 GB` — demonstrating clean
per-nesting-level multiplication from **20 bytes of input**.

### PoC 4 — killer scenario: developer follows the library's own documented security advice (`poc7_explicit_limit_bypass.py`)

Developer explicitly sets `max_buffer_size=10*1024*1024` (10 MiB) — precisely per the docstring's
own advice to "set this parameter when unpacking data from untrusted source". A **160-byte** payload
of 32 nested `array32` headers (`MSGPACK_EMBED_STACK_SIZE`), each claiming `n=10,485,760` (the
inherited `max_array_len`), drives memory reservation to 1.9 GB before hitting our own 2 GB
test-harness `RLIMIT_AS` safety cap (predicted uncapped total: 2.68 GB):

```
$ ./venv/bin/python poc7_explicit_limit_bypass.py
Developer explicitly sets max_buffer_size=10485760 (10 MiB) per the library's own documented advice.
[start] VmPeak=18696 kB VmSize=18696 kB VmRSS=10684 kB
attacker payload: 160 bytes (32 nested array32 headers, each claiming n=10485760)
developer's configured budget: 10.5 MB total
naive worst case reservation: 32 x 10485760 x 8 bytes/ptr = 2.68 GB
Exception: <class 'MemoryError'>
[after feeding 160-byte payload] VmPeak=1902948 kB VmSize=18696 kB VmRSS=11196 kB
```

**180x+ bypass** of the developer's explicitly-configured 10 MB security budget, from **160 bytes**
of attacker input, using only the library's own documented "safe" configuration pattern. The
`MemoryError` here is an artifact of our own 2 GB test-harness `RLIMIT_AS` safety cap — an attacker
without that cap in place (or against a host with more available memory / overcommit enabled) drives
this well past 2.6 GB uncapped.

## Impact

Any application that uses `msgpack.Unpacker()` (the documented streaming/socket-input API) to
deserialize attacker-controlled msgpack data — even one that follows the library's own documented
advice and sets an explicit `max_buffer_size` — can be driven to reserve gigabytes of virtual memory
from a payload of a few hundred bytes, before a single byte of actual element data has been
transmitted or validated. This is a denial-of-service primitive (memory exhaustion / OOM-kill) that
silently bypasses the library's own stated defense-in-depth mechanism.

## Suggested fix

- Do not eagerly pre-allocate `PyList_New(n)` / `PyTuple_New(n)` for the full claimed count `n`
  before any elements have been received. Either grow the container incrementally as elements
  arrive, or defer full allocation until enough buffered input exists to plausibly encode `n`
  elements (e.g. requiring at least `n` bytes remaining in the buffer, as `unpackb()` already does
  via `buf_len`).
- Decouple `max_array_len` / `max_map_len` from `max_buffer_size` by default, and/or explicitly
  divide by the per-element pointer overhead (8 bytes) and by `MSGPACK_EMBED_STACK_SIZE` when
  deriving a default, so the documented byte-budget is actually a byte-budget in the worst case
  across nesting.

## Files in this repo

- `poc1.py` — negative control: `unpackb()` one-shot path correctly rejects an oversized header.
- `poc2.py`, `poc3.py` — earlier iterations kept for provenance.
- `poc4.py` — single 5-byte streaming header -> ~800 MB VmPeak.
- `poc5_nested.py` — early nested-header iteration.
- `poc6_capped.py` — 4-level nesting compounding, 1.5 GB safety-capped.
- `poc7_explicit_limit_bypass.py` — 32-level nesting against an explicit developer-configured
  `max_buffer_size`, 2 GB safety-capped.

All PoCs were run against the real, unmodified, PyPI-installed `msgpack==1.2.1` package (compiled
`msgpack._cmsgpack` C extension) — not a modified or hypothetical build.