trevdatastreams/huntr-poc-avro-cpp-collection-count-allocation

Apache Avro C++ collection-count allocation PoC

Tested against Apache Avro commit:

44f7aa35a6b90ac7d39ef0e68dad706561235941

Summary

The C++ generic Avro decoder converts an attacker-controlled array or map block count to size_t without a limit, then resizes its in-memory collection to that count before decoding any item.

The 103-byte trigger contains a valid OCF header and a top-level array<long> schema. Its three-byte array block count declares one million items, while the bounded data block ends immediately after the count.

The normal public DataFileReader<GenericDatum>::read() path allocates the million-element vector<GenericDatum> before it discovers the missing first item. Across three runs, peak RSS increased from about 21.6 MB for the same-sized valid control to 103.0–103.2 MB for the trigger, then the decoder reported EOF. Larger counts can request correspondingly larger allocations without making the file larger.

Reproduce

./reproduce.sh

The script uses the current local Avro sanitizer build unless AVRO_SRC and AVRO_BUILD override it.

Expected control:

decoded=1 max_rss_bytes=21643264

Expected trigger:

EXCEPTION EOF reached max_rss_bytes=103219200

The harness catches the final EOF only to print peak resident memory. The allocation occurs inside Avro before that error is raised.

Fixtures

db4c236fc5c338667a2c88c7471e3d8a2060dcb18a10fc808cc54e265079b5ee  control-one-array-item.avro
3f7466b310e92e0e294bd3df9b5b9830b9d866b4fc23ec427b99bfa95023482a  trigger-million-array-items.avro

Both fixtures are 103 bytes and share the same OCF header, schema, object count, and sync marker. The control encodes one array item and its terminator. The trigger uses the same three payload bytes for the count 1,000,000.

Root cause

At lang/c++/impl/BinaryDecoder.cc:163-169, the decoded long becomes a size_t with no collection-size or remaining-input check:

size_t BinaryDecoder::doDecodeItemCount() {
    auto result = doDecodeLong();
    if (result < 0) {
        doDecodeLong();
        return static_cast<size_t>(-(result + 1)) + 1;
    }
    return static_cast<size_t>(result);
}

At lang/c++/impl/Generic.cc:102-113, the array reader preallocates all declared elements before decoding the first:

for (size_t m = d.arrayStart(); m != 0; m = d.arrayNext()) {
    r.resize(r.size() + m);
    for (; start < r.size(); ++start) {
        r[start] = GenericDatum(nn);
        read(r[start], d, isResolving);
    }
}

The map path repeats the same pattern at Generic.cc:116-128.

Prior-art distinction

AVRO-3617 fixed inconsistent integer types in the C++ Validator counters. It did not add a collection-size limit to BinaryDecoder or GenericReader.

Java Avro has SystemLimitException checks for collection lengths. The current C++ decoder does not expose or enforce an equivalent limit.

The previously reported negative OCF object-count issue affects DataFileReaderBase::objectCount_ and causes an unbounded stream of records. This issue uses a positive outer object count and a separate nested collection count that reaches vector::resize.

Suggested fix

Add configurable maximum array and map lengths to the C++ decoder, matching the existing Java system-limit model. Before resizing, reject:

• counts above the configured collection limit;
• r.size() + m overflow;
• counts that cannot be represented by the bounded block's remaining bytes.

Throw an Avro parse exception before allocating collection storage.