trevdatastreams/apache-avro-negative-object-count-poc

Apache Avro C++ negative object-count infinite record stream

Summary

Apache Avro C++ accepts a negative top-level object count in an Object Container File data block. DataFileReaderBase::hasMore() treats every nonzero count as available data, and DataFileReader<T>::read() decrements the count after every successful record.

An empty-record schema consumes zero bytes per record. A 113-byte file with objectCount_ == -1 therefore makes the public C++ reader return records continuously without advancing the data stream or reaching EOF.

The control differs by one byte:

• control block count: 1, Avro zig-zag byte 0x02;
• trigger block count: -1, Avro zig-zag byte 0x01;
• mutation offset: 95.

Affected revision

• Apache Avro commit: 44f7aa35a6b90ac7d39ef0e68dad706561235941
• Component: lang/c++
• Public API: avro::DataFileReader<avro::GenericDatum>::read()

Root cause

lang/c++/impl/DataFile.cc:541 stores the untrusted signed block count without validating that it is non-negative:

avro::decode(*decoder_, objectCount_);

DataFile.cc:470-476 treats -1 as data:

if (eof_) {
    return false;
} else if (objectCount_ != 0) {
    return true;
}

lang/c++/include/avro/DataFile.hh:402-406 then decrements the negative count and decodes another record:

if (base_->hasMore()) {
    base_->decr();
    avro::decode(base_->decoder(), datum);
    return true;
}

For an empty record, avro::decode() consumes zero bytes. The sequence is -1, -2, -3, ...; hasMore() never observes zero.

Reproduction

Build current Avro C++ with sanitizers:

cmake -S /path/to/avro/lang/c++ -B /path/to/avro-build \
  -DCMAKE_BUILD_TYPE=Debug \
  -DAVRO_BUILD_TESTS=OFF \
  -DAVRO_BUILD_EXECUTABLES=ON \
  -DAVRO_BUILD_SHARED=OFF \
  -DAVRO_BUILD_STATIC=ON \
  -DCMAKE_CXX_FLAGS="-fsanitize=address,undefined -fno-omit-frame-pointer" \
  -DCMAKE_EXE_LINKER_FLAGS="-fsanitize=address,undefined"
cmake --build /path/to/avro-build --parallel 4

Run the packaged reproducer:

AVRO_SRC=/path/to/avro/lang/c++ \
AVRO_BUILD=/path/to/avro-build \
./reproduce.sh

Expected output:

== control ==
EOF records=1
== trigger ==
LIMIT_REACHED records=1000000 reader_still_returning=true
trigger reproduced: the reader returned 1,000,000 zero-byte records without EOF

The one-million-record cap belongs to the PoC harness. Removing it leaves the ordinary while (reader.read(datum)) loop running indefinitely.

Verification

Three instrumented runs used a five-million-record safety cap:

• control: EOF after exactly one record in 3/3 runs;
• trigger: still returned records at 5,000,000 in 3/3 runs;
• trigger user CPU time: 0.57–0.58 seconds per capped run;
• sanitizer errors: none required; this is a parser state-machine flaw.

Artifacts:

• models/control-one-empty-record.avro
  • size: 113 bytes
  • SHA-256: 01f0e4309fd0e1a1b222945b98050e4982d83afc7c107a58cfe2306d8ba32463
• models/trigger-negative-object-count.avro
  • size: 113 bytes
  • SHA-256: 8fc76abb89503e59d424a66de7e4775dfb0a7a1be4a29f3fd64d51301d87257d

Security impact

A service, converter, ETL process, or CLI that iterates over records from an untrusted Avro OCF can enter an unbounded CPU loop. Downstream consumers that materialize, count, log, or forward the returned records also experience unbounded output amplification from a 113-byte file.

The file does not need compression, a large allocation, or a large payload. The malformed count is the only changed byte.

Suggested fix

Reject negative top-level data-block object counts immediately after decoding them in DataFileReaderBase::readDataBlock(). Also reject negative byteCount values before converting them to size_t.

Add regression cases for:

• objectCount_ == -1 with an empty-record schema;
• objectCount_ == INT64_MIN;
• negative byteCount;
• zero-count and one-count valid controls.

Prior art

The public scan found no exact report of the top-level C++ OCF objectCount_ state-machine flaw.

Closest distinct work:

• AVRO-4228: negative array block counts in arrayNext();
• AVRO-4278/4294: array/map allocation and zero-byte-element limits;
• PR #3623: negative OCF byte size validation in the C implementation;
• AVRO-3587: a 2022 compressed-block heap-buffer-overflow in readDataBlock().

None validates the C++ OCF object count before hasMore() and decr().