| --- |
| license: apache-2.0 |
| --- |
| |
| # TensorFlow tensor_bundle checkpoint reader: NULL-pointer-dereference DoS via oversized DT_STRING shape |
|
|
| **Target:** `github.com/tensorflow/tensorflow` β `tensorflow/core/util/tensor_bundle/tensor_bundle.cc` |
| (`BundleReader::GetValue`) and `tensorflow/core/framework/tensor.cc` |
| (`Tensor::TotalBytes` / `Helper<tstring>::TotalBytes` / `TypedAllocator::Allocate`). |
|
|
| **Tested against:** `tensorflow-cpu==2.21.0` (`v2.21.0-rc1-5-ga481b10260d`), installed from PyPI, |
| unmodified. Root cause confirmed present in the current `tensorflow/tensorflow` `master` branch |
| source as of 2026-07-06. |
|
|
| ## Summary |
|
|
| Loading an attacker-supplied TensorFlow checkpoint whose metadata (`.index`) file declares a |
| `DT_STRING` tensor with an astronomically large 1-D shape crashes the host process with a |
| `SIGSEGV` (NULL pointer dereference). No corresponding `.data-00000-of-00001` shard file is |
| required at all β the crash happens purely while TensorFlow validates the tensor's metadata, |
| before it ever tries to read the tensor's byte range from disk. The malicious `.index` file in |
| this repo is 142 bytes. |
|
|
| This is reachable through the standard, widely used checkpoint-inspection API: |
| `tf.train.load_checkpoint(prefix).get_tensor(key)` (this is what |
| `tensorflow.python.training.py_checkpoint_reader.CheckpointReader.get_tensor` / |
| `NewCheckpointReader` / the `inspect_checkpoint.py` CLI tool / many downstream libraries that peek |
| at checkpoint contents by key use under the hood). |
|
|
| ## Root cause |
|
|
| 1. `BundleReader::GetValue()` (`tensor_bundle.cc`) constructs a fresh `Tensor` straight from the |
| untrusted `BundleEntryProto`'s `dtype`/`shape` fields: |
| ```cpp |
| const TensorShape stored_shape(TensorShape(entry.shape())); |
| if (val->NumElements() == 0) { |
| ret = new Tensor(entry.dtype(), stored_shape); |
| } |
| ``` |
| `TensorShape::IsValid()` only rejects shapes whose element count overflows `int64`; it happily |
| accepts a single dimension of, e.g., `2^62 + 7` elements. |
|
|
| 2. `Tensor::Tensor(Allocator*, DataType, TensorShape)` (`tensor.cc`) allocates the backing buffer |
| via `TypedAllocator::Allocate<T>()`, which **by design** returns `nullptr` when the tensor is |
| too large to allocate (`typed_allocator.h`: *"May return NULL if the tensor has too many |
| elements to represent in a single allocation."*). The `Tensor` constructor does **not** check |
| this and unconditionally wraps the null-backed `Buffer<T>` β the resulting `Tensor` looks |
| "constructed" but is not actually backed by memory. |
|
|
| 3. Back in `GetValue()`, right after constructing that `Tensor`, for `DT_STRING` entries the code |
| computes a sanity bound by calling `ret->TotalBytes()`: |
| ```cpp |
| const size_t lower_bound = ret->NumElements() + ret->TotalBytes() - |
| sizeof(tstring) * ret->NumElements(); |
| ``` |
| `Tensor::TotalBytes()` dispatches to `Helper<tstring>::TotalBytes(buf_, shape_.num_elements())`. |
| That specialization's second parameter is declared as a 32-bit `int` (unlike the generic |
| template, which uses `int64_t`), so the 64-bit element count is silently truncated. It then |
| unconditionally dereferences the (null) string array: |
| ```cpp |
| static int64_t TotalBytes(TensorBuffer* in, int n) { |
| int64_t tot = in->size(); |
| const tstring* p = in->base<const tstring>(); // == nullptr |
| for (int i = 0; i < n; ++i, ++p) tot += p->size(); // crash: p->size() on nullptr |
| return tot; |
| } |
| ``` |
| With `n > 0` (guaranteed by choosing the low 32 bits of the shape to be a small positive |
| number) this is a guaranteed NULL pointer dereference β no OOM, no huge malloc attempt, no |
| timing dependency. It happens **before** `entry.size()`, `entry.offset()`, `entry.crc32c()`, or |
| the `.data` shard are ever consulted. |
|
|
| ## Attacker model / reachability |
|
|
| The attacker only needs to control the checkpoint's `.index` metadata file (a simple |
| leveldb/sstable-style table of serialized `BundleEntryProto`s) β a totally standard "malicious |
| model/checkpoint shared with a victim" scenario (e.g. a checkpoint uploaded to a model hub, or |
| bundled inside a SavedModel/ckpt directory a user is asked to load or merely *inspect*). |
| `get_tensor()`/`load_checkpoint()` is exactly the API many tools and libraries use to peek at a |
| checkpoint's contents (list/print/convert weights) without first building or restoring a full |
| graph, so the crash is reachable without the victim doing anything beyond pointing a checkpoint |
| inspection call at the attacker's file. |
|
|
| Note: `tf.raw_ops.RestoreV2` (the op behind `tf.train.Checkpoint.restore()`) goes through |
| `OpKernelContext::allocate_output`, which *does* check for a failed allocation and returns a clean |
| `ResourceExhaustedError` instead of crashing β so the vulnerable surface is specifically the |
| direct `BundleReader::Lookup` / `CheckpointReader::GetTensor` C++ API (`get_tensor`, |
| `get_variable_to_dtype_map` is safe, only `get_tensor` on a `DT_STRING` key triggers it), not the |
| `RestoreV2` graph op. |
|
|
| ## Reproduction |
|
|
| ``` |
| pip install tensorflow-cpu==2.21.0 crc32c |
| python3 build_malicious_ckpt.py /tmp/poison_ckpt/model.ckpt # writes only model.ckpt.index (142 bytes) |
| python3 poc_trigger_fh.py /tmp/poison_ckpt/model.ckpt |
| ``` |
|
|
| Observed output (see `crash_evidence.log` for a full transcript): |
|
|
| ``` |
| Loading checkpoint: /tmp/poison_ckpt/model.ckpt |
| dtype map: {'poison': tf.string} |
| Calling get_tensor('poison') -- expecting SIGSEGV from the real BundleReader/Tensor code path |
| Fatal Python error: Segmentation fault |
| |
| Current thread 0x00007f4ea2d82200 (most recent call first): |
| File ".../tensorflow/python/training/py_checkpoint_reader.py", line 66 in get_tensor |
| File "poc_trigger_fh.py", line 10 in <module> |
| ``` |
|
|
| Shell exit code is 139 (128 + SIGSEGV) in both `poc_trigger.py` (no `faulthandler`) and |
| `poc_trigger_fh.py` (with `faulthandler.enable()`, which confirms the crash is a genuine hardware |
| signal, not a caught Python exception). |
|
|
| ## Files |
|
|
| - `build_malicious_ckpt.py` β stand-alone leveldb/sstable-table encoder (re-implemented from |
| `xla/tsl/lib/io/{format,table_builder,block_builder}.cc` + `tensor_bundle.proto`) that hand-crafts |
| the malicious 142-byte `.index` metadata file. No TensorFlow write APIs are used to build the |
| malicious file β it is built entirely from raw bytes to demonstrate a real attacker only needs |
| to control the file format, not any TensorFlow API. |
| - `poison_model.ckpt.index` β the pre-built malicious checkpoint metadata (142 bytes, no `.data` |
| shard needed). |
| - `poc_trigger.py`, `poc_trigger_fh.py` β trigger scripts via `tf.train.load_checkpoint(...).get_tensor()`. |
| - `poc_restorev2.py` β comparison script showing the (safe) `tf.raw_ops.RestoreV2` behavior for |
| context. |
| - `crash_evidence.log` β captured transcript of the actual crash on `tensorflow-cpu==2.21.0`. |
|
|
| ## Impact |
|
|
| Denial of service: any process that inspects/loads an attacker-supplied checkpoint via |
| `tf.train.load_checkpoint(...).get_tensor()` (or equivalent `CheckpointReader` usage) crashes |
| immediately and unrecoverably (SIGSEGV, not a catchable Python exception) on a 142-byte malicious |
| input file, with no `.data` shard required. |
|
|
| ## Scope note (for the triager) |
|
|
| TensorFlow's own `SECURITY.md` states that "loading untrusted checkpoints or graphs is equivalent |
| to running untrusted code" and that memory corruption is only considered a TensorFlow-side |
| security issue when reachable through a "production-grade, benign model." We flag this |
| transparently: TensorFlow upstream would likely triage a checkpoint-triggered crash as |
| out-of-policy for a TF-side CVE/advisory. We are reporting it here because huntr's TensorFlow |
| checkpoint-reader bounty target treats "a malicious/untrusted model or checkpoint file crashes the |
| loader" as precisely the in-scope threat model for the ML-supply-chain bounty program (that is the |
| premise of the model-file-format bounty category), and the crash is a concrete, deterministic, |
| 100%-reproducible NULL dereference (not a fuzz-only theoretical OOM) triggered by 142 bytes with |
| no real tensor payload. |
|
|