hdf5poc / README.md
pixelated404's picture
Update README.md
e07d7b8 verified
|
Raw
History Blame Contribute Delete
2.61 kB
---
license: mit
library_name: keras
tags:
- security
- proof-of-concept
- keras
- hdf5
---
## Technical Background
`keras.models.load_model()` supports loading Keras models from `.h5` and `.keras`
files. Internally, weight datasets are read through `safe_get_h5_dataset()`, which
implements a decompression bomb guard that checks whether the declared dataset
size (based on shape and dtype) exceeds a 4 GiB floor and is more than 1000 times
the stored size on disk.
The guard logic is:
```python
declared_bytes = math.prod(dataset.shape) * dataset.dtype.itemsize
stored_bytes = dataset.id.get_storage_size()
if declared_bytes > _H5_DATASET_BOMB_FLOOR_BYTES and declared_bytes > 1000 * stored_bytes:
raise ValueError(...)
```
The constant `_H5_DATASET_BOMB_FLOOR_BYTES` is set to `1 << 32` (4 GiB).
Any dataset whose `declared_bytes` is **at or below** this floor is never
checked against the stored-disk size, regardless of the expansion ratio.
A dataset with shape `(1073741823,)` and dtype `float32` has a declared size
of 4,294,967,292 bytes (4 GiB − 4 bytes), which is below the 4 GiB floor.
When stored as a fill-value-only dataset with no chunks written, the stored
size on disk is zero bytes. The expansion ratio is approximately 3 million to one,
but the guard does not evaluate it.
Callers of `safe_get_h5_dataset()` pass the returned dataset handle directly
to `np.array()` or `np.asarray()`, which performs the full allocation before
any later shape validation.
## Tested Versions
- Python 3.12.10
- Keras 3.15.0
- TensorFlow 2.21.0
- h5py 3.14.0
## Reproducing
Both `.h5` and `.keras` variants are provided.
### Legacy HDF5 format (`.h5`)
```python
keras.models.load_model("bomb.h5")
```
### Native Keras format (`.keras`)
```python
keras.models.load_model("bomb.keras")
```
### Using the verification script
```python
python verify_poc.py
```
This script prints package versions, attempts both loads, and reports which
stage triggered a `MemoryError` or `ValueError`.
## Expected Behavior
When the model file is loaded, the dataset guard evaluates but does not reject
the dataset because its declared size (under 4 GiB) is below the guard floor.
The process then attempts to allocate memory for the declared shape. On systems
with insufficient available memory, this raises `MemoryError`. On systems with
sufficient memory, the allocation succeeds and a subsequent shape validation
raises `ValueError` because the bomb weight shape does not match the layer
architecture.
In both cases the memory allocation occurs **before** any model validation
can reject the file.