Instructions to use pixelated404/hdf5poc with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Keras
How to use pixelated404/hdf5poc with Keras:
# Available backend options are: "jax", "torch", "tensorflow". import os os.environ["KERAS_BACKEND"] = "jax" import keras model = keras.saving.load_model("hf://pixelated404/hdf5poc") - Notebooks
- Google Colab
- Kaggle
| license: mit | |
| library_name: keras | |
| tags: | |
| - security | |
| - proof-of-concept | |
| - keras | |
| - hdf5 | |
| ## Technical Background | |
| `keras.models.load_model()` supports loading Keras models from `.h5` and `.keras` | |
| files. Internally, weight datasets are read through `safe_get_h5_dataset()`, which | |
| implements a decompression bomb guard that checks whether the declared dataset | |
| size (based on shape and dtype) exceeds a 4 GiB floor and is more than 1000 times | |
| the stored size on disk. | |
| The guard logic is: | |
| ```python | |
| declared_bytes = math.prod(dataset.shape) * dataset.dtype.itemsize | |
| stored_bytes = dataset.id.get_storage_size() | |
| if declared_bytes > _H5_DATASET_BOMB_FLOOR_BYTES and declared_bytes > 1000 * stored_bytes: | |
| raise ValueError(...) | |
| ``` | |
| The constant `_H5_DATASET_BOMB_FLOOR_BYTES` is set to `1 << 32` (4 GiB). | |
| Any dataset whose `declared_bytes` is **at or below** this floor is never | |
| checked against the stored-disk size, regardless of the expansion ratio. | |
| A dataset with shape `(1073741823,)` and dtype `float32` has a declared size | |
| of 4,294,967,292 bytes (4 GiB − 4 bytes), which is below the 4 GiB floor. | |
| When stored as a fill-value-only dataset with no chunks written, the stored | |
| size on disk is zero bytes. The expansion ratio is approximately 3 million to one, | |
| but the guard does not evaluate it. | |
| Callers of `safe_get_h5_dataset()` pass the returned dataset handle directly | |
| to `np.array()` or `np.asarray()`, which performs the full allocation before | |
| any later shape validation. | |
| ## Tested Versions | |
| - Python 3.12.10 | |
| - Keras 3.15.0 | |
| - TensorFlow 2.21.0 | |
| - h5py 3.14.0 | |
| ## Reproducing | |
| Both `.h5` and `.keras` variants are provided. | |
| ### Legacy HDF5 format (`.h5`) | |
| ```python | |
| keras.models.load_model("bomb.h5") | |
| ``` | |
| ### Native Keras format (`.keras`) | |
| ```python | |
| keras.models.load_model("bomb.keras") | |
| ``` | |
| ### Using the verification script | |
| ```python | |
| python verify_poc.py | |
| ``` | |
| This script prints package versions, attempts both loads, and reports which | |
| stage triggered a `MemoryError` or `ValueError`. | |
| ## Expected Behavior | |
| When the model file is loaded, the dataset guard evaluates but does not reject | |
| the dataset because its declared size (under 4 GiB) is below the guard floor. | |
| The process then attempts to allocate memory for the declared shape. On systems | |
| with insufficient available memory, this raises `MemoryError`. On systems with | |
| sufficient memory, the allocation succeeds and a subsequent shape validation | |
| raises `ValueError` because the bomb weight shape does not match the layer | |
| architecture. | |
| In both cases the memory allocation occurs **before** any model validation | |
| can reject the file. | |