EnigmaConsultant/huntr-poc-joblib-stopbyte

modelscan fails to detect RCE in an uncompressed .joblib file: numpy array alignment-padding byte (0x2e) is misparsed by pickletools.genops as a STOP opcode, halting the scan before the payload while joblib.load executes it

Target / affected component

• huntr "Model File Formats" dropdown value: joblib (.joblib)
• Vulnerable tool: modelscan (Protect AI) — the static scanner that is supposed to flag the file as unsafe.
• Version tested: modelscan 0.8.8 (latest on PyPI at time of writing).
• Supporting environment: joblib 1.5.3, numpy 2.4.6, CPython 3.12. The desync is a property of the joblib on-disk array format and of pickletools.genops, so it is not specific to these point versions.
• Scanner module exercised: modelscan.scanners.PickleUnsafeOpScan (confirmed in CLI output below — modelscan does select its pickle scanner for the .joblib file; it simply mis-parses it).

Severity

High. This is a complete bypass of the scanner's core guarantee for a first-class supported format: a file that achieves arbitrary code execution on joblib.load() is reported by modelscan as containing zero issues, with no scan error and no skipped file. A user who follows the canonical "scan untrusted models before loading them" workflow is given an explicit clean bill of health and then compromised on load.

For the huntr Model File Formats program this corresponds to the scanner-evasion / detection-bypass tier (the scanner returns a false-negative on a malicious model file). I am claiming the bypass itself, not a new code-execution sink in joblib — joblib pickle RCE is known and by design; the defect is that modelscan declares the crafted file clean.

Summary

joblib stores a numpy ndarray inside an otherwise ordinary pickle stream using its NumpyArrayWrapper mechanism. The array's raw little-endian bytes (and an alignment/padding region) are written inline into the same byte stream, after the wrapper's BUILD opcode and before the stream's terminating STOP. These inline bytes are data, not pickle opcodes.

Two parsers read this one stream and disagree about where the opcodes are:

• The real loader, joblib.numpy_pickle.NumpyUnpickler, overrides BUILD. When it reaches the wrapper it calls NumpyArrayWrapper.read_array(), which knows the array's dtype and shape, reads the 1-byte padding length, skips that many padding bytes, then consumes exactly count * itemsize raw data bytes from the file handle — and resumes the pickle VM at the first byte after the array payload.
• modelscan's pickle scanner, modelscan.tools.picklescanner._list_globals(), runs pickletools.genops() linearly over the raw bytes. It has no knowledge of read_array(), no dtype/shape context, and therefore cannot skip the inline array payload. It parses those data bytes as if they were opcodes.

I exploit this by setting the array's alignment padding-length byte to 0x2e. To joblib that byte is the integer 46 ("skip 46 padding bytes"). To pickletools.genops, 0x2e is the ASCII '.' — the STOP opcode. genops therefore believes the pickle ended cleanly immediately after the benign wrapper, having seen only innocuous globals (joblib...NumpyArrayWrapper, numpy.ndarray, numpy.dtype). The malicious GLOBAL os.system / REDUCE opcodes placed further down the stream are never parsed by the scanner, but joblib.load resumes exactly on them and executes them.

This also defeats modelscan's multiple_pickles mitigation (the loop intended to catch payloads hidden after extra STOP opcodes): after the spoofed STOP, the loop re-enters genops at the next byte, which is raw padding (0xff) — not a valid opcode — so genops raises immediately. Because some benign globals were already collected, modelscan treats this as "valid pickle, nothing dangerous" and returns a clean result instead of an error.

Root cause

The bug is a parser differential between modelscan's linear opcode walk and joblib's content-aware array reader. Exact locations (paths from the tested install):

modelscan — the false negative (modelscan/tools/picklescanner.py):

_list_globals() (lines ~50–119) drives the scan with a linear genops loop and treats a genops exception as "end of a pickle, keep whatever globals we already have":

# picklescanner.py  (_list_globals)
last_byte = b"dummy"
while last_byte != b"":
    try:
        ops = list(pickletools.genops(data))          # <-- linear opcode walk; no array-payload awareness
    except Exception as e:
        globals_opt = globals if len(globals) > 0 else None
        raise GenOpsError(str(e), globals_opt)         # <-- carries already-found (benign) globals upward
    ...
    if not multiple_pickles:
        break

genops reads the inline numpy data as opcodes. The 0x2e padding byte is decoded as STOP, ending iteration 0 with only the benign wrapper globals. Iteration 1 begins on a 0xff padding byte, which genops rejects as an unknown opcode and raises. The caller (scan_pickle_bytes and the pickle scanner in modelscan/scanners/pickle/scan.py) treats a GenOpsError that carries a non-empty global set as a successfully-parsed-but-benign pickle, so the file is reported scanned, 0 issues. The downstream GLOBAL os.system is never examined.

joblib — why the same bytes are safe to it and lead to execution (joblib/numpy_pickle.py):

NumpyUnpickler.load_build() (lines ~438–467) intercepts BUILD and hands off to the wrapper:

# numpy_pickle.py  (load_build)
dispatch[pickle.BUILD[0]] = load_build
...
_array_payload = array_wrapper.read(self, self.ensure_native_byte_order)

NumpyArrayWrapper.read_array() (lines ~159–213) consumes the padding byte, the padding, and exactly the declared number of data bytes — then returns, so the VM continues after the array:

# numpy_pickle.py  (read_array)
padding_byte = unpickler.file_handle.read(1)              # reads 0x2e as a LENGTH (46), not an opcode
padding_length = int.from_bytes(padding_byte, "little")
if padding_length != 0:
    unpickler.file_handle.read(padding_length)            # skips 46 padding bytes
...
data = _read_bytes(unpickler.file_handle, read_size, "array data")   # consumes N declared data bytes

Because joblib advances the file handle past 1 + 46 + N bytes that modelscan tried to interpret as opcodes, the two parsers are desynchronised by exactly that span. joblib's VM resumes on the attacker's GLOBAL os.system ; (cmd,) ; REDUCE ; STOP.

One-line statement of the flaw: modelscan parses a joblib-embedded numpy array as pickle opcodes; a single array byte equal to 0x2e is read as STOP, so the scanner stops before the payload, while joblib — which reads that byte as an array padding length — does not.

Proof of concept

Self-contained script: reproduce.py (in this directory). It (1) builds the file, (2) scans with modelscan, (3) loads with joblib.load. The default payload is a benign marker write.

Build (what the bytes are)

A single uncompressed pickle stream:

[ NumpyArrayWrapper opcodes ... BUILD (0x62) ]   # benign globals only: NumpyArrayWrapper, numpy.ndarray, numpy.dtype
[ 0x2e ]                                          # alignment padding-LENGTH byte == '.'  (46 to joblib, STOP to genops)
[ 0xff * 46 ]                                     # padding bytes; joblib skips them
[ 0x00 * N ]                                      # N raw uint8 array-data bytes; joblib consumes them as the array
[ c posix\nsystem\n  <cmd>  TUPLE1 REDUCE STOP ] # joblib resumes HERE; genops never reaches it

The NumpyArrayWrapper prefix is sliced byte-for-byte out of a genuine joblib.dump(np.zeros(N, np.uint8)), so the wrapper is exactly what joblib emits. Only the single padding-length byte is chosen by the attacker (0x2e); everything before it is legitimate joblib output, which is why the file loads without error.

Two-part assertion (captured output, host = CPython 3.12, modelscan 0.8.8, joblib 1.5.3, numpy 2.4.6)

Part A — modelscan reports the malicious file CLEAN (Python API, from reproduce.py):

[*] PART A -- scan with modelscan (the defense) ...
    total_issues   0
    by_severity    {'LOW': 0, 'MEDIUM': 0, 'HIGH': 0, 'CRITICAL': 0}
    total_scanned  1
    scanned_files  ['poc_clean_but_rce.joblib']
    errors         []
    issues         []
    => scanner reports CLEAN: True

The same result via the official modelscan CLI (UTF-8 console), confirming it is the real pickle scanner and the real report path:

$ modelscan -p poc_clean_but_rce.joblib
Scanning .../poc_clean_but_rce.joblib using modelscan.scanners.PickleUnsafeOpScan model scan

--- Summary ---

 No issues found! (celebration emoji)

Part B — joblib.load executes the embedded reduce (from reproduce.py):

[*] PART B -- load with joblib.load (the victim's real loader) ...
    joblib.load returned: int 0
    marker file created : True
    marker path/content : /tmp/modelscan_joblib_poc_marker -> 'POC_RCE'

==== VERDICT ====
A) modelscan: 0 issues, scanned>=1, no errors : True
B) joblib.load executed embedded code         : True

RESULT: CONFIRMED -- modelscan-clean yet joblib.load RCE.

(The captured marker path above shows the portable /tmp location reproduce.py uses on Linux; on the Windows host where this was also run it was %TEMP%\modelscan_joblib_poc_marker. The payload is os.system("echo POC_RCE > <marker>") — a stand-in for arbitrary command execution. An arbitrary-file-read variant, cat /etc/passwd > <marker>, is included commented-out in reproduce.py.)

Mechanism trace (why the scanner stops early)

Instrumenting modelscan's exact multiple_pickles loop over the crafted file:

malicious 'posix.system' / 'nt.system' opcode region starts at offset: 269   (varies with payload length)
iter 0: genops OK [0:221] ops_count=68 ends_with=STOP          # the 0x2e padding byte decoded as STOP
iter 1: genops ERROR at ~222: opcode b'\xff' unknown  (globals so far: [])   # next byte is raw padding
   -> modelscan raises GenOpsError(globals nonempty) -> returns benign result, scanned=1
=> the os.system payload is NEVER seen by modelscan, but joblib executes it.

The scanner's maximum parse offset (221) is far short of the payload offset (269): the malicious opcodes are structurally unreachable by the linear walk.

Impact

Realistic threat model — the exact workflow modelscan exists to protect:

1. A practitioner downloads an untrusted model.joblib (Hugging Face, a shared bucket, a teammate, a CI artifact). .joblib is a first-class, officially supported modelscan format and a very common scikit-learn/joblib serialization.
2. They run modelscan -p model.joblib (or the equivalent in a gate/CI step). modelscan returns 0 issues / No issues found. There is no error and no "skipped" notice that might prompt manual review — the file looks fully and successfully scanned.
3. Trusting that result, they joblib.load("model.joblib"). The embedded reduce runs with the user's privileges: arbitrary command execution, arbitrary file read/exfiltration (e.g. cloud credentials, ~/.ssh, /etc/passwd), reverse shell, or lateral movement in CI.

The bypass needs no compression (so it is not the previously reported compressed-stream evasion), requires only that the victim loads with joblib (the format's native loader), and produces a file that is otherwise a valid, loadable joblib array object — so it survives casual inspection and joblib.load round-trips cleanly. A scanner false-negative here is arguably worse than no scanner, because it converts "I should be cautious with this file" into "the tool told me it's safe."

Honest duplicate / prior-art note

Pickle-based RCE via joblib is well known and is not what I am claiming. What I am claiming is this specific modelscan false-negative on an uncompressed joblib file driven by a numpy-array padding byte equal to 0x2e being consumed as a STOP opcode. Nearest public prior art and why this is distinct:

• The published modelscan compression-bypass / multiple-pickles work. Prior bypasses hide the payload behind compression or behind extra real STOP opcodes, which is precisely what the multiple_pickles=True loop was added to defeat. This finding uses no compression and no extra real STOP: the "STOP" is a single raw byte inside the numpy array payload (an alignment padding length), and the bytes between it and the real payload are raw little-endian array data that genops cannot parse but joblib deliberately skips via read_array(). The multiple_pickles mitigation does not help — it re-enters genops on a 0xff padding byte and aborts with the benign globals already banked. So this survives the patch that closed the earlier multi-STOP bypasses.

• arXiv:2508.19774 (model-serialization scanner-evasion survey). That work does not cover this vector. It does not discuss joblib's NumpyArrayWrapper, the substitution of a STOP using raw numpy array bytes, or padding-byte STOP spoofing (using the array's alignment padding-length field as the fake terminator). The mechanism here is specific to joblib's content-aware array reader desynchronising from a content-blind opcode walk, which that survey does not describe.

If a triager nonetheless considers this the same root class as a previously accepted joblib/modelscan desync, I defer to that judgment — but the trigger (alignment padding-length byte = 0x2e, uncompressed, defeating multiple_pickles) is, to my knowledge, not publicly documented.

Remediation

The root issue is that modelscan walks joblib's array bytes as if they were opcodes. Options, strongest first:

1. Parse joblib arrays the way joblib does. When the stream contains a NumpyArrayWrapper BUILD, the scanner must replicate read_array()'s skip — read the dtype/shape from the wrapper state, consume the padding-length byte + padding + count * itemsize data bytes — and resume genops after the array, instead of letting genops interpret those bytes. This removes the desync at the source.
2. Stop trusting a GenOpsError-with-globals as "scanned & clean." A genops failure on a byte that is not a valid opcode (here, 0xff right after a STOP) means the scanner lost stream sync. modelscan should treat "parse error before EOF, with trailing unparsed bytes" as inconclusive/error, not as a clean pickle — i.e. surface it as a scan error or a HIGH "unscannable region" issue so the user does not get a false all-clear. At minimum, a STOP that is immediately followed by bytes which fail to parse as a fresh pickle should not be accepted as a normal stream terminator.
3. Defense in depth: when scanning .joblib, detect the NumpyArrayWrapper global and require the array region to be accounted for byte-exactly; if trailing bytes remain after the declared arrays and the final STOP, scan them as an additional pickle (extending the spirit of multiple_pickles) rather than discarding them.

Option 1 is the correct long-term fix; option 2 alone already converts this from a silent false-negative into a visible, actionable error.