FoolDev Claude Opus 4.7 commited on
Commit
aaef90f
·
1 Parent(s): 2bf2adf

heal-hf: validate the rewritten manifest via 'ollama show' + roll back on failure

Browse files

Before this, `make heal-hf` trusted its own rewrite: after the
`jq` digest-substitution succeeded and the atomic `mv` into place
landed, the script proceeded to remove the old qwen36 blob and
declare success. A rewrite that produced jq-parseable but
ollama-rejected JSON — e.g. layer order off, missing mediaType,
digest no longer matching the on-disk blob bytes — would only
surface at the user's next `ollama run`, by which point the old
blob was gone and the tag was stuck in a half-broken state.

Two changes to make the script crash-safe:

1. **Backup before rewrite, restore on validation failure.** Step 5
now `cp`s the original manifest to `${MANIFEST}.heal-backup`
before invoking jq, runs the rewrite + atomic `mv` as before,
then runs `ollama show ${TAG} >/dev/null` as a self-test.
`ollama show` parses the manifest, walks layer digests, and
reads enough of the model blob to extract metadata (arch,
params, etc.) — it's exactly the surface that needs to hold
for `ollama run` to subsequently load. On failure the script
`mv`s the backup back over the manifest, prints a diagnostic,
and exits non-zero. On success the backup is removed.

2. **Reordered old-blob cleanup to come AFTER validation.** What
was step 6 (remove old qwen36 blob if no other manifest refs
it) is now step 7, gated behind step 6 (validate). This means
a rollback always lands on a consistent state: original
manifest still pointing at the original blob, both still
present. The new qwen35 blob is left in the store as an
orphan; ollama auto-prunes unreferenced blobs on next service
restart, so disk usage tidies up on its own.

Doesn't change runtime materially — `ollama show` finishes in
milliseconds because it only reads GGUF metadata (KV header), not
tensor data.

Verified end-to-end this session against the actual broken pull:
manifest rewrite + `ollama show` validation pass, smoke + tool-call
round-trip continue to succeed on the healed `hf.co/...` tag. The
rollback path is defensive and exercised by reading the code, not
by an integration test (the heal logic itself works; the rollback
exists for a hypothetical future regression).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Files changed (2) hide show
  1. CHANGELOG.md +18 -0
  2. scripts/heal_hf_pull.sh +30 -1
CHANGELOG.md CHANGED
@@ -7,6 +7,24 @@ and documentation**, not the underlying base model.
7
 
8
  ## [Unreleased]
9
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
10
  ## [0.6.0] - 2026-05-19 — `c336f44`
11
 
12
  Headline changes since 0.5.0 (over a two-week burst on 2026-05-19):
 
7
 
8
  ## [Unreleased]
9
 
10
+ ### Added
11
+ - `scripts/heal_hf_pull.sh` now validates the rewritten manifest by
12
+ running `ollama show <tag>` immediately after the atomic mv into
13
+ place, with rollback-on-failure semantics. Before this, a buggy
14
+ rewrite that produced jq-parseable but ollama-rejected JSON (e.g.
15
+ layer order off, missing required mediaType, digest that no
16
+ longer matches the on-disk blob bytes) would only be discovered
17
+ at the user's `ollama run` step — and by then the old qwen36
18
+ blob had already been removed, leaving the tag in a half-broken
19
+ state with no easy recovery. Now: the script `cp`s the original
20
+ manifest to a `.heal-backup` sidecar before the rewrite, runs
21
+ `ollama show` against the rewritten manifest, and on failure
22
+ `mv`s the backup back into place and exits non-zero. The old
23
+ qwen36 blob cleanup (step 7, was step 6) only runs after
24
+ validation passes, so a rollback always lands on a consistent
25
+ state (original manifest + original blob still present). Backup
26
+ is deleted on success.
27
+
28
  ## [0.6.0] - 2026-05-19 — `c336f44`
29
 
30
  Headline changes since 0.5.0 (over a two-week burst on 2026-05-19):
scripts/heal_hf_pull.sh CHANGED
@@ -135,6 +135,13 @@ trap - EXIT
135
 
136
  # ---- 5. Rewrite the manifest's model layer ----------------------------------
137
 
 
 
 
 
 
 
 
138
  TMP_MANIFEST="$(mktemp -t thanatos-heal-manifest.XXXXXX.json)"
139
  trap 'rm -f "${TMP_MANIFEST}"' EXIT
140
  jq --arg new "sha256:${NEW_HASH}" \
@@ -152,12 +159,34 @@ NEW_DIGEST_IN_MANIFEST="$(jq -r '
152
  ' "${TMP_MANIFEST}")"
153
  if [[ "${NEW_DIGEST_IN_MANIFEST}" != "sha256:${NEW_HASH}" ]]; then
154
  red "[!] manifest rewrite failed (digest mismatch); not committing."
 
155
  exit 1
156
  fi
157
  mv "${TMP_MANIFEST}" "${MANIFEST}"
158
  trap - EXIT
159
 
160
- # ---- 6. Remove the old qwen36 blob if no other manifest references it -------
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
161
 
162
  OLD_DIGEST="sha256:${MODEL_HASH}"
163
  if ! grep -rlF -- "${OLD_DIGEST}" "${OLLAMA_MODELS}/manifests/" >/dev/null 2>&1; then
 
135
 
136
  # ---- 5. Rewrite the manifest's model layer ----------------------------------
137
 
138
+ # Keep a sidecar copy of the original manifest so we can roll back if
139
+ # `ollama show` rejects the rewrite. The blob cleanup in step 7 only
140
+ # happens AFTER validation passes, so a rollback always lands on a
141
+ # consistent (original manifest, original blob still present) state.
142
+ BACKUP_MANIFEST="${MANIFEST}.heal-backup"
143
+ cp "${MANIFEST}" "${BACKUP_MANIFEST}"
144
+
145
  TMP_MANIFEST="$(mktemp -t thanatos-heal-manifest.XXXXXX.json)"
146
  trap 'rm -f "${TMP_MANIFEST}"' EXIT
147
  jq --arg new "sha256:${NEW_HASH}" \
 
159
  ' "${TMP_MANIFEST}")"
160
  if [[ "${NEW_DIGEST_IN_MANIFEST}" != "sha256:${NEW_HASH}" ]]; then
161
  red "[!] manifest rewrite failed (digest mismatch); not committing."
162
+ rm -f "${BACKUP_MANIFEST}"
163
  exit 1
164
  fi
165
  mv "${TMP_MANIFEST}" "${MANIFEST}"
166
  trap - EXIT
167
 
168
+ # ---- 6. Validate the rewritten manifest via ollama show ---------------------
169
+
170
+ # `ollama show` parses the manifest, walks the layer digests, and reads
171
+ # enough of the model blob to extract metadata (arch, params, etc.).
172
+ # A buggy rewrite that produces jq-accepted JSON but breaks an ollama
173
+ # invariant (layer order, mediaType set, digest-vs-blob-bytes
174
+ # mismatch) trips here, before we lose the rollback path by removing
175
+ # the old qwen36 blob. On failure we restore from BACKUP_MANIFEST.
176
+ blue "[*] validating rewritten manifest with 'ollama show'..."
177
+ if ! ollama show "${TAG}" >/dev/null 2>&1; then
178
+ red "[!] ollama show ${TAG} failed after the manifest rewrite."
179
+ blue "[*] rolling back: restoring original manifest from ${BACKUP_MANIFEST}"
180
+ mv "${BACKUP_MANIFEST}" "${MANIFEST}"
181
+ red " Tag is back to its pre-heal (qwen36) state. The new qwen35"
182
+ red " blob is left in the store; ollama auto-prunes unreferenced"
183
+ red " blobs on next restart."
184
+ exit 1
185
+ fi
186
+ rm -f "${BACKUP_MANIFEST}"
187
+ green "[+] manifest validates."
188
+
189
+ # ---- 7. Remove the old qwen36 blob if no other manifest references it -------
190
 
191
  OLD_DIGEST="sha256:${MODEL_HASH}"
192
  if ! grep -rlF -- "${OLD_DIGEST}" "${OLLAMA_MODELS}/manifests/" >/dev/null 2>&1; then