docs/FAILURES.md

Failure Notes

Purpose

This file tracks reproducible failures and fallback behavior for Objectverse Diary.

Use it for model/runtime/deployment/data issues, not for UI polish notes.

Current Status

MiniCPM-V 2.6 is wired as an optional vision backend. Hosted Space ZeroGPU validation now passes for public mug, keyboard, and shoe images after adding an HF_TOKEN Space secret with access to the gated openbmb/MiniCPM-V-2_6 model.

The app includes a hidden /vision_runtime_probe API and scripts/check_space_vlm.py writes probe output into the Space VLM report before image validation. This probe identified the previous failure as a gated-model access issue rather than a GPU or dependency issue.

The published LoRA v2 GGUF for local text smoke testing is available and has passed local llama.cpp smoke:

• repo: qqyule/objectverse-diary-qwen15b-lora
• file: objectverse-diary-qwen15b-lora-v2-q4_k_m.gguf
• helper: scripts/check_llama_cpp_smoke.py

Known non-blocking warning:

• Gradio emits deprecation warnings for upcoming 6.0 API changes during local tests. This does not break the current Gradio Blocks build and can be handled with the later UI/API polish pass.

Failure Record Template

## YYYY-MM-DD - Short Failure Name

- Area:
- Reproduction:
- Expected:
- Actual:
- Impact:
- Fallback used:
- Resolution:
- Evidence:

2026-06-08 - Hosted ZeroGPU MiniCPM-V Falls Back To Mock

• Area: Hugging Face Space vision runtime.
• Reproduction: Run scripts/check_space_vlm.py with --configure-space --hardware zero-a10g --rollback-to-mock against build-small-hackathon/ObjectverseDiary.
• Expected: mug, keyboard, and shoe validations use minicpm-v object understanding without vision-fallback-to-mock.
• Actual: all three validations returned schema-valid traces, but every trace included vision-fallback-to-mock.
• Impact: hosted Space MiniCPM-V evidence is not ready for submission; stable mock demo remains usable.
• Fallback used: mock object understanding plus mock text runtime.
• Resolution: unresolved; inspect Space runtime logs or add non-secret fallback diagnostics for the MiniCPM-V load/chat exception.
• Evidence: docs/SPACE_VLM_REPORT.md, docs/SPACE_VLM_REPORT.json, and data/traces/space-vlm/.

2026-06-08 - Hosted ZeroGPU MiniCPM-V Validation Passes After HF_TOKEN Secret

• Area: Hugging Face Space vision runtime.
• Reproduction: Run scripts/check_space_vlm.py with --configure-space --hardware zero-a10g --rollback-to-mock against build-small-hackathon/ObjectverseDiary.
• Expected: mug, keyboard, and shoe validations use minicpm-v object understanding without vision-fallback-to-mock.
• Actual: all three validations passed; probe reported minicpm_load_ok=True; traces include only mock-text-runtime.
• Impact: OpenBMB / hosted MiniCPM-V vision evidence is ready. Public Space still rolls back to mock-safe defaults after validation.
• Resolution: resolved by adding an HF_TOKEN Space secret with gated model access.
• Evidence: docs/SPACE_VLM_REPORT.md, docs/SPACE_VLM_REPORT.json, and data/traces/space-vlm/.

Previous Space VLM Validation Failure

• Updated: 2026-06-08 08:33:19 UTC
• Area: Hugging Face Space vision runtime.
• Probe backend: minicpm-v
• MiniCPM load attempted: True
• MiniCPM load ok: False
• Probe errors: minicpm_load=OSError
• Failed checks: mug: vision fallback marker was present; keyboard: vision fallback marker was present; shoe: vision fallback marker was present
• Fallback used: mock object understanding plus mock text runtime if validation reaches generation.
• Resolution: unresolved; keep the public Space mock-safe until this section reports a passing VLM validation.

2026-06-08 - LoRA v2 GGUF Local Smoke Passed

• Area: llama.cpp text runtime evidence.
• Reproduction: Run scripts/check_llama_cpp_smoke.py with models/objectverse-diary-qwen15b-lora-v2-q4_k_m.gguf after optional llama-cpp-python installation.
• Expected: trace records llama-cpp text generation, persona/diary/chat run without text-fallback-to-mock.
• Actual: passed locally; trace included only mock-vision-runtime because text was real and vision remained mock for the smoke input.
• Impact: local llama.cpp text runtime evidence is ready. Public Space text runtime is still not validated with this GGUF.
• Fallback used: none for text.
• Resolution: resolved locally by using the merged LoRA v2 Q4_K_M GGUF and conservative JSON extraction / decoding settings.
• Evidence: scripts/check_llama_cpp_smoke.py, docs/RUNTIME.md, and the Hub file in qqyule/objectverse-diary-qwen15b-lora.

2026-06-08 - Hugging Face Xet GGUF Upload Stalled

• Area: Hugging Face model file upload.
• Reproduction: Upload models/objectverse-diary-qwen15b-lora-v2-q4_k_m.gguf with the default Hub client path.
• Expected: upload completes and commits objectverse-diary-qwen15b-lora-v2-q4_k_m.gguf.
• Actual: the first upload stalled with Xet TLS EOF / CLOSE_WAIT after partial progress.
• Impact: upload needed a retry; local GGUF file was unaffected.
• Fallback used: stopped the stalled upload process and retried with HF_HUB_DISABLE_XET=1.
• Resolution: resolved; ordinary Hub/LFS upload succeeded.
• Evidence: Hub file https://huggingface.co/qqyule/objectverse-diary-qwen15b-lora/blob/main/objectverse-diary-qwen15b-lora-v2-q4_k_m.gguf.

Anticipated Failure Areas

Vision Runtime

• MiniCPM-V or fallback VLM fails to load in local or Space environment.
• Uploaded image is unsupported, too large, or not an object photo.
• Model output is not valid JSON.
• Object recognition is too vague for persona generation.

Fallback:

• use manual object description
• use stable example flow
• record fallback marker in trace
• vision-fallback-to-mock means MiniCPM-V failed or returned invalid JSON and mock object understanding was used.

Text Runtime

• GGUF model path is missing.
• llama.cpp or llama-cpp-python import fails.
• Generation output does not match PersonaEnvelope or DiaryEntry schema.
• Chat response loses persona consistency.

Fallback:

• use deterministic mock text runtime
• repair JSON before schema validation
• record fallback marker in trace

Dataset And Fine-Tuning

• Candidate samples contain private information.
• JSONL rows fail to parse.
• Curated data is too repetitive.
• Training run fails or adapter quality is weaker than mock prompts.

Fallback:

• keep mock preview data for schema validation only
• separate raw candidates from curated rows
• publish only reviewed examples

Hugging Face Space

• Space cannot start because of missing packages.
• Model files are too large or not available.
• CPU runtime is too slow.
• Secrets or private paths appear in logs.

Fallback:

• launch with mock runtime first
• keep model files out of the repo
• document runtime mode clearly in README