--- license: cc-by-4.0 base_model: laion/CLIP-ViT-B-32-laion2B-s34B-b79K pipeline_tag: zero-shot-classification library_name: pointcal-c tags: - 3d - point-cloud - modelnet40-c - clip - calibration - selective-prediction - uncertainty - zero-shot --- # PointCal-C — selective zero-shot 3D recognition under corruption A post-hoc **calibration and abstention head** on top of a frozen OpenCLIP ViT-B/32, for zero-shot 3D point-cloud recognition under corruption, plus the complete evaluation artifacts backing it. Code: ## What the model is The backbone is **never trained**. `laion/CLIP-ViT-B-32-laion2B-s34B-b79K` is loaded frozen, in eval mode, with `requires_grad_(False)`. The fitted model is **four scalars**, in `runs/full/results/calibration.json`, fit on clean calibration objects only — corrupted labels, corruption identity and severity never touch a fitted parameter: | parameter | value | role | |---|---|---| | `temperature` | 1.2309 | divides logits before softmax | | `bias` | -1.5723 | combined-score intercept | | `weight_confidence` | +1.7071 | on `logit(p_calibrated)` | | `weight_disagreement` | +4.9158 | on cross-view disagreement `d` | Combined score: `sigmoid(bias + w_conf * logit(p_cal) + w_dis * d)`, where `d` is `1 -` mean pairwise Jensen-Shannon divergence across the six views. Converged, unclamped, not degenerate. Fit in 0.94 s on CPU. ## How inference works ``` point cloud (N,1024,3) -> centroid-center, unit-radius normalize -> 6 orthographic cameras [front right back left top bottom] -> 64x64 splat raster + nearest-depth z-buffer -> upsample to 224x224 -> frozen OpenCLIP ViT-B/32 image encoder -> per-view logits vs a fixed 8-template prompt ensemble <-- cached here -> mean over views -> the prediction -> four confidence scores over that one prediction ``` All four confidence methods (`msp`, `temperature`, `disagreement`, `combined`) rank the **same** predictions. A confidence method that changed a prediction would be a different classifier, and `assert_predictions_unchanged` raises if one ever does. ## Results All 75 corrupted conditions pooled against clean, 1975 evaluation objects, 95% bootstrap intervals resampled over base object IDs: | | accuracy | ECE (MSP) | ECE (combined) | AURC (MSP) | AURC (combined) | |---|---|---|---|---|---| | clean | 0.2896 | 0.1160 | 0.0245 | 0.4339 | 0.4147 | | corrupted | 0.2354 | 0.1416 | 0.0229 | 0.5217 | 0.5008 | Accuracy degrades under corruption and MSP calibration degrades with it. The clean-fit combined score holds ECE near **0.023** under shift and lowers AURC by 0.021. The ECE improvement is large and consistent. **The AURC gain is small** and should be read against the intervals in `results.csv`, not as a headline. This is a reliability audit and a bounded abstention baseline, not a state-of-the-art claim. Accuracy by severity: 0.2896 (clean), 0.2593, 0.2515, 0.2388, 0.2252, 0.2023 — monotonically non-increasing. ### Pre-registered hypotheses Of the four pre-registered hypotheses, three are mechanically checked. H2 and H3 resolve True. **H1 does not hold strictly:** accuracy, NLL, Brier and AURC all worsen monotonically with severity, but ECE dips at severities 3→4 and again from clean→1. A paired object-grouped bootstrap places both reversals within noise of zero, so H1 is supported in direction but not as strict monotonicity. H4 is read off the family-scope rows rather than reduced to a verdict. ## Repository contents The expensive artifact is `runs/*/logits/` — per-view logits for every condition under both prompt modes. Every metric, ablation, bootstrap interval and figure here was computed on CPU from those caches, so **any re-analysis is free**: a different aggregation, disagreement statistic, coverage level or bootstrap depth costs no GPU time, because CLIP never runs again. ``` runs// logits/.npz per-view logits, both prompt modes (~1.4 MB each) results/calibration.json the fitted scalars above results/results.csv|.json metrics table with intervals results/ablations.csv view-count, prompt-mode, disagreement ablations results/predictions.npz per-sample confidences and correctness results/results_summary.md auto-generated tables + hypothesis checks figures/fig1..fig4 accuracy, ECE, risk-coverage, cost provenance/ data manifest, run manifest, split audit ledger_inference.json measured GPU-hours, dollars, throughput, memory artifacts/split.json the frozen calibration/evaluation object split env/requirements.lock.txt exact resolved environment (171 packages) ``` Tiers: `xs` (5 conditions, 100 objects), `s` (13 conditions), `full` (all 76 = clean + 15 corruptions x 5 severities). ## Training / evaluation data Evaluated on **ModelNet40-C** (2468 objects, 40 classes, 15 corruption types, 5 severities). The calibration scalars are fit on the clean split of 493 calibration objects; 1975 evaluation objects are held out. The split is over **base object IDs**, not samples, and every corruption array is verified row-aligned with the clean array, so holding an object out of calibration holds it out under all 76 conditions. Audited: `overlap: 0`, 0 row-misaligned conditions, all 40 classes present on both sides. ## Compute | tier | conditions | GPU-hours | USD | views/s | peak VRAM | |---|---|---|---|---|---| | xs | 5 | 0.001 | 0.0003 | 1091 | 2.5 GB | | s | 13 | 0.015 | 0.004 | 2816 | 12.1 GB | | full | 76 | 0.089 | 0.025 | 2808 | 12.1 GB | **0.105 GPU-hours and about $0.03** of inference in total, on a single Runpod RTX 4000 Ada at $0.28/hr (2026-08-16). ## Known defects See **[ERRATUM.md](ERRATUM.md)** for the full list. In short: the `ledger` block embedded in `runs/full/provenance/run_manifest.json` is wrong — it records a cache-hit re-invocation, not the real run. **Trust `ledger_inference.json`,** which every figure and the compute table above are derived from. `git_dirty` in that manifest is not meaningful either, because the dirtiness check counts expected untracked artifacts. ## Limitations - `full` uses **200** bootstrap replicates, not the preregistered 1000. The CPU bootstrap dominated runtime (~2 h at 1000 vs ~5 min of GPU work). Deeper intervals recompute from the cached logits at no GPU cost. - NLL and Brier require a distribution over all 40 classes, so they are reported for `msp` and `temperature` only. For the two ranking scores they are `null`, not a lookalike substitute. - `delta_*_vs_clean` rows in `results.csv` carry `NaN` intervals: degradation is reported as point estimates without paired bootstrap intervals. Method differences should likewise not be read from the marginal intervals alone. - Cross-view disagreement is prior art, and calibration degrading under 3D corruption is a confirmation rather than a discovery. - Absolute accuracy is low (~0.29 clean). This is zero-shot transfer of a 2D model to depth renders, with no 3D training whatsoever. - Severity is reported **1-indexed**, matching the on-disk release. ## Provenance - Frozen spec hash: `ca487597c978ab17217bd640ab8bf95c8a21db0eaacd51ae5ba4b00618dcc820` - Backbone checkpoint SHA-256: `1bd3c7172de5b207ceac554f5ab5266166f3b9baccc9af5989bc801016d080ad` - Split fingerprint: `cc5ed4b3ce7108154039f497d474f3ac6bdbfe7eb7dbfe4d0f90785e1829bb31` ## Licence and attribution Cached logits and metrics are **derived from ModelNet40-C** and redistributed under **CC BY 4.0**, as the source record declares. - ModelNet40-C — Zenodo [6017834](https://zenodo.org/records/6017834), DOI `10.5281/zenodo.6017834`, CC BY 4.0 - Generation code — , BSD-3-Clause - Underlying ModelNet40 carries its own original terms - OpenCLIP — MIT · PointCal-C code — MIT The multi-view depth-projection approach follows the PointCLIP paper (CVPR 2022); the projector is an independent reimplementation from the paper text, with no PointCLIP repository code used.