DATA_AVAILABILITY.md

Data & Code Availability

This repository (Chucks90/covtoken on the Hugging Face Hub) holds the code and experiment artifacts for the covtoken study (label-free mid-layer lesion subspaces for token-economical medical imaging). All compute was run as Hugging Face Jobs; every reported number is reproducible from the scripts here against the public backbones and datasets listed below.

What is in this repository

• Code — jobs/ (PEP-723 uv job scripts, one per experiment), subspace/, coverage/, gate/, arch/, eval/, data/, backbone/, and tests/ (incl. the label-leak guard).
• Decision records — gate_reports/ (per-gate JSON with metric, comparator, threshold, and statistical test; NEGATIVE_RESULT.md; SUMMARY.md).
• Research-program results — research_v2/ (S1–S5), research_v3/ (F1–F4), research_v4/ (G1/spectra/rarity-route) as JSON + summaries.
• Manuscripts & figures — paper/ (three drafts, make_figures.py, figures/).
• Specs — research_specs/, configs/thresholds.lock.json.

What is NOT in this repository (and why)

Raw token banks, model weights, and materialized image/mask tensors are not included: they are large, and the imaging data are governed by their original third-party licenses. They are regenerated deterministically by the scripts in jobs/ from the public sources below. Reported metrics depend only on those public sources + the scripts here.

Backbones (public, frozen — no fine-tuning)

• MedDINOv3 ViT-B/16 (CT-3M) — ricklisz123/MedDINOv3-ViTB-16-CT-3M (Hugging Face)
• DINOv2-base — facebook/dinov2-base
• Cross-objective controls: google/vit-base-patch16-224 (supervised), facebook/vit-mae-base (MAE)

Imaging datasets (public, third-party — used eval-only; labels never touch subspace construction)

• LIDC-IDRI (lung CT) — The Cancer Imaging Archive: https://www.cancerimagingarchive.net/collection/lidc-idri/
• KiTS23 (kidney CT) — https://kits-challenge.org/kits23/
• Medical Segmentation Decathlon — Task03 Liver, Task07 Pancreas (CT) — http://medicaldecathlon.com/
• BUSI (breast ultrasound) — Al-Dhabyani et al., Data in Brief 2020 (Dataset of breast ultrasound images)

Each dataset retains its original license/terms; obtain it from the source above.

Reproducing a result

Every experiment is a self-contained job script. Example:

hf jobs uv run --flavor t4-medium --timeout 2h --secrets HF_TOKEN \
  -v hf://buckets/<your-bucket>:/mnt --detach jobs/<experiment>_job.py

Each script declares its inline dependencies (PEP 723), reads inputs from the mounted bucket, writes a result JSON, and prints a *_RESULT line. The mapping from claims to scripts/artifacts is in each gate_reports/*.json and the research_v*/SUMMARY.md files.

Citing this repository

A DOI for the archival snapshot is available via the repository's Settings → Generate DOI on the Hugging Face Hub; cite that DOI in the manuscript's Data Availability Statement.