# Data & Code Availability

This repository (`Chucks90/covtoken` on the Hugging Face Hub) holds the **code and experiment
artifacts** for the covtoken study (label-free mid-layer lesion subspaces for token-economical
medical imaging). All compute was run as Hugging Face Jobs; every reported number is reproducible
from the scripts here against the public backbones and datasets listed below.

## What is in this repository
- **Code** — `jobs/` (PEP-723 `uv` job scripts, one per experiment), `subspace/`, `coverage/`,
  `gate/`, `arch/`, `eval/`, `data/`, `backbone/`, and `tests/` (incl. the label-leak guard).
- **Decision records** — `gate_reports/` (per-gate JSON with metric, comparator, threshold, and
  statistical test; `NEGATIVE_RESULT.md`; `SUMMARY.md`).
- **Research-program results** — `research_v2/` (S1–S5), `research_v3/` (F1–F4), `research_v4/`
  (G1/spectra/rarity-route) as JSON + summaries.
- **Manuscripts & figures** — `paper/` (three drafts, `make_figures.py`, `figures/`).
- **Specs** — `research_specs/`, `configs/thresholds.lock.json`.

## What is NOT in this repository (and why)
Raw token banks, model weights, and materialized image/mask tensors are **not** included: they are
large, and the imaging data are governed by their original third-party licenses. They are
regenerated deterministically by the scripts in `jobs/` from the public sources below. Reported
metrics depend only on those public sources + the scripts here.

## Backbones (public, frozen — no fine-tuning)
- **MedDINOv3 ViT-B/16 (CT-3M)** — `ricklisz123/MedDINOv3-ViTB-16-CT-3M` (Hugging Face)
- **DINOv2-base** — `facebook/dinov2-base`
- Cross-objective controls: `google/vit-base-patch16-224` (supervised), `facebook/vit-mae-base` (MAE)

## Imaging datasets (public, third-party — used eval-only; labels never touch subspace construction)
- **LIDC-IDRI** (lung CT) — The Cancer Imaging Archive: https://www.cancerimagingarchive.net/collection/lidc-idri/
- **KiTS23** (kidney CT) — https://kits-challenge.org/kits23/
- **Medical Segmentation Decathlon** — Task03 Liver, Task07 Pancreas (CT) — http://medicaldecathlon.com/
- **BUSI** (breast ultrasound) — Al-Dhabyani et al., *Data in Brief* 2020 (Dataset of breast ultrasound images)

Each dataset retains its original license/terms; obtain it from the source above.

## Reproducing a result
Every experiment is a self-contained job script. Example:

```bash
hf jobs uv run --flavor t4-medium --timeout 2h --secrets HF_TOKEN \
  -v hf://buckets/<your-bucket>:/mnt --detach jobs/<experiment>_job.py
```

Each script declares its inline dependencies (PEP 723), reads inputs from the mounted bucket,
writes a result JSON, and prints a `*_RESULT` line. The mapping from claims to scripts/artifacts is
in each `gate_reports/*.json` and the `research_v*/SUMMARY.md` files.

## Citing this repository
A DOI for the archival snapshot is available via the repository's **Settings → Generate DOI** on the
Hugging Face Hub; cite that DOI in the manuscript's Data Availability Statement.