File size: 9,600 Bytes
d63b1e4 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 | # BARG-CC — document-only TNNLS manuscript and experimental package
**Allowed datasets: RVL-CDIP and FUNSD only.**
The two tasks are **not interchangeable**:
* **RVL-CDIP:** page-level document category clustering, K=16, evaluated with ARI/NMI using official page-category labels.
* **FUNSD:** *word-level* semantic role clustering, K=4 (header/question/answer/other), evaluated on OCR-detected words matched to ground-truth annotated word boxes. Parent form IDs define data splits and uncertainty units. FUNSD **does not** supply 16-class document-category labels.
The source here contains a complete 12-page IEEEtran two-column draft, document-specific data parsers, and a strict train/validation/test benchmark runner. It **does not** include either dataset or genuine benchmark accuracy. The raw images/annotations were not available locally. The software tests use generated fixtures only. The draft is **not submission-ready** until the actual data are processed and the empirical tables are populated from verified runs.
## Files
- `main.tex`: editable LaTeX manuscript (IEEEtran, 10pt journal).
- `main.pdf`: compiled 12-page manuscript.
- `prepare_documents.py`: two-corpus image/OCR/geometry extraction and split integrity.
- `run_document_benchmark.py`: frozen-protocol neural-denoising, masking, PCA, shuffled-score and prior controls, and ARI/NMI readout.
- `paired_group_bootstrap.py`: paired ARI/NMI uncertainty by independent page/form group.
- `test_document_protocol.py`: synthetic software, group-bootstrap, and theorem checks.
- `DATASET_STATUS.json`: honest dataset/execution status.
- `ENVIRONMENT.json`: local software versions.
- `SHA256SUMS.txt`: integrity hashes.
- `requirements.txt`: Python dependencies.
## Source-identity preflight (required before a journal benchmark)
Run `preflight_corpora.py` before feature extraction. With default strict counts it
refuses a partial dataset while recording available SHA-256 metadata; an explicit
`--allow-subset` flag labels any pilot as a subset. **Counts and local hashes alone
do not independently authenticate provenance or grant redistribution rights.**
```
python preflight_corpora.py rvlcdip --root /path/to/RVL-CDIP \\
--output attestations/rvlcdip_preflight.json
python preflight_corpora.py funsd --root /path/to/FUNSD \\
--output attestations/funsd_preflight.json
```
RVL-CDIP checks the 320,000/40,000/40,000 official list counts and verifies
the referenced paths exist, sample image hashes, category ranges and split identity.
FUNSD checks its 149/50 form counts, image and annotation integrity, word-box
schemas and all local media hashes. Neither preflight uses the labels for training.
## Dataset 1: RVL-CDIP
Supply an authorized RVL-CDIP unpacked root that contains:
```
rvl_root/
labels/
train.txt
val.txt
test.txt
images/
... images referenced by the three label lists ...
```
Each label-list line must be `<relative_image_path> <category_integer_0_to_15>`.
The script defaults to a *clearly labeled resource-limited pilot*: 6000 training, 1000 validation and 1000 test pages, selected by a stable file-ID hash **without inspecting labels**. Set all three `--max-*` counts to `0` for complete official splits; OCR on the full corpus can require considerable storage and computation.
```
python prepare_documents.py rvlcdip \
--root /path/to/RVL-CDIP --out data/rvlcdip.npz \
--max-train 6000 --max-validation 1000 --max-test 1000
```
Do not describe a hash-subset run as a full RVL-CDIP benchmark. Exact-byte duplicates across partitions cause a fail-closed error.
## Dataset 2: FUNSD
Supply the canonical image and annotation hierarchy:
```
funsd_root/
training_data/
images/*.png
annotations/*.json
testing_data/
images/*.png
annotations/*.json
```
Word-level roles come from the `form[].label` fields **only for final scoring**; OCR text, detected boxes and word crops supply features. Every OCR detection is retained as an unlabeled clustering record. A one-to-one spatial match against annotated boxes attaches a role solely for evaluation; unmatched detections receive label -1 (unscored, **not** the official "other" category). Both matched and unmatched predictions are exported. The benchmark's `n_test_scored`/`n_test` and `test_detection_match_fraction` describe OCR-detection match coverage; the preparation audit separately reports annotation recall and per-form coverage (including forms with zero matches). Different words from the same page never enter different splits. FUNSD's official 50 test forms remain untouched.
```
python prepare_documents.py funsd \
--root /path/to/FUNSD --out data/funsd.npz --min-iou 0.30
```
The script fails if OCR yields no scoreable words in an essential split. Please inspect the coverage audit in the adjacent `funsd.json` before interpreting word-role clustering.
## Frozen evaluation (repeat for EACH dataset)
```
python run_document_benchmark.py freeze \
--npz data/funsd.npz --lock protocols/funsd_lock.json
```
Review the protocol **before** any test-score release. It locks the source-file SHA-256 and train/validation/test sample identity. Scoring is explicitly gated:
```
python run_document_benchmark.py execute \
--npz data/funsd.npz --lock protocols/funsd_lock.json \
--output results/funsd --release-test
```
Run equivalent commands with the RVL-CDIP NPZ and a separate lock.
The seven local methods are masked concatenation, train-only PCA16, denoised uniform gating, denoised predicted gating, denoised shuffled scoring, denoised fixed-view prior and identity-anchored fusion. All methods use train-fitted K-means and held-out assignments. The runner also saves `predictions.csv.gz`, allowing independent recomputation of per-group ARI/NMI uncertainty. The model is a **train-only MLP denoising reference**; it is **not an executed reproduction of the full jointly contrastive BARG-CC objective**.
### Limitations you must not misstate
1. The implemented stress tests add **feature-space** dropout and noise, not real image blur, crop, bleed-through or OCR reruns.
2. FUNSD word-level conditional ARI is calculated only for role-matched OCR words, although *all* OCR detections are clustered. It does not establish document-category performance or measure missed annotation words; always publish both detection-match fraction and annotation recall, including zero-match forms.
3. The supplied package has **no executed official-code recent neural comparators**. Never insert external benchmark numbers without source/checkpoint/prediction provenance.
4. Freezing a protocol file does not, by itself, demonstrate prospective preregistration.
5. Repeated neural seeds on the same page/form test set are not independent dataset replicates.
6. No dataset labels enter feature preparation or training; labels are used for K according to the published ontology and final scoring.
## Reproducibility checks
```
python -m unittest -v test_document_protocol.py
latexmk -pdf -interaction=nonstopmode -halt-on-error main.tex
pdfinfo main.pdf | grep Pages
```
The LaTeX build is self-contained (TikZ included); it does not depend on cached figures from earlier versions.
**Raw data are not redistributed.** Dataset download, rights, authenticity and journal requirements must be verified by the submitting authors.
## Published-method comparison identity contract
The script below exports the exact same held-out sample/seed/condition keys
from the local masked baseline, **without labels or model predictions**:
```
python export_official_reference.py \\
--predictions results/funsd/predictions.csv.gz \\
--reference-out results/funsd/official_reference_ids.csv \\
--contract-out results/funsd/official_reference_contract.json
```
This is an **unexecuted comparator interface**, not an official-code result.
Once a licensed official implementation has actually been run with the exact
features, masks, train/test identities and training-label restrictions, verify
its provenance and keyed predictions with:
```
python ../archived_evidence/reproducibility/verify_official_comparator.py \\
--manifest /path/to/verified_official_run.json \\
--results /path/to/official_predictions.csv \\
--reference results/funsd/official_reference_ids.csv
```
A provenance contract or GitHub URL alone does NOT constitute an executed
published baseline. No comparator result has been added to this package.
## Paired, form-grouped uncertainty (after a genuine FUNSD run)
```
python paired_group_bootstrap.py \
--predictions results/funsd/predictions.csv.gz \
--dataset funsd --condition M20_D50 --seed 11 \
--method-a denoised_gated --method-b masked \
--replicates 500 --output results/funsd/group_uncertainty_seed11.json
```
This resamples complete *forms*, not individual OCR words. The same command accepts
`--dataset rvlcdip` for page-level groups or deduplicated source-document IDs.
It is a conditional inference check for that fixed test split and does not
make repeated training seeds independent datasets.
## New, distinct joint-contrastive reference implementation
`run_joint_contrastive_reference.py` adds jointly trained view encoders,
reliability-conditioned residual maps, learned alignment projections and all
three loss terms. Its source-locked predictions are compatible with local
baseline scoring, but it is **not** an original checkpoint replay or a result
on canonical RVL-CDIP/FUNSD. Use `compare_joint_reference.py` for source-matched,
group-level descriptive differences. Full instructions:
`JOINT_REFERENCE_AND_DATA_GUIDE.md`.
|