File size: 9,600 Bytes
d63b1e4
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
# BARG-CC — document-only TNNLS manuscript and experimental package

**Allowed datasets: RVL-CDIP and FUNSD only.**

The two tasks are **not interchangeable**:

* **RVL-CDIP:** page-level document category clustering, K=16, evaluated with ARI/NMI using official page-category labels.
* **FUNSD:** *word-level* semantic role clustering, K=4 (header/question/answer/other), evaluated on OCR-detected words matched to ground-truth annotated word boxes. Parent form IDs define data splits and uncertainty units. FUNSD **does not** supply 16-class document-category labels.

The source here contains a complete 12-page IEEEtran two-column draft, document-specific data parsers, and a strict train/validation/test benchmark runner. It **does not** include either dataset or genuine benchmark accuracy. The raw images/annotations were not available locally. The software tests use generated fixtures only. The draft is **not submission-ready** until the actual data are processed and the empirical tables are populated from verified runs.

## Files

- `main.tex`: editable LaTeX manuscript (IEEEtran, 10pt journal).
- `main.pdf`: compiled 12-page manuscript.
- `prepare_documents.py`: two-corpus image/OCR/geometry extraction and split integrity.
- `run_document_benchmark.py`: frozen-protocol neural-denoising, masking, PCA, shuffled-score and prior controls, and ARI/NMI readout.
- `paired_group_bootstrap.py`: paired ARI/NMI uncertainty by independent page/form group.
- `test_document_protocol.py`: synthetic software, group-bootstrap, and theorem checks.
- `DATASET_STATUS.json`: honest dataset/execution status.
- `ENVIRONMENT.json`: local software versions.
- `SHA256SUMS.txt`: integrity hashes.
- `requirements.txt`: Python dependencies.

## Source-identity preflight (required before a journal benchmark)

Run `preflight_corpora.py` before feature extraction. With default strict counts it
refuses a partial dataset while recording available SHA-256 metadata; an explicit
`--allow-subset` flag labels any pilot as a subset. **Counts and local hashes alone
do not independently authenticate provenance or grant redistribution rights.**

```
python preflight_corpora.py rvlcdip --root /path/to/RVL-CDIP \\
  --output attestations/rvlcdip_preflight.json
python preflight_corpora.py funsd --root /path/to/FUNSD \\
  --output attestations/funsd_preflight.json
```

RVL-CDIP checks the 320,000/40,000/40,000 official list counts and verifies
the referenced paths exist, sample image hashes, category ranges and split identity.
FUNSD checks its 149/50 form counts, image and annotation integrity, word-box
schemas and all local media hashes. Neither preflight uses the labels for training.

## Dataset 1: RVL-CDIP

Supply an authorized RVL-CDIP unpacked root that contains:

```
rvl_root/
  labels/
    train.txt
    val.txt
    test.txt
  images/
    ... images referenced by the three label lists ...
```

Each label-list line must be `<relative_image_path> <category_integer_0_to_15>`.

The script defaults to a *clearly labeled resource-limited pilot*: 6000 training, 1000 validation and 1000 test pages, selected by a stable file-ID hash **without inspecting labels**. Set all three `--max-*` counts to `0` for complete official splits; OCR on the full corpus can require considerable storage and computation.

```
python prepare_documents.py rvlcdip \
  --root /path/to/RVL-CDIP --out data/rvlcdip.npz \
  --max-train 6000 --max-validation 1000 --max-test 1000
```

Do not describe a hash-subset run as a full RVL-CDIP benchmark. Exact-byte duplicates across partitions cause a fail-closed error.

## Dataset 2: FUNSD

Supply the canonical image and annotation hierarchy:

```
funsd_root/
  training_data/
    images/*.png
    annotations/*.json
  testing_data/
    images/*.png
    annotations/*.json
```

Word-level roles come from the `form[].label` fields **only for final scoring**; OCR text, detected boxes and word crops supply features. Every OCR detection is retained as an unlabeled clustering record. A one-to-one spatial match against annotated boxes attaches a role solely for evaluation; unmatched detections receive label -1 (unscored, **not** the official "other" category). Both matched and unmatched predictions are exported. The benchmark's `n_test_scored`/`n_test` and `test_detection_match_fraction` describe OCR-detection match coverage; the preparation audit separately reports annotation recall and per-form coverage (including forms with zero matches). Different words from the same page never enter different splits. FUNSD's official 50 test forms remain untouched.

```
python prepare_documents.py funsd \
  --root /path/to/FUNSD --out data/funsd.npz --min-iou 0.30
```

The script fails if OCR yields no scoreable words in an essential split. Please inspect the coverage audit in the adjacent `funsd.json` before interpreting word-role clustering.

## Frozen evaluation (repeat for EACH dataset)

```
python run_document_benchmark.py freeze \
  --npz data/funsd.npz --lock protocols/funsd_lock.json
```

Review the protocol **before** any test-score release. It locks the source-file SHA-256 and train/validation/test sample identity. Scoring is explicitly gated:

```
python run_document_benchmark.py execute \
  --npz data/funsd.npz --lock protocols/funsd_lock.json \
  --output results/funsd --release-test
```

Run equivalent commands with the RVL-CDIP NPZ and a separate lock.

The seven local methods are masked concatenation, train-only PCA16, denoised uniform gating, denoised predicted gating, denoised shuffled scoring, denoised fixed-view prior and identity-anchored fusion. All methods use train-fitted K-means and held-out assignments. The runner also saves `predictions.csv.gz`, allowing independent recomputation of per-group ARI/NMI uncertainty. The model is a **train-only MLP denoising reference**; it is **not an executed reproduction of the full jointly contrastive BARG-CC objective**.

### Limitations you must not misstate

1. The implemented stress tests add **feature-space** dropout and noise, not real image blur, crop, bleed-through or OCR reruns.
2. FUNSD word-level conditional ARI is calculated only for role-matched OCR words, although *all* OCR detections are clustered. It does not establish document-category performance or measure missed annotation words; always publish both detection-match fraction and annotation recall, including zero-match forms.
3. The supplied package has **no executed official-code recent neural comparators**. Never insert external benchmark numbers without source/checkpoint/prediction provenance.
4. Freezing a protocol file does not, by itself, demonstrate prospective preregistration.
5. Repeated neural seeds on the same page/form test set are not independent dataset replicates.
6. No dataset labels enter feature preparation or training; labels are used for K according to the published ontology and final scoring.

## Reproducibility checks

```
python -m unittest -v test_document_protocol.py
latexmk -pdf -interaction=nonstopmode -halt-on-error main.tex
pdfinfo main.pdf | grep Pages
```

The LaTeX build is self-contained (TikZ included); it does not depend on cached figures from earlier versions.

**Raw data are not redistributed.** Dataset download, rights, authenticity and journal requirements must be verified by the submitting authors.

## Published-method comparison identity contract

The script below exports the exact same held-out sample/seed/condition keys
from the local masked baseline, **without labels or model predictions**:

```
python export_official_reference.py \\
  --predictions results/funsd/predictions.csv.gz \\
  --reference-out results/funsd/official_reference_ids.csv \\
  --contract-out results/funsd/official_reference_contract.json
```

This is an **unexecuted comparator interface**, not an official-code result.
Once a licensed official implementation has actually been run with the exact
features, masks, train/test identities and training-label restrictions, verify
its provenance and keyed predictions with:

```
python ../archived_evidence/reproducibility/verify_official_comparator.py \\
  --manifest /path/to/verified_official_run.json \\
  --results /path/to/official_predictions.csv \\
  --reference results/funsd/official_reference_ids.csv
```

A provenance contract or GitHub URL alone does NOT constitute an executed
published baseline. No comparator result has been added to this package.

## Paired, form-grouped uncertainty (after a genuine FUNSD run)

```
python paired_group_bootstrap.py \
  --predictions results/funsd/predictions.csv.gz \
  --dataset funsd --condition M20_D50 --seed 11 \
  --method-a denoised_gated --method-b masked \
  --replicates 500 --output results/funsd/group_uncertainty_seed11.json
```

This resamples complete *forms*, not individual OCR words. The same command accepts
`--dataset rvlcdip` for page-level groups or deduplicated source-document IDs.
It is a conditional inference check for that fixed test split and does not
make repeated training seeds independent datasets.

## New, distinct joint-contrastive reference implementation
`run_joint_contrastive_reference.py` adds jointly trained view encoders,
reliability-conditioned residual maps, learned alignment projections and all
three loss terms. Its source-locked predictions are compatible with local
baseline scoring, but it is **not** an original checkpoint replay or a result
on canonical RVL-CDIP/FUNSD. Use `compare_joint_reference.py` for source-matched,
group-level descriptive differences. Full instructions:
`JOINT_REFERENCE_AND_DATA_GUIDE.md`.