Debmalya's picture
Upload 18 files
d63b1e4 verified
|
Raw History Blame Contribute Delete
9.6 kB

BARG-CC — document-only TNNLS manuscript and experimental package

Allowed datasets: RVL-CDIP and FUNSD only.

The two tasks are not interchangeable:

  • RVL-CDIP: page-level document category clustering, K=16, evaluated with ARI/NMI using official page-category labels.
  • FUNSD: word-level semantic role clustering, K=4 (header/question/answer/other), evaluated on OCR-detected words matched to ground-truth annotated word boxes. Parent form IDs define data splits and uncertainty units. FUNSD does not supply 16-class document-category labels.

The source here contains a complete 12-page IEEEtran two-column draft, document-specific data parsers, and a strict train/validation/test benchmark runner. It does not include either dataset or genuine benchmark accuracy. The raw images/annotations were not available locally. The software tests use generated fixtures only. The draft is not submission-ready until the actual data are processed and the empirical tables are populated from verified runs.

Files

  • main.tex: editable LaTeX manuscript (IEEEtran, 10pt journal).
  • main.pdf: compiled 12-page manuscript.
  • prepare_documents.py: two-corpus image/OCR/geometry extraction and split integrity.
  • run_document_benchmark.py: frozen-protocol neural-denoising, masking, PCA, shuffled-score and prior controls, and ARI/NMI readout.
  • paired_group_bootstrap.py: paired ARI/NMI uncertainty by independent page/form group.
  • test_document_protocol.py: synthetic software, group-bootstrap, and theorem checks.
  • DATASET_STATUS.json: honest dataset/execution status.
  • ENVIRONMENT.json: local software versions.
  • SHA256SUMS.txt: integrity hashes.
  • requirements.txt: Python dependencies.

Source-identity preflight (required before a journal benchmark)

Run preflight_corpora.py before feature extraction. With default strict counts it refuses a partial dataset while recording available SHA-256 metadata; an explicit --allow-subset flag labels any pilot as a subset. Counts and local hashes alone do not independently authenticate provenance or grant redistribution rights.

python preflight_corpora.py rvlcdip --root /path/to/RVL-CDIP \\
  --output attestations/rvlcdip_preflight.json
python preflight_corpora.py funsd --root /path/to/FUNSD \\
  --output attestations/funsd_preflight.json

RVL-CDIP checks the 320,000/40,000/40,000 official list counts and verifies the referenced paths exist, sample image hashes, category ranges and split identity. FUNSD checks its 149/50 form counts, image and annotation integrity, word-box schemas and all local media hashes. Neither preflight uses the labels for training.

Dataset 1: RVL-CDIP

Supply an authorized RVL-CDIP unpacked root that contains:

rvl_root/
  labels/
    train.txt
    val.txt
    test.txt
  images/
    ... images referenced by the three label lists ...

Each label-list line must be <relative_image_path> <category_integer_0_to_15>.

The script defaults to a clearly labeled resource-limited pilot: 6000 training, 1000 validation and 1000 test pages, selected by a stable file-ID hash without inspecting labels. Set all three --max-* counts to 0 for complete official splits; OCR on the full corpus can require considerable storage and computation.

python prepare_documents.py rvlcdip \
  --root /path/to/RVL-CDIP --out data/rvlcdip.npz \
  --max-train 6000 --max-validation 1000 --max-test 1000

Do not describe a hash-subset run as a full RVL-CDIP benchmark. Exact-byte duplicates across partitions cause a fail-closed error.

Dataset 2: FUNSD

Supply the canonical image and annotation hierarchy:

funsd_root/
  training_data/
    images/*.png
    annotations/*.json
  testing_data/
    images/*.png
    annotations/*.json

Word-level roles come from the form[].label fields only for final scoring; OCR text, detected boxes and word crops supply features. Every OCR detection is retained as an unlabeled clustering record. A one-to-one spatial match against annotated boxes attaches a role solely for evaluation; unmatched detections receive label -1 (unscored, not the official "other" category). Both matched and unmatched predictions are exported. The benchmark's n_test_scored/n_test and test_detection_match_fraction describe OCR-detection match coverage; the preparation audit separately reports annotation recall and per-form coverage (including forms with zero matches). Different words from the same page never enter different splits. FUNSD's official 50 test forms remain untouched.

python prepare_documents.py funsd \
  --root /path/to/FUNSD --out data/funsd.npz --min-iou 0.30

The script fails if OCR yields no scoreable words in an essential split. Please inspect the coverage audit in the adjacent funsd.json before interpreting word-role clustering.

Frozen evaluation (repeat for EACH dataset)

python run_document_benchmark.py freeze \
  --npz data/funsd.npz --lock protocols/funsd_lock.json

Review the protocol before any test-score release. It locks the source-file SHA-256 and train/validation/test sample identity. Scoring is explicitly gated:

python run_document_benchmark.py execute \
  --npz data/funsd.npz --lock protocols/funsd_lock.json \
  --output results/funsd --release-test

Run equivalent commands with the RVL-CDIP NPZ and a separate lock.

The seven local methods are masked concatenation, train-only PCA16, denoised uniform gating, denoised predicted gating, denoised shuffled scoring, denoised fixed-view prior and identity-anchored fusion. All methods use train-fitted K-means and held-out assignments. The runner also saves predictions.csv.gz, allowing independent recomputation of per-group ARI/NMI uncertainty. The model is a train-only MLP denoising reference; it is not an executed reproduction of the full jointly contrastive BARG-CC objective.

Limitations you must not misstate

  1. The implemented stress tests add feature-space dropout and noise, not real image blur, crop, bleed-through or OCR reruns.
  2. FUNSD word-level conditional ARI is calculated only for role-matched OCR words, although all OCR detections are clustered. It does not establish document-category performance or measure missed annotation words; always publish both detection-match fraction and annotation recall, including zero-match forms.
  3. The supplied package has no executed official-code recent neural comparators. Never insert external benchmark numbers without source/checkpoint/prediction provenance.
  4. Freezing a protocol file does not, by itself, demonstrate prospective preregistration.
  5. Repeated neural seeds on the same page/form test set are not independent dataset replicates.
  6. No dataset labels enter feature preparation or training; labels are used for K according to the published ontology and final scoring.

Reproducibility checks

python -m unittest -v test_document_protocol.py
latexmk -pdf -interaction=nonstopmode -halt-on-error main.tex
pdfinfo main.pdf | grep Pages

The LaTeX build is self-contained (TikZ included); it does not depend on cached figures from earlier versions.

Raw data are not redistributed. Dataset download, rights, authenticity and journal requirements must be verified by the submitting authors.

Published-method comparison identity contract

The script below exports the exact same held-out sample/seed/condition keys from the local masked baseline, without labels or model predictions:

python export_official_reference.py \\
  --predictions results/funsd/predictions.csv.gz \\
  --reference-out results/funsd/official_reference_ids.csv \\
  --contract-out results/funsd/official_reference_contract.json

This is an unexecuted comparator interface, not an official-code result. Once a licensed official implementation has actually been run with the exact features, masks, train/test identities and training-label restrictions, verify its provenance and keyed predictions with:

python ../archived_evidence/reproducibility/verify_official_comparator.py \\
  --manifest /path/to/verified_official_run.json \\
  --results /path/to/official_predictions.csv \\
  --reference results/funsd/official_reference_ids.csv

A provenance contract or GitHub URL alone does NOT constitute an executed published baseline. No comparator result has been added to this package.

Paired, form-grouped uncertainty (after a genuine FUNSD run)

python paired_group_bootstrap.py \
  --predictions results/funsd/predictions.csv.gz \
  --dataset funsd --condition M20_D50 --seed 11 \
  --method-a denoised_gated --method-b masked \
  --replicates 500 --output results/funsd/group_uncertainty_seed11.json

This resamples complete forms, not individual OCR words. The same command accepts --dataset rvlcdip for page-level groups or deduplicated source-document IDs. It is a conditional inference check for that fixed test split and does not make repeated training seeds independent datasets.

New, distinct joint-contrastive reference implementation

run_joint_contrastive_reference.py adds jointly trained view encoders, reliability-conditioned residual maps, learned alignment projections and all three loss terms. Its source-locked predictions are compatible with local baseline scoring, but it is not an original checkpoint replay or a result on canonical RVL-CDIP/FUNSD. Use compare_joint_reference.py for source-matched, group-level descriptive differences. Full instructions: JOINT_REFERENCE_AND_DATA_GUIDE.md.