|
Download docs/DATASETS.md from prosperious/ocr-tech-k2: direct link, hf CLI and curl.
- Browser
- Download file 11.5 kB
-
https://huggingface.co/prosperious/ocr-tech-k2/resolve/main/docs/DATASETS.md
- Command line
-
hf download hf://prosperious/ocr-tech-k2/docs/DATASETS.md
-
curl -L -o DATASETS.md https://huggingface.co/prosperious/ocr-tech-k2/resolve/main/docs/DATASETS.md
11.5 kB
| # Dataset Workflow | |
| The dataset workflow is deliberately strict. If train/eval contamination gets in here, every later validation result is suspect. | |
| ## Prepare | |
| Convert raw labels to JSONL: | |
| ```bash | |
| ocrtech prepare-data labels.csv \ | |
| --out data/manifests \ | |
| --dataset-name nepali-lines \ | |
| --split all \ | |
| --image-field image_path \ | |
| --text-field text \ | |
| --strict-chars | |
| ``` | |
| If the source is noisy and you need an explicit reject list instead of failing at the first bad row: | |
| ```bash | |
| ocrtech prepare-data labels.csv \ | |
| --out data/manifests \ | |
| --dataset-name nepali-lines \ | |
| --split all \ | |
| --image-field image_path \ | |
| --text-field text \ | |
| --skip-invalid | |
| ``` | |
| That writes: | |
| - `data/manifests/nepali-lines-all.jsonl` | |
| - `data/manifests/nepali-lines-all-prepare.json` | |
| - `data/manifests/nepali-lines-all-prepare.md` | |
| - `data/manifests/nepali-lines-all-rejects.jsonl` when any rows are skipped | |
| Verified Hugging Face sources can be converted directly: | |
| ```bash | |
| ocrtech prepare-data gauravgiri/nepali-ocr-dataset \ | |
| --hf \ | |
| --split train \ | |
| --limit 100 \ | |
| --skip-invalid \ | |
| --out data/manifests | |
| ocrtech prepare-data prashant0919/nepali-synthetic-ocr-lines \ | |
| --hf \ | |
| --split train \ | |
| --limit 100 \ | |
| --out data/manifests | |
| ocrtech prepare-data nvidia/OCR-Synthetic-Multilingual-v1 \ | |
| --hf \ | |
| --hf-subset en \ | |
| --split test \ | |
| --limit 1000 \ | |
| --out data/manifests-english-test | |
| ``` | |
| Before those conversion commands, record a schema inspection so the training | |
| corpus has evidence that fields were verified from the actual source: | |
| ```bash | |
| ocrtech inspect-hf-dataset gauravgiri/nepali-ocr-dataset \ | |
| --split train \ | |
| --limit 3 \ | |
| --out outputs/hf-inspect-training-v1 | |
| ocrtech inspect-hf-dataset prashant0919/nepali-synthetic-ocr-lines \ | |
| --split train \ | |
| --limit 3 \ | |
| --out outputs/hf-inspect-training-v1 | |
| ocrtech inspect-hf-dataset nvidia/OCR-Synthetic-Multilingual-v1 \ | |
| --hf-subset en \ | |
| --split test \ | |
| --limit 3 \ | |
| --out outputs/hf-inspect-training-v1 | |
| ``` | |
| Verified on `k2` on `2026-06-15`: | |
| | dataset | split/subset | mode | inferred image | inferred text | result | | |
| | --- | --- | --- | --- | --- | --- | | |
| | `gauravgiri/nepali-ocr-dataset` | `train` | `datasets.load_dataset` | `image` | `text` | pass | | |
| | `prashant0919/nepali-synthetic-ocr-lines` | `train` | HF repo `metadata.jsonl` | `file_name` | `text` | pass | | |
| | `nvidia/OCR-Synthetic-Multilingual-v1` | `test/en` | HDF5 `line_bboxes` | `images + annotations.line_bboxes[].bbox` | `annotations.line_bboxes[].text` | pass | | |
| Artifacts: | |
| - `outputs/hf-inspect-training-v1/gauravgiri__nepali-ocr-dataset-train-inspect.json` | |
| - `outputs/hf-inspect-training-v1/prashant0919__nepali-synthetic-ocr-lines-train-inspect.json` | |
| - `outputs/hf-inspect-training-v1/nvidia__OCR-Synthetic-Multilingual-v1__en-test-inspect.json` | |
| ## Public Benchmarks | |
| Public document parsing benchmarks are tracked separately from training data. | |
| Do not mix them into recognizer/corrector training manifests. They are final | |
| claim evidence. | |
| List the known registry: | |
| ```bash | |
| ocrtech list-public-benchmarks | |
| ``` | |
| Current target order: | |
| 1. `opendatalab/OmniDocBench` for broad document parsing. | |
| 2. `PaddlePaddle/Real5-OmniDocBench` for real-world physical distortions. | |
| 3. `allenai/olmOCR-bench` for PDF-to-Markdown/unit-test style evaluation. | |
| Prepare the verified OmniDocBench schema: | |
| ```bash | |
| ocrtech prepare-public-benchmark opendatalab/OmniDocBench \ | |
| --out data/public/omnidocbench \ | |
| --limit 100 \ | |
| --language english \ | |
| --require-tables | |
| ``` | |
| This converter is schema-bound. It expects the verified Hugging Face layout: | |
| `OmniDocBench.json` as a list of rows, each row with `page_info.image_path`, | |
| `page_info.page_attribute`, and `layout_dets`. Table references come from | |
| `layout_dets[].html`; text references come from ordered `layout_dets[].text` | |
| plus table HTML text. Images are resolved under `images/`. | |
| Before benchmarking a public pack, audit its manifest: | |
| ```bash | |
| ocrtech audit-public-benchmark data/public/omnidocbench/manifests/omnidocbench-eval.jsonl \ | |
| --benchmark opendatalab/OmniDocBench \ | |
| --out outputs/public-omnidocbench-audit | |
| ``` | |
| Each public manifest row must carry provenance: | |
| - `metadata.public_benchmark = true` | |
| - `metadata.benchmark_name` | |
| - `metadata.source_url` | |
| - `metadata.reference_path` | |
| - `metadata.slices` containing `public` | |
| This is intentionally stricter than `prepare-data`. The agent must verify the | |
| actual downloaded dataset schema before writing a converter for any public | |
| benchmark. No schema guessing. | |
| Current claim-sized public OmniDocBench table pack on `k2`: | |
| - manifest: `data/public/omnidocbench-table100/manifests/omnidocbench-eval.jsonl` | |
| - prepare report: `data/public/omnidocbench-table100/public-benchmark-prepare.json` | |
| - audit: `outputs/public-omnidocbench-table100-audit/public-benchmark-audit.json` | |
| - audit status: passed | |
| - samples: `100` | |
| - slices: `public=100`, `document_parsing=100`, `english=100`, `table=100`, | |
| `table_hard=32`, `figure=22` | |
| This is the first public OmniDocBench manifest in the repo that can satisfy the | |
| configured `100`-page public document gate and the `40`-page public table gate. | |
| For the current Nepali starter corpus, merge verified sources with the built-in command instead of shell concatenation: | |
| ```bash | |
| ocrtech merge-manifests \ | |
| data/manifests/gauravgiri__nepali-ocr-dataset-train.jsonl \ | |
| data/manifests/prashant0919__nepali-synthetic-ocr-lines-train.jsonl \ | |
| --out data/manifests/nepali-mixed-all.jsonl | |
| ``` | |
| Do not pretend all HF datasets look the same. This repo only has verified adapters for the schemas we checked: | |
| - `gauravgiri/nepali-ocr-dataset`: `image`, `text` | |
| - `prashant0919/nepali-synthetic-ocr-lines`: `train/metadata.jsonl` plus image files | |
| - `cfilt/RoundTripOCR-nepali`: `ocr`, `correct`, `font` | |
| - `nvidia/OCR-Synthetic-Multilingual-v1`: HDF5 shards with `line_bboxes` | |
| Known adapters also stamp verified slice metadata so later audits and experiments do not depend on hand-added tags: | |
| - `gauravgiri/nepali-ocr-dataset`: `nepali`, `real` | |
| - `prashant0919/nepali-synthetic-ocr-lines`: `nepali`, `synthetic` | |
| - `nvidia/OCR-Synthetic-Multilingual-v1 --hf-subset en`: `english`, `synthetic`, `line_crop`, `en` | |
| Verified on `k2` on June 14, 2026: | |
| - `gauravgiri/nepali-ocr-dataset --limit 100 --skip-invalid`: `97` prepared, `3` rejected due to empty `text` | |
| - `prashant0919/nepali-synthetic-ocr-lines --limit 100`: `100` prepared, `0` rejected | |
| - merged starter corpus: `197` samples, no merge warnings | |
| - deterministic split: `167` train / `30` eval, leakage check passed | |
| Each manifest row contains: | |
| - `sample_id` | |
| - `dataset` | |
| - `split` | |
| - `image_path` | |
| - `text` | |
| - `sha256` | |
| - `metadata.text_sha256` | |
| - `metadata.sample_sha256` | |
| ## Audit | |
| Before splitting or training: | |
| ```bash | |
| ocrtech audit-manifest data/manifests/nepali-lines-all.jsonl \ | |
| --out outputs/manifest-audit \ | |
| --require-slice nepali \ | |
| --require-slice english \ | |
| --strict-chars | |
| ``` | |
| The audit checks: | |
| - missing input files | |
| - image SHA-256 mismatches | |
| - duplicate sample IDs | |
| - duplicate paths and hashes | |
| - slice counts | |
| - script counts | |
| - unsupported characters when requested | |
| ## Split | |
| Create train/eval manifests with deterministic grouping: | |
| ```bash | |
| ocrtech split-manifest data/manifests/nepali-lines-all.jsonl \ | |
| --out data/splits \ | |
| --eval-ratio 0.15 \ | |
| --seed 13 \ | |
| --stratify-by slices | |
| ``` | |
| The splitter groups entries by `metadata.sample_sha256` before assigning splits by default. If the same image+text sample appears twice, it stays in one split. | |
| For rendered packs with multiple degraded variants of the same underlying text, group by `text_sha256` instead. This keeps clean, blur, scan, low-contrast, and skew variants of the same target text on the same side of the train/eval boundary: | |
| ```bash | |
| ocrtech split-manifest data/eval-pack-table-figure-dev-v1/manifests/ocrtech-eval-pack.jsonl \ | |
| --out data/splits-table-figure-dev-v1 \ | |
| --eval-ratio 0.2 \ | |
| --seed 17 \ | |
| --stratify-by document_type \ | |
| --group-by text_sha256 | |
| ``` | |
| Supported grouping keys are: | |
| - `sample_signature` | |
| - `text_sha256` | |
| - `image_sha256` | |
| - `image_path` | |
| - `sample_id` | |
| Outputs: | |
| ```text | |
| data/splits/ | |
| train.jsonl | |
| eval.jsonl | |
| split-summary.json | |
| split-summary.md | |
| ``` | |
| The split summary records the grouping key. For the table/figure development pack, the verified `k2` split produced: | |
| - `60` train samples | |
| - `20` eval samples | |
| - `16` text groups | |
| - `group_by = text_sha256` | |
| - leakage check passed | |
| ## Rebalance | |
| Use `rebalance-manifest` only on the training split when a slice is obviously underrepresented and you want a reproducible upsampling step instead of an undocumented training-time sampler choice: | |
| ```bash | |
| ocrtech rebalance-manifest data/splits/train.jsonl \ | |
| --out data/splits/train-rebalanced.jsonl \ | |
| --slice english \ | |
| --slice nepali \ | |
| --seed 13 | |
| ``` | |
| Outputs: | |
| ```text | |
| data/splits/ | |
| train-rebalanced.jsonl | |
| train-rebalanced-rebalance.json | |
| train-rebalanced-rebalance.md | |
| ``` | |
| This command duplicates rows with explicit provenance in metadata: | |
| - `rebalance_duplicate` | |
| - `rebalance_source_sample_id` | |
| - `rebalance_slice` | |
| Do not rebalance the unsplit combined manifest. That would duplicate samples before train/eval assignment and corrupt the benchmark. | |
| ## Train | |
| Use only the split outputs: | |
| ```bash | |
| ocrtech train-recognizer \ | |
| --train-manifest data/splits/train.jsonl \ | |
| --eval-manifest data/splits/eval.jsonl \ | |
| --out runs/recognizer | |
| ``` | |
| For generated document packs, do not train a recognizer on whole rendered pages unless the model is explicitly document-level. The current full-page HF branch failed badly. Extract line/cell/caption crops first, then split those crop rows with text-level grouping: | |
| ```bash | |
| ocrtech extract-crops \ | |
| data/eval-pack-table-figure-dev-v1/manifests/ocrtech-eval-pack.jsonl \ | |
| --out data/crops-table-figure-dev-v1 \ | |
| --crop-type line \ | |
| --crop-type table-cell \ | |
| --crop-type figure-caption | |
| ocrtech split-manifest data/crops-table-figure-dev-v1/manifests/crops.jsonl \ | |
| --out data/splits-crops-table-figure-dev-v1 \ | |
| --eval-ratio 0.2 \ | |
| --seed 17 \ | |
| --stratify-by document_type \ | |
| --group-by text_sha256 | |
| ``` | |
| The crop extractor only reads explicit renderer layout metadata from the reference JSON. If that metadata is absent, it fails instead of guessing bboxes. | |
| Before training, build the recognizer corpus from verified manifests with | |
| explicit bucket targets: | |
| ```bash | |
| ocrtech build-recognizer-corpus \ | |
| data/manifests/english-lines.jsonl \ | |
| data/manifests/nepali-lines.jsonl \ | |
| data/splits-crops-table-figure-dev-v1/train.jsonl \ | |
| --out data/splits/recognizer-train-balanced.jsonl \ | |
| --target-latin-only 200 \ | |
| --target-devanagari-only 200 \ | |
| --target-mixed 100 \ | |
| --target-real 100 \ | |
| --target-synthetic 100 | |
| ``` | |
| This command upsamples existing examples only. If a requested bucket is missing | |
| from the inputs, it fails; add source data instead of weakening the audit gate. | |
| For OCR correction from the verified CFILT dataset: | |
| ```bash | |
| ocrtech train-corrector \ | |
| --hf-dataset cfilt/RoundTripOCR-nepali \ | |
| --hf-split train \ | |
| --limit 1000 \ | |
| --out runs/corrector-cfilt | |
| ``` | |
| Do not point training at the combined manifest. | |
| ## Validation | |
| Pass the training manifests into claim validation: | |
| ```bash | |
| ocrtech validate-claim \ | |
| --benchmark-report outputs/benchmark/report.json \ | |
| --eval-manifest data/eval-pack/manifests/ocrtech-eval-pack.jsonl \ | |
| --train-manifest data/splits/train.jsonl \ | |
| --config validation.json \ | |
| --out outputs/validation | |
| ``` | |
| The validation step checks train/eval leakage again. That is intentional redundancy. | |