Revolution Crossroads DataLab Chandra OCR 2 Outputs

This repository contains the complete processing outputs generated by DataLab's Chandra OCR 2, a Vision Language Model (VLM) for document understanding, for materials processed as part of the Revolution Crossroads project.

Rather than embedding every processing artifact directly within the primary Hugging Face datasets, these files are stored separately and referenced from the corresponding dataset records. This keeps the primary datasets compact while preserving access to the complete outputs for researchers who wish to inspect, validate, reproduce, or build upon the processing results.

About Revolution Crossroads

Revolution Crossroads is a collaborative initiative between the Smithsonian Institution, the Library of Congress, and the National Archives exploring how artificial intelligence can responsibly improve access to Revolutionary Era collections.

The project develops openly available datasets and AI workflows to support historical research, transparency, and public engagement while encouraging critical examination of both the potential and limitations of AI in libraries, archives, and museums.

Learn more about the project on the Revolution Crossroads website.

About the Processing

The outputs in this repository were generated by DataLab's Chandra OCR 2, a 4-billion parameter Vision Language Model (VLM) designed for document understanding.

Unlike conventional OCR systems that primarily extract text, Chandra OCR 2 analyzes each page as a structured visual document. In addition to extracting text, the model identifies document structure, reading order, layout, tables, mathematical notation, and other structural elements.

The outputs preserve page layout and locality information through layout block classifications and bounding boxes that map extracted content back to its original location within the source document. The model also extracts embedded visual elements into separate image files and generates descriptions of those visual elements when appropriate. These may include illustrations, figures, stamps, seals, diagrams, icons, maps, and other non-textual content.

The extracted text from these outputs has been incorporated directly into the Revolution Crossroads PDF-level datasets. This repository preserves the complete processing outputs for users who wish to examine the model's results in greater detail or utilize additional information that is not included in the primary datasets.

Processing Provenance

Repository Organization

Each processed source document has its own folder containing the complete set of outputs produced by Chandra OCR 2.

The associated Revolution Crossroads datasets include links to these folders, allowing users to access the complete processing outputs when needed.

For most users, the extracted text included directly within the Revolution Crossroads PDF-level datasets will be sufficient. This repository is intended for users who require the complete processing outputs, intermediate representations, quality metrics, extracted images, or page-level layout information.

The overall workflow is:

Source document
        │
        ▼
DataLab Chandra OCR 2 processing
        │
        ├── Markdown
        ├── HTML
        ├── LaTeX
        ├── Structured JSON
        ├── Layout analysis
        ├── Quality metrics
        ├── Extracted images
        └── Other processing outputs
                 │
                 ▼
Published in this repository
                 │
                 ▼
Referenced from the Revolution Crossroads datasets
                 │
                 ▼
Extracted text incorporated into the PDF-level Parquet datasets

Beyond Plain Text

While the extracted OCR text is incorporated into the Revolution Crossroads PDF-level datasets, this repository preserves the complete set of outputs generated by Chandra OCR 2.

These outputs include:

• Extracted text
• Document structure and reading order
• Page layout and locality information
• Layout block classifications
• Bounding boxes for individual content blocks
• Structured representations in Markdown, HTML, JSON, and LaTeX
• Per-page quality assessments
• Images extracted from the source document
• Model-generated descriptions of extracted visual elements

Together, these outputs provide a rich representation of each document that supports transparency and reproducibility while enabling research beyond text extraction, including document layout analysis, multimodal retrieval, computer vision applications, and future AI workflows.

Processing Output Files

For every processed document, Chandra OCR 2 generates the following companion files:

Relationship to the Revolution Crossroads Datasets

These processing outputs correspond to records in the published Revolution Crossroads datasets on Hugging Face.

The primary datasets contain:

• Standardized metadata
• Source identifiers
• Links to these processing outputs (when available)
• The extracted OCR text (for PDF-level datasets)

Not every record in the Revolution Crossroads datasets currently has corresponding Chandra OCR 2 outputs.

The initial processing, completed in March 2026, included:

• Smithsonian collection records selected because they were likely to contain textual content
• All Chronicling America newspaper issues included in the project at that point
• National Archives Revolutionary War Pension Files that had not yet been human transcribed at the time of processing

Since that initial processing, the Revolution Crossroads datasets have been refreshed with additional source materials as the contributing collections have continued to evolve. A second round of Chandra OCR 2 processing is currently underway. As those outputs become available, they will be added to this repository and linked from the corresponding dataset records. The extracted text will also be incorporated into future releases of the PDF-level datasets.

Users interested primarily in text analysis should begin with the Revolution Crossroads datasets. This repository exists to preserve the complete processing outputs for transparency, reproducibility, evaluation, and downstream research while making available rich document representations that extend beyond the extracted text included in the primary datasets.

Notes

The files in this repository represent the direct outputs of DataLab's Chandra OCR 2 processing pipeline. They have not been manually corrected or edited beyond organization for publication. Researchers seeking cleaned, standardized, or integrated data should use the primary Revolution Crossroads datasets.