Hsyshasuzh/longmemory-geo/project/datasets - Storage Bucket

RGB remote-sensing corpus

This directory is being built. corpus/stats.json is the readiness authority: do not train while training_ready is false or while the file is absent. No final model is trained by these scripts.

Reproduce

Run commands from the workspace root in PowerShell:

uv venv --python 3.13 .venv
uv pip install --python .venv\Scripts\python.exe -r requirements.lock
.\.venv\Scripts\python.exe acquire.py --oem-limit 400

acquire.py retrieves an attribution snapshot, inventories the official OpenEarthMap ZIP, selects paired RGB images and real pixel masks across regions, and downloads only selected ZIP members. Each member is checked against its published ZIP CRC and receives a local SHA256. The full-archive MD5 is not claimed verified after partial range downloads. Reruns check cached member CRCs. Unpaired images and regions with unknown image terms are not selected.

DOTA archive URLs, byte sizes and expected SHA256 values are in build_config.json. Obtain its two selected archives in raw/ using those exact names. The image archive comes from the pinned mirror referenced by TorchGeo, not a relicensed dataset. Its imagery is v1.0; annotations are v1.5.

Once both acquisitions are complete:

.\.venv\Scripts\python.exe prepare.py --output datasets\corpus
.\.venv\Scripts\python.exe validate.py --root datasets\corpus
.\.venv\Scripts\python.exe validate.py --root datasets\corpus --previews-only

Inspect all generated preview sheets and record the outcome in the visual review record corpus/visual_review.json before marking the corpus ready. The record must contain passed, content_sha256 copied from the validation report, inspected_files listing every preview path, and written observations. Then run:

.\.venv\Scripts\python.exe validate.py --root datasets\corpus --finalize

Finalization rechecks every file and matches the reviewed content hash. Validation does not automatically claim that a human/agent saw the previews. Existing corpus output is not overwritten: choose a new output directory for a different selection.

Load from Prompt 2

After readiness is confirmed:

.\.venv\Scripts\python.exe load_dataset.py --task detection --split train --license-scope all
.\.venv\Scripts\python.exe load_dataset.py --task segmentation --split train --license-scope all

from load_dataset import dataset

training = dataset("datasets/corpus", "detection", "train", license_scope="all")
sample = training[0]

The loader is framework-neutral and does not install PyTorch or start training. Images are HWC uint8 RGB. Detection boxes are float32 xywh; class IDs are the stable 1-based ontology IDs, not automatically contiguous head indices. Segmentation masks are uint8 canonical IDs, with ignore value 255. Use an explicit canonical-to-head index mapping if a model expects 0-based IDs.

Required training behavior

• Respect valid_mask: it excludes image padding and, for detection, difficult or severely clipped objects. A detector adapter must mask unsupported classification/objectness losses and preserve valid regions through augmentation.
• Source raster nodata masks and fully black RGB pixels are excluded from supervision. This deliberately conservative rule may ignore a small number of valid zero-valued pixels; it does not invent labels under black borders.
• Respect annotated_classes and unknown_classes. DOTA does not exhaustively label buildings, roads or land cover; unannotated classes are not negatives.
• Detection annotations retain the original source OBB, source annotation ID, original class, visible fraction and derived tile polygon. OBBs are not pixel-accurate instance masks. The supplied COCO-like index is not sufficient by itself for partial-label/ignored-region training.
• obb_polygon uses original source pixels; clipped_polygon and bbox use tile pixels. Some clipped polygons have more than four vertices. Do not assume every clipped polygon is an oriented rectangle.
• Padding is used rather than upscaling small images. Large images use native 1024-pixel tiles with an 819-pixel stride and edge-aligned final windows.
• Objects with less than 70% visible polygon area or a derived side below two pixels are ignored, not silently turned into background.
• Keep source-scene counts separate from tile counts. Overlap can place the same source annotation in multiple tiles; use provenance to count unique objects.
• Use dynamic, label-aware flips and rotations, moderate photometric changes, blur/compression and conservative scale jitter. Do not generate permanent augmented copies, interpolate masks bilinearly, or downsample tiny vehicles beyond useful resolution.
• Use class-aware sampling or capped inverse-square-root frequency weights, not thousands of duplicated files. See corpus/metadata/sampling.json.

Splits and limitations

OpenEarthMap splits group entire attributed cities/regions, including matching region aliases. DOTA splits group original source scenes. Its selected official validation release is repurposed for this internal corpus; results must not be reported as official DOTA benchmark results.

DOTA does not provide absolute geolocation for every scene. Scene separation is not proof of complete geographic independence across datasets. Exact and near-duplicate checks reduce but cannot eliminate this uncertainty. Preserve this limitation in any model report; do not claim a fully geospatially certified benchmark.

Railway infrastructure, parking areas and construction have no reliable coverage in the initial DOTA/OpenEarthMap selection. They remain ontology entries with zero counts, not fabricated labels. xView is a future candidate for some gaps; rail vehicles and train stations are not railway tracks.

License handling

An all scope can contain academic-only data and is not commercial clearance. DOTA prohibits commercial use. OpenEarthMap labels inherit source terms, except its labels on public-domain/unspecified-license imagery use CC-BY-NC-SA-4.0. Unspecified image permissions are excluded even when label terms are stated.

Each sample retains its source license and license group. The loader defaults to commercial_compatible; selecting all or academic_only is explicit. Attribution/share-alike obligations still apply to permissively selected sources. Do not redistribute a flattened combined archive without satisfying all terms. No code repository's MIT/BSD license is treated as an imagery license.

Layout

• SOURCES.json: research inventory and source decisions; unknowns are explicit.
• ontology.json: canonical IDs and proposed/reviewed source mappings.
• build_config.json: bounded source selection and acquisition metadata.
• raw/: downloaded archives and original selected image/mask files.
• metadata/: acquisition, archive-member and license manifests.
• corpus/detection/{train,val,test}/: PNG images, per-image labels and COCO index.
• corpus/segmentation/{train,val,test}/: PNG images and separate indexed masks.
• corpus/metadata/: shared samples/scenes, selection and sampling metadata.
• corpus/previews/: class-specific detection sheets and segmentation overlays.
• corpus/rejected.jsonl: explicit exclusion and selection-limit reasons.
• corpus/stats.json: measured counts, size distributions and readiness.

Keep the source archives and acquisition manifests for reproducibility. The 20 GB approved budget covers original downloads, output and local tooling. Writers count existing workspace files, including prior outputs and partial downloads, before reserving further writes. Do not run unrelated writers in the workspace concurrently with a build. Revalidation immediately revokes readiness; the training loader also verifies each loaded image/label against approved hashes.