project/ROADMAP_AND_PROGRESS.md

LongMemory-Geo roadmap and progress

Updated: 2026-11-05

This is the operational handoff for building a high-quality private multi-task geospatial perception model. It supplements PROJECT_STATE.md; it does not overwrite historical evidence, test reports, or checkpoints.

Current execution roadmap — 99% target

The researched plan is ROADMAP_99_PERCENT.md; primary sources are recorded in docs/reviews/2026-11-05-99-roadmap-sources.md and hashed local evidence is in evaluation/runs/2026-11-05-99-roadmap-evidence.json.

This supersedes the next-action interpretation below: correct and audit sampling and evaluation before further model experiments. The class-balanced sampler uses coverage/source-image labels instead of eligible tile positives in the historical runs, the run named rehearsal900 actually used 812 DOTA tiles, and all recent runs completed one epoch. Their candidate checkpoints regressed, but these runs do not establish that more data or class balancing cannot help after convergence. The measured best retained SODA-A development checkpoint remains unpromoted.

Phase 0 progress: the repaired sampler now derives class membership from the same clipped/ignored targets consumed by training and logs replacement exposure; SODA-A maps both airplane and helicopter to aircraft through their distinct reviewed detector head classes. The full train/validation-only audit is evaluation/runs/2026-11-05-soda-a-supervision-audit.json: 25,446 train and 13,740 validation tiles, with 12,927/8,822 unique eligible aircraft source annotations, respectively. It read no official test annotations or imagery. Adapter plus detection/evaluation regression suites passed (41 broader tests). The remaining Phase 0 gates are ignore-policy sensitivity, mining’s one-to-one matching repair, and a full-scene baseline; do not start another training run until those measurement checks are complete.

The first two remaining checks are now complete: the ignore-policy sensitivity report is evaluation/runs/2026-11-05-soda-a-ignore-policy-sensitivity.json, and the regenerated six-class DOTA train-only review queue is evaluation/runs/2026-11-05-dota-train-review-queue-v2.json with sheets in the matching *-sheets-v2/ directory. It is pending source-label review and is not training data. The next gate is the full-scene baseline.

The user's target is broad satellite coverage, including correctly labeled objects. Existing six-class detection cannot certify every object category. Pursue broad coverage and require an explicit, supported all-profile/class acceptance gate; never describe a finite benchmark pass as a universal guarantee.

2026-11-05 SODA-A validation frontier

• VERIFIED: added evaluation/score_soda_a_frontier.py, which evaluates only the validation tile IDs recorded by a SODA-A training run and never materializes the official test split.
• VERIFIED: evaluated training/runs/2026-11-04-soda-a-small-baseline/best.pt on all 60 recorded SODA-A validation tiles using the reviewed small OBB source checkpoint, native 1024 inference, CUDA T4, and the existing 50% ignore-overlap policy. The report reproduces the training metrics: mAP50-95 0.3869758267, AP50 0.7344298656, precision 0.4921524664, recall 0.9070247934.
• VERIFIED: the empirical same-validation 99%-precision frontier is vehicle recall 0.2733, large-vehicle recall 0.1944, aircraft recall 0.7010, vessel has no feasible threshold, and storage-tank recall 0.6433. SODA-A contains no bridge targets, so this run provides no bridge evidence.
• VERIFIED: initial evaluator output was discarded as invalid because it counted unsupported bridge-head predictions. The corrected evaluator filters to SODA-A's exact mapped classes; the corrected report is evaluation/runs/2026-11-05-soda-a-small-baseline-val-v2.json with raw records beside it.
• VERIFIED: tests/model/test_soda_a_adapter.py passed and the new evaluator passes Python compilation. These are development diagnostics only; no model was promoted and the SODA-A official test remains untouched.
• VERIFIED: generated evaluation/runs/2026-11-05-soda-a-error-summary.json from the retained SODA-A validation and DOTA selection records. SODA-A tile-level errors are vehicle 133 FP/23 misses, large vehicle 77/5, aircraft 96/4, vessel 58/1, and storage tank 89/12. DOTA selection also shows bridge 87 FP/3 misses and storage tank 124 FP/4 misses. These are development counts, not qualitative label-error findings.
• VERIFIED: class-balanced sampling was rejected under training/runs/2026-11-05-soda-a-class-balanced/: mAP50-95 0.37258 vs uniform-450 0.38698, with lower same-validation 99%-precision recall for every SODA-A class. DOTA rehearsal-900 was also rejected at mAP50-95 0.35433; storage-tank frontier recall improved modestly, but vehicle fell to 0.1570. Increasing the SODA-A pool to 1,200 tiles was rejected at mAP50-95 0.37447, with vehicle frontier recall collapsing to 0.0581 even though storage-tank recall rose to 0.6959. None of these checkpoints is promoted.
• INFERRED: the current short-run bottleneck is not simply class balance, DOTA rehearsal quantity, or more SODA-A tiles under the fixed 900-second budget. Qualitative review of high-confidence false positives and misses is required before another training variable is selected.
• VERIFIED: localized representative high-confidence false positives and missed targets by class in evaluation/runs/2026-11-05-detection-error-locations.json. The report is based only on retained SODA-A validation and DOTA selection records and does not alter labels or use protected test data.
• VERIFIED: reviewed both rendered error sheets. The observations and limits are recorded in docs/reviews/2026-11-05-soda-a-error-review.md; they identify vehicle/vessel, bridge/linear-infrastructure, and tank/industrial structure confusion hypotheses without treating them as label corrections.
• VERIFIED: mined 812 DOTA training tiles with the retained uniform SODA-A candidate to create a source-label review queue at confidence >= 0.5: 20 vehicle, 20 large-vehicle, 20 vessel, 12 storage-tank, and 7 bridge candidates. The queue is evaluation/runs/2026-11-05-dota-train-hard-negative-review-queue.json and is explicitly ineligible for training until source-label review.

Product direction

The target is a practical RGB overhead-imagery model with a shared backbone and three task heads:

1. Oriented-object detection (OBB): bridge, vehicle, large vehicle, aircraft, vessel, and storage tank.
2. Semantic segmentation: building, road, water, vegetation, agricultural area, bare ground, and developed area.
3. Optional scene context: scene-level classification or representation learning. This is where PatternNet belongs; it must not be converted into fabricated object boxes.

The target is to outperform selected public baselines on a fixed, documented benchmark contract. “Better than every public model” is not a meaningful claim without specifying datasets, target imagery, precision/recall operating points, hardware, and latency budget.

Non-negotiable measurement rules

• Keep train, selection-validation, and final test imagery separate.
• Preserve geographic/source grouping; do not train on quarantined audit data.
• Select checkpoints and thresholds only on validation data.
• Report per-class precision, recall, AP50, mAP50-95, object-size results, false-positive/false-negative evidence, latency, and memory.
• A 99% precision/99% recall assertion requires every supported class to meet both targets on a frozen, untouched, adequately supported test set. Training loss, reused development scenes, and in-sample fitted thresholds do not establish the claim.

Dataset roles

Dataset	Role	Task	Current status
Existing DOTA corpus	Rehearsal and historical diagnostics	OBB detection	Existing source; retain rehearsal to reduce forgetting.
FAIR1M mirror	Adaptation/development	OBB detection	1,732 paired images; footprint-overlap groups are enforced; audit groups quarantined. Geographic independence remains unresolved.
SODA-A	New detector source	OBB detection	Full archive acquired and adapter/trainer implemented. Official test split remains untouched.
Existing OpenEarthMap corpus	Semantic training/evaluation	Segmentation	Shipped segmenter is the best accepted semantic candidate.
PatternNet	Optional future context/pretraining task	Scene classification	Not detector supervision: image-level labels, no object boxes.

SODA-A contract

• Official source records: 2,513 images.
• Official split inventory: 1,067 train, 576 validation, 870 test images.
• Derived native 1024px tiles: 25,446 train, 13,740 validation, and 20,755 test tiles.
• Exact detector mappings: airplane→aircraft, small-vehicle→vehicle, large-vehicle→large vehicle, ship→vessel, storage-tank→storage tank.
• Unsupported SODA-A categories, official ignore areas, and malformed non-quadrilateral annotations remain ignored regions. They are never made into background or artificial OBB labels.

Implemented continuation work (2026-11-04)

SODA-A acquisition and adapter

• Downloaded datasets/external/soda-a/raw/Annotations.zip and Images.zip.
• Added training/soda_a.py:
  • validates official image/annotation pairing;
  • reads the flat image archive and official split annotations;
  • produces padded 1024px source tiles with source-coordinate OBBs;
  • preserves partial/ignored regions;
  • retains SODA-A test data only for future frozen evaluation.
• Extended training/detection_data.py so SODA-A ignored categories use the same explicit source-ignore contract as FAIR1M/DIOR.
• Added tests/model/test_soda_a_adapter.py.
• Focused SODA-A plus FAIR1M adapter tests passed.
• Real GPU smoke passed on an annotated SODA-A tile: finite loss 2.7731, 12 supervised positives, and peak allocation approximately 1,386.5 MiB on the T4.

SODA-A bounded baseline

Run: training/runs/2026-11-04-soda-a-small-baseline/

• Detector: reviewed yolo26s-obb small architecture.
• Training: 600 SODA-A tiles + 450 DOTA rehearsal tiles.
• Validation: 60 official SODA-A validation tiles.
• Resolution: 1024; batch 1; AdamW; LR 3e-6; frozen BatchNorm statistics.
• Budget: 900 seconds; one full epoch completed; status TIME_LIMIT.
• SODA-A test split was not loaded for model selection, threshold fitting, or training.

Metric	Epoch 0	Epoch 1
SODA-A validation mAP50-95	0.2998	0.3870
SODA-A validation AP50	0.6108	0.7344
Low-threshold precision	0.4701	0.4922
Low-threshold recall	0.7459	0.9070
Mean training loss	—	1.2995

The epoch-1 checkpoint is the current SODA-A development candidate. It is not promoted.

Cross-dataset diagnostic for the SODA-A candidate

Checkpoint: training/runs/2026-11-04-soda-a-small-baseline/best.pt

Evidence: evaluation/runs/2026-11-04-soda-a-small-baseline-dota.json and retained raw records in evaluation/runs/2026-11-04-soda-a-small-baseline-dota-records.json.

Diagnostic	mAP50-95	AP50	Low-threshold precision	Low-threshold recall
DOTA selection (development only)	0.5702	0.8141	0.5473	0.9647
Fixed DOTA quick (development only)	0.5346	—	0.5427	0.9725

Same-selection 99%-precision frontier (optimistic because thresholds are fitted and scored on the same selection rows):

Class	Recall at empirical >=99% precision
Bridge	No feasible threshold (7 targets)
Vehicle	43.6%
Large vehicle	60.2%
Aircraft	100.0%
Vessel	94.3%
Storage tank	45.1%

Conclusion: SODA-A adaptation substantially improves low-threshold recall and does not show catastrophic DOTA collapse, but it does not meet the intended 99% precision/recall objective. Bridges, vehicles, and tanks need targeted work. These DOTA diagnostics are not independent final benchmarks.

FAIR1M latest controlled run

Run: training/runs/2026-11-04-fair1m-small-lr3e-6/

• Grouped FAIR1M split; historical audit groups quarantined.
• 600 FAIR1M train tiles, 450 DOTA rehearsal tiles, 40 DIOR bridge images, 200 reviewed DIOR bridge expansions, and 60 grouped FAIR1M validation tiles.
• Small detector; LR 3e-6; 900-second cap; one epoch completed.

Metric	Initialization	Epoch 1
Grouped FAIR1M mAP50-95	0.2528	0.3141
AP50	0.4707	0.6204
Low-threshold precision	0.4387	0.3106
Low-threshold recall	0.5252	0.7905

This is development evidence only. No FAIR1M checkpoint is promoted.

Roadmap

Phase 1 — establish trustworthy evaluation

1. Preserve existing final/test evidence without retuning on it.
2. Build independent, geographically/source-separated evaluation scenes for intended deployment sensors and resolutions.
3. Audit duplicates/near-duplicates and source overlap across FAIR1M, SODA-A, DOTA, DIOR, and future datasets.
4. Set a frozen benchmark contract: per-class P/R, AP, size strata, segmentation IoU, scene accuracy, tiled-image behavior, latency, and RAM.

Phase 2 — detector data and diagnostics

1. Run a SODA-A validation precision frontier without reading its official test split.
2. Generate false-negative/false-positive reports by class, object size, density, tile boundary, source, and scene type.
3. Inspect high-confidence errors before changing labels or policies.
4. Use DOTA rehearsal and protected source splits in every mixed-source run.

Phase 3 — controlled detector experiments

1. Treat the current SODA-A epoch-1 checkpoint as the baseline.
2. Test exactly one variable at a time, beginning with class-balanced/hard negative sampling for vehicle, bridge, and tank errors.
3. Compare native 1024px tiles with a justified high-resolution or multiscale alternative only on fixed validation scenes.
4. Compare small vs nano or another architecture only with matching data, seed, budget, and evaluation rows.
5. Freeze thresholds on selection data and score once on untouched test data.

Phase 4 — segmentation and scene context

1. Retain the shipped semantic segmenter as the current accepted baseline.
2. Add new segmentation data only with validated class mapping and geographic separation.
3. Evaluate PatternNet separately as scene-context supervision or backbone pretraining; do not merge image-level labels into detector loss.
4. Add a scene head only if it improves measured downstream context or uncertainty handling without harming detector/segmenter quality.

Phase 5 — large-image inference and release

1. Benchmark tile size, overlap, fusion, and seam behavior on full scenes.
2. Keep oriented NMS/fusion and source-coordinate transformations under unit and qualitative tests.
3. Package frozen weights, configs, thresholds, dataset provenance, metrics, and failure analysis.
4. Publish a model card that distinguishes validated claims from development diagnostics.

Immediate next actions

1. Render and review the train-only queue against original source annotations; approve only unambiguous negatives and preserve uncertain examples.
2. Design one hard-negative/confusion-focused run from approved train-only records; retain the uniform 600 SODA-A / 450 DOTA baseline, architecture, LR, validation IDs, and budget.
3. Run and evaluate that one-variable experiment; prioritize reducing high-confidence vehicle, large-vehicle, tank, and bridge false positives while preserving recall. Do not train on SODA-A test or FAIR1M audit groups.
4. Keep PatternNet isolated until a scene-head or pretraining experiment is explicitly designed and benchmarked.

Known limitations

• The project has no certified independent geographic evaluation suite yet.
• Current 99%-precision frontiers are selection-fitted diagnostics, not final reliability estimates.
• FAIR1M footprint and cross-source dependence remain unresolved.
• SODA-A’s official test is correctly protected but has not yet been used for a single frozen final evaluation.
• PatternNet provides scene labels only; it cannot supply detector boxes.
• The detector ONNX export still lacks strict parity; use PyTorch detector inference for validated results.