docs/dataset_standard.md

# Typical Marine Ecological Environment Feature Dataset Standard

This project uses a manifest-driven dataset layout for marine ecological environment
feature recognition. It is designed to keep different patch sizes, satellites,
sensors, spatial resolutions, and fusion states instead of forcing everything into
one fixed tile format.

## Scope

Target feature families:

- `green_tide`: Enteromorpha / green tide / seaweed.
- `red_tide`: red tide / harmful algal bloom.
- `golden_tide`: Sargassum / golden tide.
- `aquaculture`: aquaculture area, rafts, cages, ponds, or related facilities.
- `water`: valid water background.
- `land`: land or non-water hard negative.
- `invalid`: black border, no-data, saturated, missing, or otherwise unusable pixels.
- `cloud_shadow`: cloud, haze, cloud shadow, or bright atmospheric interference.
- `other`: valid but not assigned to the categories above.

No external coastline, land-mask vector, or GIS mask is assumed to exist. Land,
invalid area, and water/background handling must be represented by dataset labels,
hard-negative samples, or image-derived validity rules.

## Canonical Directory Layout

```text
data_marine_features/
  manifests/
    assets_raw.jsonl
    samples.jsonl
    splits/
      train.txt
      val.txt
      test.txt
  images/
    <asset_id>.<ext>          # optional symlink or copied tile
  masks/
    <sample_id>.<ext>         # optional symlink or copied mask
  previews/
    <sample_id>.png
  reports/
    asset_inventory.csv
```

The canonical source of truth is `manifests/samples.jsonl`. Files may remain in
their original locations; `images/` and `masks/` are optional conveniences.

## `samples.jsonl` Schema

Each line is one image sample or tile.

```json
{
  "sample_id": "gf6_20250604_green_tide_000001",
  "element": "green_tide",
  "task_type": "semantic_segmentation",
  "image_path": "D:/.../image.tif",
  "mask_path": "D:/.../mask.tif",
  "annotation_path": null,
  "annotation_format": "mask",
  "label_encoding": {"0": "background", "1": "green_tide"},
  "satellite": "GF6",
  "sensor": "PMS",
  "resolution_m": 2.0,
  "patch_size": 256,
  "bands": ["blue", "green", "red", "nir"],
  "band_count": 4,
  "dtype": "uint16",
  "fusion": {
    "state": "fused_product",
    "method": "unknown_vendor_product",
    "sources": [
      {"role": "MSS", "path": null, "resolution_m": 8.0},
      {"role": "PAN", "path": null, "resolution_m": 2.0}
    ],
    "target_resolution_m": 2.0,
    "native_multispectral_resolution_m": 8.0,
    "persisted": true,
    "reproducible": false,
    "spectral_preservation": "unknown",
    "notes": "Fused image supplied as a finished raster; source PAN/MSS not guaranteed."
  },
  "acquired_at": "2025-06-04",
  "source_project": "EntGreenTide",
  "source_dataset": "GF6_PMS_E121.1_N33.6_20250604_L1A1420584616",
  "split": "train",
  "quality_flags": ["valid_image"],
  "notes": ""
}
```

Required fields:

- `sample_id`
- `element`
- `task_type`
- `image_path`
- `satellite`
- `sensor`
- `patch_size`
- `band_count`
- `fusion`
- `source_project`

Recommended fields:

- `mask_path`
- `annotation_path`
- `annotation_format`
- `resolution_m`
- `bands`
- `dtype`
- `acquired_at`
- `split`
- `quality_flags`

## Multi-Scale Policy

Patch sizes such as 128, 256, 512, and full-scene windows are all valid. They are
not merged destructively. Training code should sample them with metadata-aware
transforms:

- Keep `patch_size` in the manifest.
- Resize only inside the training transform when the model requires it.
- Preserve the original spatial resolution in `resolution_m`.
- For full-scene inference, use sliding windows whose size is a runtime parameter.

## Multi-Sensor And Fusion Policy

Different satellites and sensors are expected:

- GF1, GF2, GF6, and other optical satellites can coexist.
- Fused PMS/MUX products and raw PAN+MSS products can coexist.
- If PAN+MSS are available, keep both paths in `fusion.sources`.
- Do not assume band order from the filename alone; record `bands` when known.

Fusion must be represented explicitly. A boolean such as `is_fused=true` is not
enough because it loses the source products, method, target resolution, and
reproducibility.

Recommended `fusion` object for an already fused raster:

```json
{
  "state": "fused_product",
  "method": "unknown_vendor_product",
  "sources": [
    {"role": "MSS", "path": "D:/.../MSS2.tiff", "resolution_m": 8.0},
    {"role": "PAN", "path": null, "resolution_m": 2.0}
  ],
  "target_resolution_m": 2.0,
  "native_multispectral_resolution_m": 8.0,
  "persisted": true,
  "reproducible": false,
  "spectral_preservation": "unknown",
  "notes": "Fused image supplied as a finished raster; original PAN is unavailable."
}
```

Recommended `fusion` object for streaming PAN+MSS fusion:

```json
{
  "state": "runtime_fusion",
  "method": "additive_component_substitution",
  "sources": [
    {"role": "MSS", "path": "D:/.../MSS2.tiff", "resolution_m": 8.0},
    {"role": "PAN", "path": "D:/.../PAN2.tiff", "resolution_m": 2.0}
  ],
  "target_resolution_m": 2.0,
  "native_multispectral_resolution_m": 8.0,
  "persisted": false,
  "reproducible": true,
  "tile_aligned": true,
  "notes": "Fusion is performed per inference tile and is not written as a full-scene raster."
}
```

Allowed `fusion.state` values:

- `none`: native product, no fusion.
- `fused_product`: fused raster exists on disk.
- `runtime_fusion`: fusion happens during training/inference.
- `unknown`: insufficient information; do not pretend it is native.

## Label Policy

The final model should not be binary seaweed/background. It should learn ecological
elements and scene context. A recommended unified semantic target is:

| Class ID | Name |
| --- | --- |
| 0 | other/background |
| 1 | invalid |
| 2 | water |
| 3 | land |
| 4 | cloud_shadow |
| 5 | green_tide |
| 6 | red_tide |
| 7 | golden_tide |
| 8 | aquaculture |

When legacy binary masks are imported, they should be represented as task-specific
labels plus metadata. Do not silently treat unlabeled pixels as true negatives for
all other ecological elements.

## Quality Gate Policy

Discovered samples are not automatically training samples. A sample may enter
`accepted_samples.jsonl` only after an explicit adapter verifies:

- the data source is relevant to marine or coastal remote sensing;
- image and label assets are reliably paired;
- label semantics map to a registry element card;
- label format is parsed successfully, such as mask, polygon, GeoJSON, SHP, or RLE;
- box-only labels are rejected from SAMPoly-style polygon training because they
  cannot supervise true vertices or boundaries;
- unusable images, empty labels, unknown elements, and ambiguous licenses are
  rejected or kept for manual review;
- quality flags record why the sample was accepted.

Unknown or unpaired samples must not be treated as negative samples.

## Inventory Rules

The inventory scanner should:

- Preserve original file paths.
- Infer element type from directory names and known project folders.
- Infer satellite/sensor/acquisition date from filenames when possible.
- Infer fusion state from filenames such as `fuse`, `PAN`, and `MSS`, but mark
  uncertain products as `unknown` instead of guessing method details.
- Pair image and mask files by stem when masks exist.
- Record unpaired full-scene images as inference assets.
- Flag ambiguous data instead of guessing labels.