Sync from GitHub 2c58fb3
Browse files
README.md
CHANGED
|
@@ -10,7 +10,9 @@ short_description: Major TOM Core samples on a 3D globe, read in the browser
|
|
| 10 |
|
| 11 |
> **CLAUDE-GENERATED DOCUMENT** — automated LLM-generated content. Verify before relying on it.
|
| 12 |
|
| 13 |
-
> **Review status:**
|
|
|
|
|
|
|
| 14 |
|
| 15 |
# Major TOM Raster Viewer
|
| 16 |
|
|
@@ -45,6 +47,10 @@ AlphaEarth Embeddings (aef.js): footprints/AEF.parquet maps each Core-S2L2A cell
|
|
| 45 |
proxy at 10 m (~15 s). A cell's window is joined from every tile it touches (24.6% need two to four). The COGs are
|
| 46 |
stored south-up: origin and row direction from each file. View menu: single bands at 10 m. Downloads: the shown
|
| 47 |
image (.png), the shown band or all 64 at 10 m (GeoTIFF written in the browser), Python (rasterio.merge)
|
|
|
|
|
|
|
|
|
|
|
|
|
| 48 |
Downloads (download.js): the shown band as stored (.tif); every band, the thumbnail and metadata.json (.zip); a
|
| 49 |
Python snippet that reads the same row group from Hugging Face
|
| 50 |
in the background: the other datasets' footprints once the first is drawn; after a click, the same cell's samples
|
|
@@ -61,6 +67,10 @@ in the background: the other datasets' footprints once the first is drawn; after
|
|
| 61 |
| `samples.js` | one sample's thumbnail from its image file |
|
| 62 |
| `bands.js` | a sample's band columns: read, decode (geotiff.js), stretch, composites |
|
| 63 |
| `download.js` | band GeoTIFFs as stored, all bands as a zip (fflate), the Python snippet |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 64 |
| `parquet.js` | range requests for hyparquet, with bytes and time counted; each file's footer read once |
|
| 65 |
| `tiles.js`, `labels.js`, `places.json` | copied from the Climate Globe: cells and tiles, place names |
|
| 66 |
| `build_index.py` | Core `metadata.parquet` -> `footprints/<name>.parquet` and `footprints/datasets.json` |
|
|
@@ -72,11 +82,25 @@ in the background: the other datasets' footprints once the first is drawn; after
|
|
| 72 |
|---|---|---|
|
| 73 |
| S2L2A footprints (coverage of 2,245,886 samples) | 3.9 MB in 37 requests | 0.7 s |
|
| 74 |
| One S2L2A thumbnail, first from its image file (footer 1.5 MB) | 1.1 MB in 4 requests | 1.8 s |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 75 |
|
| 76 |
Reading each dataset's `metadata.parquet` from Hugging Face instead was measured first: S2L2A's has one row group
|
| 77 |
per image file (4,492), so its positions took 462 s in 4,492 requests; S1RTC's and DEM's have two row groups, so
|
| 78 |
one sample's row cost 23-27 MB. Hence the footprint files.
|
| 79 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 80 |
## Run locally
|
| 81 |
|
| 82 |
```
|
|
@@ -86,6 +110,15 @@ pytest -m network # se
|
|
| 86 |
|
| 87 |
The page needs a server that answers HTTP range requests (the tests' server does; `python -m http.server` does not).
|
| 88 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 89 |
## Credit
|
| 90 |
|
| 91 |
Developed by [Mikolaj Czerkawski](https://github.com/mikonvergence), [Asterisk Labs](https://asterisk.coop), as part of
|
|
|
|
| 10 |
|
| 11 |
> **CLAUDE-GENERATED DOCUMENT** — automated LLM-generated content. Verify before relying on it.
|
| 12 |
|
| 13 |
+
> **Review status:** not yet reviewed. First version 2026-11-02; Space created (private) and deployed 2026-11-02.
|
| 14 |
+
> Changed 2026-11-04: Core-Weather panel in the flow; `weather.js`, `viirs.js`, `aef.js` and `zstd.js` in the
|
| 15 |
+
> Files and Reads tables; the private-Space import rule; the test files.
|
| 16 |
|
| 17 |
# Major TOM Raster Viewer
|
| 18 |
|
|
|
|
| 47 |
proxy at 10 m (~15 s). A cell's window is joined from every tile it touches (24.6% need two to four). The COGs are
|
| 48 |
stored south-up: origin and row direction from each file. View menu: single bands at 10 m. Downloads: the shown
|
| 49 |
image (.png), the shown band or all 64 at 10 m (GeoTIFF written in the browser), Python (rasterio.merge)
|
| 50 |
+
Core-Weather (weather.js): the sample's acquisition (S2L2A, S2L1C, S1RTC; for DEM, VIIRS and AlphaEarth the cell's
|
| 51 |
+
S2L2A acquisition, said in the panel) -> hourly/ and daily/ day files on source.coop major-tom/core-weather
|
| 52 |
+
-> grid_cell and acquired columns -> the sample's rows -> charts: hourly ±24 h (ERA5 temperature, precipitation,
|
| 53 |
+
cloud cover), daily ±30 d (ERA5 temperature and precipitation, CAMS PM2.5, Sentinel-5P NO₂)
|
| 54 |
Downloads (download.js): the shown band as stored (.tif); every band, the thumbnail and metadata.json (.zip); a
|
| 55 |
Python snippet that reads the same row group from Hugging Face
|
| 56 |
in the background: the other datasets' footprints once the first is drawn; after a click, the same cell's samples
|
|
|
|
| 67 |
| `samples.js` | one sample's thumbnail from its image file |
|
| 68 |
| `bands.js` | a sample's band columns: read, decode (geotiff.js), stretch, composites |
|
| 69 |
| `download.js` | band GeoTIFFs as stored, all bands as a zip (fflate), the Python snippet |
|
| 70 |
+
| `weather.js` | Core-Weather rows of one acquisition: hourly (±24 h) and daily (±30 d) tables, one SVG chart per variable |
|
| 71 |
+
| `viirs.js` | one VIIRS tile from its zip shard (one range request), radiance in inferno on a log scale 0.1-100 nW/cm²/sr, corners from the GeoTIFF bounds |
|
| 72 |
+
| `aef.js` | AlphaEarth window of a cell: its COG tiles per year, joined; PCA preview at 160 m, PCA proxy from 8 bands at 10 m, single bands |
|
| 73 |
+
| `zstd.js` | zstd decoder (fzstd) registered in geotiff.js for compression 50000 (VIIRS and AlphaEarth GeoTIFFs); reads nothing |
|
| 74 |
| `parquet.js` | range requests for hyparquet, with bytes and time counted; each file's footer read once |
|
| 75 |
| `tiles.js`, `labels.js`, `places.json` | copied from the Climate Globe: cells and tiles, place names |
|
| 76 |
| `build_index.py` | Core `metadata.parquet` -> `footprints/<name>.parquet` and `footprints/datasets.json` |
|
|
|
|
| 82 |
|---|---|---|
|
| 83 |
| S2L2A footprints (coverage of 2,245,886 samples) | 3.9 MB in 37 requests | 0.7 s |
|
| 84 |
| One S2L2A thumbnail, first from its image file (footer 1.5 MB) | 1.1 MB in 4 requests | 1.8 s |
|
| 85 |
+
| Core-Weather for one sample (`weather.js`): `hourly/` and `daily/` day files, source.coop `major-tom/core-weather` | 3.3 MB (LOG.md, 2026-11-03) | not measured |
|
| 86 |
+
| One VIIRS tile (`viirs.js`): the tile's bytes in its zip shard, Hugging Face `Major-TOM/Core-VIIRS-Nighttime-Light` | the tile's size, 1 request | not measured |
|
| 87 |
+
| AlphaEarth preview (`aef.js`): each tile's 160 m level, 64 bands, source.coop `tge-labs/aef` | ~10 MB, 1 request per tile | ~4.5 s |
|
| 88 |
+
| AlphaEarth 10 m PCA proxy (`aef.js`): 8 bands at 10 m, the tiles under the window | not measured | ~15 s |
|
| 89 |
|
| 90 |
Reading each dataset's `metadata.parquet` from Hugging Face instead was measured first: S2L2A's has one row group
|
| 91 |
per image file (4,492), so its positions took 462 s in 4,492 requests; S1RTC's and DEM's have two row groups, so
|
| 92 |
one sample's row cost 23-27 MB. Hence the footprint files.
|
| 93 |
|
| 94 |
+
The AlphaEarth figures are from the comments in `aef.js` and the flow above, not a fresh measurement.
|
| 95 |
+
|
| 96 |
+
## Private Space: every local module is imported by app.js
|
| 97 |
+
|
| 98 |
+
On the private Space the app runs in an iframe, where the login cookie is not sent. A file is served only if its
|
| 99 |
+
URL or its referer carries the `?__sign=` token. Only `index.html` and `app.js` carry it, so a module imported only
|
| 100 |
+
by another module gets a 401 and the app does not start. Every local module is therefore imported directly by
|
| 101 |
+
`app.js` (`app.js:18-29`; `zstd.js` is imported there as well as by `viirs.js` and `aef.js`).
|
| 102 |
+
`tests/test_imports.py` checks this without the network.
|
| 103 |
+
|
| 104 |
## Run locally
|
| 105 |
|
| 106 |
```
|
|
|
|
| 110 |
|
| 111 |
The page needs a server that answers HTTP range requests (the tests' server does; `python -m http.server` does not).
|
| 112 |
|
| 113 |
+
## Tests
|
| 114 |
+
|
| 115 |
+
| File | What it checks | Needs |
|
| 116 |
+
|---|---|---|
|
| 117 |
+
| `tests/test_imports.py` | every `./x.js` imported by another module is also imported by `app.js` | nothing (`pytest tests/test_imports.py`) |
|
| 118 |
+
| `tests/test_app.py` | headless Chromium (Playwright) at Loch Lomond (624U_29L): thumbnails, VIIRS, AlphaEarth PCA, cursor, background loads, full window, image on the map, Core-Weather, bands and composites, S1 and DEM views, downloads, loading flag (13 tests) | network, `footprints/` built (skipped otherwise) |
|
| 119 |
+
|
| 120 |
+
`pytest -m network` runs `test_app.py` only; `pytest` runs both.
|
| 121 |
+
|
| 122 |
## Credit
|
| 123 |
|
| 124 |
Developed by [Mikolaj Czerkawski](https://github.com/mikonvergence), [Asterisk Labs](https://asterisk.coop), as part of
|