Update patch-8 checkpoint to chronological split run

.gitattributes

@@ -35,3 +35,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 assets/solaris_small_val0_prediction.png filter=lfs diff=lfs merge=lfs -text
 assets/solaris_small_patch8_val0_prediction.png filter=lfs diff=lfs merge=lfs -text
+assets/solaris_small_patch8_test0_prediction.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -17,21 +17,22 @@ datasets:
 
 This repository contains a Solaris-Small checkpoint trained for 12-hour multi-wavelength solar forecasting, following the Solaris pretraining setup from [Solaris: A Foundation Model of the Sun](https://arxiv.org/abs/2411.16339).
 
-This run uses patch size 8. The patch-size-4 checkpoint is published separately as `hrrsmjd/solaris_small_patch4`.
+This checkpoint supersedes the earlier patch-size-8 upload. It was trained with a chronological 80/10/10 split, a 24-hour guard band at split boundaries, AdamW weight decay `0.05`, and no EMA weights.
 
 The checkpoint was trained on `hrrsmjd/AIA_12hour_512x512` for 7750 optimizer steps using two history frames (`t-12h`, `t`) to predict all eight pretraining wavelengths at `t+12h`.
 
 ## Files
 
-- `solaris_small_patch8_model_state_dict.pt`: reusable PyTorch checkpoint containing `model_state_dict`, learned normalization coefficients, wavelengths, scale factors, patch size, training step, and final training loss.
+- `solaris_small_patch8_model_state_dict.pt`: reusable PyTorch checkpoint containing `model_state_dict`, learned normalization coefficients, wavelengths, scale factors, patch size, seed, split metadata, test metrics, training step, and final training loss.
 - `config.json`: lightweight metadata for reconstructing the model and normalization.
-- `assets/solaris_small_patch8_val0_prediction.png`: example qualitative validation prediction plot.
+- `assets/solaris_small_patch8_test0_prediction.png`: example qualitative test prediction plot.
+- `eval/solaris_pretrain_p8_chronosplit_wd005_noema_test_mse_subset_0941.md`: full chronological test-split raw-scale MSE/RMSE/MAE report.
 
 ## Example Plot
 
-The plot below shows one validation sample with rows for `input t-12h`, `input t`, `target t+12h`, `prediction t+12h`, and `prediction - target` across all eight wavelengths.
+The plot below shows one test sample with rows for `input t-12h`, `input t`, `target t+12h`, `prediction t+12h`, and `prediction - target` across all eight wavelengths.
 
-![Solaris Small Patch 8 validation prediction](assets/solaris_small_patch8_val0_prediction.png)
+![Solaris Small Patch 8 test prediction](assets/solaris_small_patch8_test0_prediction.png)
 
 ## Model Details
 
@@ -43,7 +44,12 @@ The plot below shows one validation sample with rows for `input t-12h`, `input t
 - Output wavelengths: `0094`, `0131`, `0171`, `0193`, `0211`, `0304`, `0335`, `1600`
 - Training dataset: `hrrsmjd/AIA_12hour_512x512`
 - Training target: 12-hour forecast
+- Split scheme: chronological 80/10/10 over valid samples with a 24-hour boundary guard band
+- Split counts: train `7541`, validation `939`, test `941`
 - Training budget: 7750 optimizer steps, batch size 8, gradient accumulation 4
+- Optimizer: AdamW, `lr=5e-4`, cosine decay to `5e-5`, `weight_decay=0.05`, betas `(0.9, 0.95)`
+- EMA: disabled
+- Seed: `42`
 
 ## Loading
 
@@ -68,20 +74,34 @@ norm_coeff_2 = checkpoint["norm_coeff_2"]
 
 Inputs should be normalized with the Solaris transform used during training. Model outputs are normalized intensities; multiply by the per-wavelength scale factors before comparing to raw-intensity targets.
 
+## Test Metrics
+
+Metrics below use all 941 chronological test samples and are computed on the raw intensity scale.
+
+| Wavelength (A) | MSE | RMSE | MAE |
+|---:|---:|---:|---:|
+| 0094 | 7.08928 | 2.66257 | 0.310692 |
+| 0131 | 147.528 | 12.1461 | 1.44793 |
+| 0171 | 13730.9 | 117.179 | 52.9684 |
+| 0193 | 22906.6 | 151.349 | 62.3088 |
+| 0211 | 7752.18 | 88.0464 | 35.0413 |
+| 0304 | 1556.87 | 39.4572 | 14.2213 |
+| 0335 | 48.0333 | 6.9306 | 2.0811 |
+| 1600 | 145.126 | 12.0468 | 7.3929 |
+| **Mean** | **5786.78** | **53.7272** | **21.9716** |
+
 ## Training Notes
 
 Scale factors were computed as half the average per-image maximum over unique train-split timestamps:
 
 ```text
-[58.224720422037755, 216.21549287451052, 1616.446579054541, 2551.0149615718674, 1190.0182024885178, 887.1800787601859, 112.33733897339224, 266.61844876445224]
+[57.944149103084534, 214.99738922760267, 1590.2998402078304, 2397.489401917806, 1080.261734048243, 830.778793198845, 104.45557294825853, 274.65685334356664]
 ```
 
 The final logged training-batch metrics at step 7750 were:
 
 ```text
-weighted MAE: 0.007735
-mean raw RMSE: 20.364
-per-wavelength raw RMSE: [0.667, 2.239, 50.45, 56.747, 24.077, 18.78, 1.472, 8.486]
+weighted MAE: 0.007597
+mean raw RMSE: 19.576
+per-wavelength raw RMSE: [1.210, 4.034, 49.249, 53.399, 20.770, 17.612, 1.653, 8.684]
 ```
-
-These are training-batch diagnostics, not the paper's full validation/test Table 2 metrics.

config.json

@@ -5,8 +5,21 @@
   "paper": "https://arxiv.org/abs/2411.16339",
   "dataset": "hrrsmjd/AIA_12hour_512x512",
   "training_steps": 7750,
-  "loss": 0.007734995801001787,
+  "loss": 0.007597479969263077,
   "patch_size": 8,
+  "seed": 42,
+  "split_scheme": "chronological_valid_samples_80_10_10_guard_24h_v1",
+  "split_counts": {
+    "train": 7541,
+    "val": 939,
+    "test": 941
+  },
+  "weight_decay": 0.05,
+  "betas": [
+    0.9,
+    0.95
+  ],
+  "use_ema": false,
   "embed_dim": 256,
   "encoder_depths": [
     2,
@@ -30,17 +43,29 @@
     "1600"
   ],
   "scale_factors": [
-    58.224720422037755,
-    216.21549287451052,
-    1616.446579054541,
-    2551.0149615718674,
-    1190.0182024885178,
-    887.1800787601859,
-    112.33733897339224,
-    266.61844876445224
+    57.944149103084534,
+    214.99738922760267,
+    1590.2998402078304,
+    2397.489401917806,
+    1080.261734048243,
+    830.778793198845,
+    104.45557294825853,
+    274.65685334356664
   ],
-  "norm_coeff_1": 0.5428254008293152,
-  "norm_coeff_2": 0.47638431191444397,
+  "norm_coeff_1": 0.525581419467926,
+  "norm_coeff_2": 0.4680934548377991,
   "checkpoint_file": "solaris_small_patch8_model_state_dict.pt",
-  "prediction_plot": "assets/solaris_small_patch8_val0_prediction.png"
+  "prediction_plot": "assets/solaris_small_patch8_test0_prediction.png",
+  "test_metrics_file": "eval/solaris_pretrain_p8_chronosplit_wd005_noema_test_mse_subset_0941.json",
+  "test_mean_rmse_raw": 53.72720532801895,
+  "test_rmse_raw": [
+    2.6625704727109896,
+    12.146093098231086,
+    117.17874406447952,
+    151.34917590125093,
+    88.04643830752812,
+    39.457199820072276,
+    6.930603323634012,
+    12.046817636244633
+  ]
 }

eval/solaris_pretrain_p8_chronosplit_wd005_noema_test_mse_subset_0941.json

@@ -0,0 +1,997 @@
+{
+  "run_name": "solaris_pretrain_p8_chronosplit_wd005_noema",
+  "checkpoint": "/vol/checkpoints/solaris_pretrain_p8_chronosplit_wd005_noema_final_step_07750.pt",
+  "use_ema": false,
+  "data_set": "test",
+  "split_size": 941,
+  "sample_count": 941,
+  "sample_strategy": "evenly_spaced",
+  "indices": [
+    0,
+    1,
+    2,
+    3,
+    4,
+    5,
+    6,
+    7,
+    8,
+    9,
+    10,
+    11,
+    12,
+    13,
+    14,
+    15,
+    16,
+    17,
+    18,
+    19,
+    20,
+    21,
+    22,
+    23,
+    24,
+    25,
+    26,
+    27,
+    28,
+    29,
+    30,
+    31,
+    32,
+    33,
+    34,
+    35,
+    36,
+    37,
+    38,
+    39,
+    40,
+    41,
+    42,
+    43,
+    44,
+    45,
+    46,
+    47,
+    48,
+    49,
+    50,
+    51,
+    52,
+    53,
+    54,
+    55,
+    56,
+    57,
+    58,
+    59,
+    60,
+    61,
+    62,
+    63,
+    64,
+    65,
+    66,
+    67,
+    68,
+    69,
+    70,
+    71,
+    72,
+    73,
+    74,
+    75,
+    76,
+    77,
+    78,
+    79,
+    80,
+    81,
+    82,
+    83,
+    84,
+    85,
+    86,
+    87,
+    88,
+    89,
+    90,
+    91,
+    92,
+    93,
+    94,
+    95,
+    96,
+    97,
+    98,
+    99,
+    100,
+    101,
+    102,
+    103,
+    104,
+    105,
+    106,
+    107,
+    108,
+    109,
+    110,
+    111,
+    112,
+    113,
+    114,
+    115,
+    116,
+    117,
+    118,
+    119,
+    120,
+    121,
+    122,
+    123,
+    124,
+    125,
+    126,
+    127,
+    128,
+    129,
+    130,
+    131,
+    132,
+    133,
+    134,
+    135,
+    136,
+    137,
+    138,
+    139,
+    140,
+    141,
+    142,
+    143,
+    144,
+    145,
+    146,
+    147,
+    148,
+    149,
+    150,
+    151,
+    152,
+    153,
+    154,
+    155,
+    156,
+    157,
+    158,
+    159,
+    160,
+    161,
+    162,
+    163,
+    164,
+    165,
+    166,
+    167,
+    168,
+    169,
+    170,
+    171,
+    172,
+    173,
+    174,
+    175,
+    176,
+    177,
+    178,
+    179,
+    180,
+    181,
+    182,
+    183,
+    184,
+    185,
+    186,
+    187,
+    188,
+    189,
+    190,
+    191,
+    192,
+    193,
+    194,
+    195,
+    196,
+    197,
+    198,
+    199,
+    200,
+    201,
+    202,
+    203,
+    204,
+    205,
+    206,
+    207,
+    208,
+    209,
+    210,
+    211,
+    212,
+    213,
+    214,
+    215,
+    216,
+    217,
+    218,
+    219,
+    220,
+    221,
+    222,
+    223,
+    224,
+    225,
+    226,
+    227,
+    228,
+    229,
+    230,
+    231,
+    232,
+    233,
+    234,
+    235,
+    236,
+    237,
+    238,
+    239,
+    240,
+    241,
+    242,
+    243,
+    244,
+    245,
+    246,
+    247,
+    248,
+    249,
+    250,
+    251,
+    252,
+    253,
+    254,
+    255,
+    256,
+    257,
+    258,
+    259,
+    260,
+    261,
+    262,
+    263,
+    264,
+    265,
+    266,
+    267,
+    268,
+    269,
+    270,
+    271,
+    272,
+    273,
+    274,
+    275,
+    276,
+    277,
+    278,
+    279,
+    280,
+    281,
+    282,
+    283,
+    284,
+    285,
+    286,
+    287,
+    288,
+    289,
+    290,
+    291,
+    292,
+    293,
+    294,
+    295,
+    296,
+    297,
+    298,
+    299,
+    300,
+    301,
+    302,
+    303,
+    304,
+    305,
+    306,
+    307,
+    308,
+    309,
+    310,
+    311,
+    312,
+    313,
+    314,
+    315,
+    316,
+    317,
+    318,
+    319,
+    320,
+    321,
+    322,
+    323,
+    324,
+    325,
+    326,
+    327,
+    328,
+    329,
+    330,
+    331,
+    332,
+    333,
+    334,
+    335,
+    336,
+    337,
+    338,
+    339,
+    340,
+    341,
+    342,
+    343,
+    344,
+    345,
+    346,
+    347,
+    348,
+    349,
+    350,
+    351,
+    352,
+    353,
+    354,
+    355,
+    356,
+    357,
+    358,
+    359,
+    360,
+    361,
+    362,
+    363,
+    364,
+    365,
+    366,
+    367,
+    368,
+    369,
+    370,
+    371,
+    372,
+    373,
+    374,
+    375,
+    376,
+    377,
+    378,
+    379,
+    380,
+    381,
+    382,
+    383,
+    384,
+    385,
+    386,
+    387,
+    388,
+    389,
+    390,
+    391,
+    392,
+    393,
+    394,
+    395,
+    396,
+    397,
+    398,
+    399,
+    400,
+    401,
+    402,
+    403,
+    404,
+    405,
+    406,
+    407,
+    408,
+    409,
+    410,
+    411,
+    412,
+    413,
+    414,
+    415,
+    416,
+    417,
+    418,
+    419,
+    420,
+    421,
+    422,
+    423,
+    424,
+    425,
+    426,
+    427,
+    428,
+    429,
+    430,
+    431,
+    432,
+    433,
+    434,
+    435,
+    436,
+    437,
+    438,
+    439,
+    440,
+    441,
+    442,
+    443,
+    444,
+    445,
+    446,
+    447,
+    448,
+    449,
+    450,
+    451,
+    452,
+    453,
+    454,
+    455,
+    456,
+    457,
+    458,
+    459,
+    460,
+    461,
+    462,
+    463,
+    464,
+    465,
+    466,
+    467,
+    468,
+    469,
+    470,
+    471,
+    472,
+    473,
+    474,
+    475,
+    476,
+    477,
+    478,
+    479,
+    480,
+    481,
+    482,
+    483,
+    484,
+    485,
+    486,
+    487,
+    488,
+    489,
+    490,
+    491,
+    492,
+    493,
+    494,
+    495,
+    496,
+    497,
+    498,
+    499,
+    500,
+    501,
+    502,
+    503,
+    504,
+    505,
+    506,
+    507,
+    508,
+    509,
+    510,
+    511,
+    512,
+    513,
+    514,
+    515,
+    516,
+    517,
+    518,
+    519,
+    520,
+    521,
+    522,
+    523,
+    524,
+    525,
+    526,
+    527,
+    528,
+    529,
+    530,
+    531,
+    532,
+    533,
+    534,
+    535,
+    536,
+    537,
+    538,
+    539,
+    540,
+    541,
+    542,
+    543,
+    544,
+    545,
+    546,
+    547,
+    548,
+    549,
+    550,
+    551,
+    552,
+    553,
+    554,
+    555,
+    556,
+    557,
+    558,
+    559,
+    560,
+    561,
+    562,
+    563,
+    564,
+    565,
+    566,
+    567,
+    568,
+    569,
+    570,
+    571,
+    572,
+    573,
+    574,
+    575,
+    576,
+    577,
+    578,
+    579,
+    580,
+    581,
+    582,
+    583,
+    584,
+    585,
+    586,
+    587,
+    588,
+    589,
+    590,
+    591,
+    592,
+    593,
+    594,
+    595,
+    596,
+    597,
+    598,
+    599,
+    600,
+    601,
+    602,
+    603,
+    604,
+    605,
+    606,
+    607,
+    608,
+    609,
+    610,
+    611,
+    612,
+    613,
+    614,
+    615,
+    616,
+    617,
+    618,
+    619,
+    620,
+    621,
+    622,
+    623,
+    624,
+    625,
+    626,
+    627,
+    628,
+    629,
+    630,
+    631,
+    632,
+    633,
+    634,
+    635,
+    636,
+    637,
+    638,
+    639,
+    640,
+    641,
+    642,
+    643,
+    644,
+    645,
+    646,
+    647,
+    648,
+    649,
+    650,
+    651,
+    652,
+    653,
+    654,
+    655,
+    656,
+    657,
+    658,
+    659,
+    660,
+    661,
+    662,
+    663,
+    664,
+    665,
+    666,
+    667,
+    668,
+    669,
+    670,
+    671,
+    672,
+    673,
+    674,
+    675,
+    676,
+    677,
+    678,
+    679,
+    680,
+    681,
+    682,
+    683,
+    684,
+    685,
+    686,
+    687,
+    688,
+    689,
+    690,
+    691,
+    692,
+    693,
+    694,
+    695,
+    696,
+    697,
+    698,
+    699,
+    700,
+    701,
+    702,
+    703,
+    704,
+    705,
+    706,
+    707,
+    708,
+    709,
+    710,
+    711,
+    712,
+    713,
+    714,
+    715,
+    716,
+    717,
+    718,
+    719,
+    720,
+    721,
+    722,
+    723,
+    724,
+    725,
+    726,
+    727,
+    728,
+    729,
+    730,
+    731,
+    732,
+    733,
+    734,
+    735,
+    736,
+    737,
+    738,
+    739,
+    740,
+    741,
+    742,
+    743,
+    744,
+    745,
+    746,
+    747,
+    748,
+    749,
+    750,
+    751,
+    752,
+    753,
+    754,
+    755,
+    756,
+    757,
+    758,
+    759,
+    760,
+    761,
+    762,
+    763,
+    764,
+    765,
+    766,
+    767,
+    768,
+    769,
+    770,
+    771,
+    772,
+    773,
+    774,
+    775,
+    776,
+    777,
+    778,
+    779,
+    780,
+    781,
+    782,
+    783,
+    784,
+    785,
+    786,
+    787,
+    788,
+    789,
+    790,
+    791,
+    792,
+    793,
+    794,
+    795,
+    796,
+    797,
+    798,
+    799,
+    800,
+    801,
+    802,
+    803,
+    804,
+    805,
+    806,
+    807,
+    808,
+    809,
+    810,
+    811,
+    812,
+    813,
+    814,
+    815,
+    816,
+    817,
+    818,
+    819,
+    820,
+    821,
+    822,
+    823,
+    824,
+    825,
+    826,
+    827,
+    828,
+    829,
+    830,
+    831,
+    832,
+    833,
+    834,
+    835,
+    836,
+    837,
+    838,
+    839,
+    840,
+    841,
+    842,
+    843,
+    844,
+    845,
+    846,
+    847,
+    848,
+    849,
+    850,
+    851,
+    852,
+    853,
+    854,
+    855,
+    856,
+    857,
+    858,
+    859,
+    860,
+    861,
+    862,
+    863,
+    864,
+    865,
+    866,
+    867,
+    868,
+    869,
+    870,
+    871,
+    872,
+    873,
+    874,
+    875,
+    876,
+    877,
+    878,
+    879,
+    880,
+    881,
+    882,
+    883,
+    884,
+    885,
+    886,
+    887,
+    888,
+    889,
+    890,
+    891,
+    892,
+    893,
+    894,
+    895,
+    896,
+    897,
+    898,
+    899,
+    900,
+    901,
+    902,
+    903,
+    904,
+    905,
+    906,
+    907,
+    908,
+    909,
+    910,
+    911,
+    912,
+    913,
+    914,
+    915,
+    916,
+    917,
+    918,
+    919,
+    920,
+    921,
+    922,
+    923,
+    924,
+    925,
+    926,
+    927,
+    928,
+    929,
+    930,
+    931,
+    932,
+    933,
+    934,
+    935,
+    936,
+    937,
+    938,
+    939,
+    940
+  ],
+  "first_timestamp": "2022-09-05T12:00:00",
+  "last_timestamp": "2023-12-31T00:00:00",
+  "wavelengths": [
+    "0094",
+    "0131",
+    "0171",
+    "0193",
+    "0211",
+    "0304",
+    "0335",
+    "1600"
+  ],
+  "mse_raw": [
+    7.089281522152423,
+    147.5275775508968,
+    13730.858060528793,
+    22906.573045987792,
+    7752.175298641355,
+    1556.8706176411115,
+    48.033262429566804,
+    145.12581516093474
+  ],
+  "rmse_raw": [
+    2.6625704727109896,
+    12.146093098231086,
+    117.17874406447952,
+    151.34917590125093,
+    88.04643830752812,
+    39.457199820072276,
+    6.930603323634012,
+    12.046817636244633
+  ],
+  "mae_raw": [
+    0.3106918638967333,
+    1.4479309443932828,
+    52.96843658281969,
+    62.3087988093066,
+    35.04134377748435,
+    14.221309963989963,
+    2.081098432996739,
+    7.392904130125146
+  ],
+  "mean_mse_raw": 5786.781619932825,
+  "mean_rmse_raw": 53.72720532801895,
+  "mean_mae_raw": 21.971564313126564
+}

eval/solaris_pretrain_p8_chronosplit_wd005_noema_test_mse_subset_0941.md

@@ -0,0 +1,20 @@
+# Solaris-S test subset raw-scale MSE
+
+- Checkpoint: `solaris_pretrain_p8_chronosplit_wd005_noema_final_step_07750.pt`
+- EMA weights: `False`
+- Split size: 941
+- Evaluated samples: 941
+- Sampling: evenly spaced indices across the split
+- Timestamp range in evaluated batches: `2022-09-05T12:00:00` to `2023-12-31T00:00:00`
+
+| Wavelength (A) | MSE (raw intensity^2) | RMSE (raw intensity) | MAE (raw intensity) |
+|---:|---:|---:|---:|
+| 0094 | 7.08928 | 2.66257 | 0.310692 |
+| 0131 | 147.528 | 12.1461 | 1.44793 |
+| 0171 | 13730.9 | 117.179 | 52.9684 |
+| 0193 | 22906.6 | 151.349 | 62.3088 |
+| 0211 | 7752.18 | 88.0464 | 35.0413 |
+| 0304 | 1556.87 | 39.4572 | 14.2213 |
+| 0335 | 48.0333 | 6.9306 | 2.0811 |
+| 1600 | 145.126 | 12.0468 | 7.3929 |
+| **Mean** | **5786.78** | **53.7272** | **21.9716** |

solaris_small_patch8_model_state_dict.pt

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:285e9db8db96e792cef2afc5ef4ab019ae64ef1060f9215ad1e3d2636e0c3801
-size 447240881
+oid sha256:854e86e705733bda6bc6f6ce58e156f63c4a1a197135b02355a0e85c4f58f5ec
+size 447241683