first commit

README.md

@@ -16,7 +16,7 @@ sdk: static
 sdk_version: "latest"          
 pinned: false  
 ---
-Last update: 25-04-2025
+Last update: 22-05-2025
 
 <img src="https://elias-ai.eu/wp-content/uploads/2023/09/elias_logo_big-1.png" alt="elias_logo" style="width:15%; display: inline-block; margin-right: 150px;">
 <img src="https://elias-ai.eu/wp-content/uploads/2024/01/EN_FundedbytheEU_RGB_WHITE-Outline-1.png" alt="eu_logo" style="width:20%; display: inline-block;">
@@ -44,11 +44,12 @@ Last update: 25-04-2025
 
 ## **Benchmark Results**
 
-| **Model** | **MRE A1 (%)** | **MRE A2 (%)** | **MRE B1 (%)** | **MRE B2 (%)** | **Score** | **Rank** |
-|-----------|---------------|---------------|---------------|---------------|----------|--------|
-| test2 | 0.203 | 4.579 | 0.657 | 3.965 | 1.000 | 1° |
-| test1 | 0.545 | 7.693 | 0.816 | 7.877 | 2.175 | 2° |
-| baseline | 0.998 | 12.604 | 1.065 | 7.072 | 2.825 | 3° |
+| **Model** | **MRE A1 (%)** | **MRE A2 (%)** | **MRE B1 (%)** | **MRE B2 (%)** | **Score** | **Runtime** | **Rank** |
+|-----------|---------------|---------------|---------------|---------------|----------|----------|--------|
+| Hugo | 0.203 | 4.579 | 0.657 | 3.965 | 1.000 | 1.065 | 1° |
+| Krtek | 0.545 | 7.693 | 0.816 | 7.877 | 2.175 | 0.526 | 2° |
+| Jasdeep | 0.827 | 28.894 | 1.011 | 35.625 | 3.350 | 1.252 | 3° |
+| baseline | 0.998 | 12.604 | 1.065 | 7.072 | 3.475 | 2.400 | 4° |
 
 ## **Introduction**
 
@@ -74,7 +75,7 @@ broader exploration of emission pathways aligned with sustainability goals.
 
 Participants in this challenge will develop emulators trained on provided datasets to predict spectral magnitudes (atmospheric transmittances and reflectances)
 based on input atmospheric and geometric conditions. The challenge is structured around three main tasks: (1) training ML models
-using predefined datasets, (2) predicting outputs for given test conditions, and (3) evaluating emulator performance based on accuracy and runtime.
+using predefined datasets, (2) predicting outputs for given test conditions, and (3) evaluating emulator performance based on accuracy.
 
 ### **Proposed Experiments**
 
@@ -179,8 +180,8 @@ wavelengths <- h5read(file_path, "wvl")
 Xtest <- as.matrix(read.table(file_path, sep = ",", header = TRUE))
 ```
 
-All data will be shared through a this [huggingface]. After the challenge finishes, participants will also have access to the evaluation scripts on
-[this GitLab](http://to_be_prepared) to ensure transparency and reproducibility.
+All data will be shared through a this [repository](ttps://huggingface.co/datasets/isp-uv-es/rtm_emulation/tree/main). After the challenge finishes, participants
+will also have access to the evaluation scripts on [this GitLab](http://to_be_prepared) to ensure transparency and reproducibility.
 
 
 ## **Evaluation methodology**
@@ -214,9 +215,10 @@ for 1st place, they will both receive rank 1, and the next participant will be r
 information that would allow them to fine-tune their models. The final results and ranking evaluated with all the validation data will be provided and the end-date of the challenge.  
 
 ### **Computational efficiency**
-Participants must report the runtime required to generate predictions across different emulator configurations. To facilitate fair comparisons, they should also provide
-a report with hardware specifications, including: CPU, Parallelization settings (e.g., multi-threading, GPU acceleration), RAM availability. 
-Additionally, participants should report key model characteristics, such as the number of operations required for a single prediction and the number of trainable
+Participants must report the runtime required to generate predictions across different emulator configurations. The average runtime of all scenario-track combinations
+will be calculated and reported in the table. **Runtime won't be taken into account for the final ranking**. After the competition ends, and to facilitate fair
+comparisons, participants will be requested to provide a report with hardware specifications, including: CPU, Parallelization settings (e.g., multi-threading, GPU
+acceleration), RAM availability. Additionally, participants should report key model characteristics, such as the number of operations required for a single prediction and the number of trainable
 parameters in their ML models.
 
 All evaluation scripts will be publicly available on GitLab and Huggingface to ensure fairness, trustworthiness, and transparency.
@@ -233,9 +235,8 @@ All evaluation scripts will be publicly available on GitLab and Huggingface to e
   Note that only the **`LUTdata`** matrix (i.e., the predictions) are needed. A baseline example of this file is available for participants (`baseline_Sn.h5`).
   We encourage participants to compress their hdf5 files using the deflate option.
 
-- Each prediction file must be stored in`predictions` subfolder within the corresponding
-  scenario folder (e.g., (e.g. `/scenarioA/predictions`). The prediction files should be named using the emulator/model name followed by the scenario-track ID
- (e.g. `/scenarioA/predictions/mymodel_A1.h5`). A global attributed named `runtime`must be included to report the
+- Each prediction file must be stored in the `results` folder in this repository. The prediction files should be named using the emulator/model name followed by
+  the scenario-track ID (e.g. `/results/mymodel_A1.h5`). A global attributed named `runtime` must be included to report the
   computational efficiency of your model (value expressed in seconds).
   Note that all predictions for different scenario-tracks should be stored in separate files.
 
@@ -248,7 +249,8 @@ All evaluation scripts will be publicly available on GitLab and Huggingface to e
   "affiliations": ["affiliation1", "affiliation2"],
   "description": "A brief description of the emulator",
   "url": "[OPTIONAL] URL to the model repository if it is open-source",
-  "doi": "DOI to the model publication (if available)"
+  "doi": "DOI to the model publication (if available)",
+  "email": <main_contact_email>
 }
 ```
 
@@ -257,7 +259,7 @@ All evaluation scripts will be publicly available on GitLab and Huggingface to e
 - After the deadline, teams will be contacted with their evaluation results. If any issues are identified, theams will have up to two 
 weeks to provide the necessary corrections.
 
-- Questions and discussions will be held in the discussion section of this [repository](https://huggingface.co/isp-uv-es/rtm_emulation/discussions).
+- In case of **problems with the pull request** or incorrect validity of the submitted files, all discussions will be held in the [discussion board](https://huggingface.co/isp-uv-es/rtm_emulation/discussions).
 
 - After all the participants have provided the necessary corrections, the results will be published in the discussion section of this repository.