validate downsample rate before submit

README.md

@@ -95,8 +95,8 @@ To make a submission to the [leaderboard](https://superbbenchmark.org/leaderboar
     ```python
     upstream = UpstreamExpert(ckpt="./model.pt")
     ```
-    
-    Note that we only install `torch` package so far by following the above steps. If your model needs more packages, you can modify the `requirement.txt` to meet your need and install them inside the current conda environment. We will install the packages you list in the `requirement.txt` before initializing the upstream model.
+
+    ***Package Dependency:*** Note that we only install `torch` package so far by following the above steps. If your model needs more packages, you can modify the `requirement.txt` to meet your need and install them inside the current conda environment. We will install the packages you list in the `requirement.txt` before initializing the upstream model.
 
 2. Validate the upstream model's interface meets the requirements in the [challenge policy](https://superbbenchmark.org/challenge#Upstream-Specification). If everything is correct, you should see the following message: "All submission files validated! Now you can make a submission."
 

{{cookiecutter.repo_name}}/README.md

@@ -95,8 +95,8 @@ To make a submission to the [leaderboard](https://superbbenchmark.org/leaderboar
     ```python
     upstream = UpstreamExpert(ckpt="./model.pt")
     ```
-    
-    Note that we only install `torch` package so far by following the above steps. If your model needs more packages, you can modify the `requirement.txt` to meet your need and install them inside the current conda environment. We will install the packages you list in the `requirement.txt` before initializing the upstream model.
+
+    ***Package Dependency:*** Note that we only install `torch` package so far by following the above steps. If your model needs more packages, you can modify the `requirement.txt` to meet your need and install them inside the current conda environment. We will install the packages you list in the `requirement.txt` before initializing the upstream model.
 
 2. Validate the upstream model's interface meets the requirements in the [challenge policy](https://superbbenchmark.org/challenge#Upstream-Specification). If everything is correct, you should see the following message: "All submission files validated! Now you can make a submission."
 

{{cookiecutter.repo_name}}/cli.py

@@ -20,7 +20,8 @@ def validate():
 
     try:
         upstream = UpstreamExpert(ckpt="model.pt")
-        wavs = [torch.rand(round(SAMPLE_RATE * sec)) for sec in SECONDS]
+        samples = [round(SAMPLE_RATE * sec) for sec in SECONDS]
+        wavs = [torch.rand(sample) for sample in samples]
         results = upstream(wavs)
 
         assert isinstance(results, dict)
@@ -34,9 +35,9 @@ def validate():
                 assert state.dim() == 3, "(batch_size, max_sequence_length_of_batch, hidden_size)"
                 assert state.shape == hidden_states[0].shape
 
-        for task in tasks:
             downsample_rate = upstream.get_downsample_rates(task)
             assert isinstance(downsample_rate, int)
+            assert abs(round(max(samples) / downsample_rate) - hidden_states[0].size(1)) < 5, "wrong downsample rate"
 
     except:
         print("Please check the Upstream Specification on https://superbbenchmark.org/challenge")