Upload folder using huggingface_hub

README.md

@@ -1,86 +1,86 @@
----
-language: en
-license: mit
-tags:
-- pytorch
----
-
-<h1 align="center">
-  FISHER
-</h1>
-
-<div align="center">
-    <img src="assets/rmis_curve.png" alt="Model Performances on the RMIS Benchmark" style="width:80%; max-width: 1000px">
-</div>
-
-
-## Introduction
-
-<div align="center">
-    <img src="assets/model_pipe.png" alt="Model Performances on the RMIS Benchmark" style="width:100%; max-width: 1500px">
-</div>
-
-FISHER is a **F**oundation model for **I**ndustrial **S**ignal compre**HE**nsive **R**epresentation, which models heterogeneous industrial signals (sound, vibration, voltage, etc.) in a unified manner. FISHER accepts arbitrary sampling rates and models the increment of sampling rate as the concatenation of sub-band information, which first splits a STFT spectrogram into sub-bands before processsing it by the ViT encoder. FISHER is trained by teacher student EMA self-distillation.
-
-To evaluate the model, we develop the RMIS benchmark, which will also be open-sourced in the near future. FISHER achieves the SOTA performances on the RMIS benchmark with much more efficient scaling properties.
-
-## Inference
-
-Please use the following code to infer the signal representation by FISHER.
-
-```python
-import torch
-import torchaudio
-import torch.nn.functional as F
-from transformers import AutoModel
-
-model = AutoModel.from_pretrained('jiangab/FISHER-tiny-0723', trust_remote_code=True)
-model = model.cuda()
-model.eval()
-
-wav, sr = torchaudio.load('/path/to/local/signal.wav')
-# You can replace it with your custom loading function for other signals
-
-wav = wav - wav.mean()
-STFT = torchaudio.transforms.Spectrogram(
-    n_fft=25 * sr // 1000,
-    win_length=None,
-    hop_length=10 * sr // 1000,
-    power=1,
-    center=False
-)
-spec = torch.log(torch.abs(STFT(wav)) + 1e-10)
-spec = spec.transpose(-2, -1)  # [1, time, freq]
-spec = (spec + 3.017344307886898) / (2.1531635155379805 * 2)
-
-# time-wise cutoff
-if spec.shape[-2] > 1024:
-    spec = spec[:, :1024]
-# freq-wise padding
-if spec.shape[-1] < model.cfg.band_width:
-    spec = F.pad(spec, (0, model.cfg.band_width - spec.shape[-1]))
-spec = spec.unsqueeze(1).cuda()
-
-with torch.no_grad():
-    # Use autocast for mixed precision inference. You can disable it for full precision.
-    with torch.autocast('cuda'):
-        repre = model.extract_features(spec)
-print(repre.shape)
-```
-
-## Acknowledgements
-
-FISHER is developed based on [EAT](https://github.com/cwx-worst-one/EAT) and [fairseq](https://github.com/facebookresearch/fairseq). We thank these authors for open-sourcing their works.
-
-## Citation
-
-If you find FISHER useful, please cite the following paper.
-
-```bibtex
-@article{fan2025fisher,
-  title={FISHER: A Foundation Model for Multi-Modal Industrial Signal Comprehensive Representation},
-  author={Fan, Pingyi and Jiang, Anbai and Zhang, Shuwei and Lv, Zhiqiang and Han, Bing and Zheng, Xinhu and Liang, Wenrui and Li, Junjie and Zhang, Wei-Qiang and Qian, Yanmin and Chen, Xie and Lu, Cheng and Liu, Jia},
-  journal={arXiv preprint arXiv:2507.16696},
-  year={2025}
-}
-```
+---
+language: en
+license: mit
+tags:
+- pytorch
+---
+
+<h1 align="center">
+  FISHER
+</h1>
+
+<div align="center">
+    <img src="assets/rmis_curve.png" alt="Model Performances on the RMIS Benchmark" style="width:85%; max-width: 2000px">
+</div>
+
+
+## Introduction
+
+<div align="center">
+    <img src="assets/model_pipe.png" alt="Model Performances on the RMIS Benchmark" style="width:90%; max-width: 1500px">
+</div>
+
+FISHER is a **F**oundation model for **I**ndustrial **S**ignal compre**HE**nsive **R**epresentation, which models heterogeneous industrial signals (sound, vibration, voltage, etc.) in a unified manner. FISHER accepts arbitrary sampling rates and models the increment of sampling rate as the concatenation of sub-band information, which first splits a STFT spectrogram into sub-bands before processsing it by the ViT encoder. FISHER is trained by teacher student EMA self-distillation.
+
+To evaluate the model, we develop the RMIS benchmark, which will also be open-sourced in the near future. FISHER achieves the SOTA performances on the RMIS benchmark with much more efficient scaling properties.
+
+## Inference
+
+Please use the following code to infer the signal representation by FISHER.
+
+```python
+import torch
+import torchaudio
+import torch.nn.functional as F
+from transformers import AutoModel
+
+model = AutoModel.from_pretrained('jiangab/FISHER-tiny-0723', trust_remote_code=True)
+model = model.cuda()
+model.eval()
+
+wav, sr = torchaudio.load('/path/to/local/signal.wav')
+# You can replace it with your custom loading function for other signals
+
+wav = wav - wav.mean()
+STFT = torchaudio.transforms.Spectrogram(
+    n_fft=25 * sr // 1000,
+    win_length=None,
+    hop_length=10 * sr // 1000,
+    power=1,
+    center=False
+)
+spec = torch.log(torch.abs(STFT(wav)) + 1e-10)
+spec = spec.transpose(-2, -1)  # [1, time, freq]
+spec = (spec + 3.017344307886898) / (2.1531635155379805 * 2)
+
+# time-wise cutoff
+if spec.shape[-2] > 1024:
+    spec = spec[:, :1024]
+# freq-wise padding
+if spec.shape[-1] < model.cfg.band_width:
+    spec = F.pad(spec, (0, model.cfg.band_width - spec.shape[-1]))
+spec = spec.unsqueeze(1).cuda()
+
+with torch.no_grad():
+    # Use autocast for mixed precision inference. You can disable it for full precision.
+    with torch.autocast('cuda'):
+        repre = model.extract_features(spec)
+print(repre.shape)
+```
+
+## Acknowledgements
+
+FISHER is developed based on [EAT](https://github.com/cwx-worst-one/EAT) and [fairseq](https://github.com/facebookresearch/fairseq). We thank these authors for open-sourcing their works.
+
+## Citation
+
+If you find FISHER useful, please cite the following paper.
+
+```bibtex
+@article{fan2025fisher,
+  title={FISHER: A Foundation Model for Multi-Modal Industrial Signal Comprehensive Representation},
+  author={Fan, Pingyi and Jiang, Anbai and Zhang, Shuwei and Lv, Zhiqiang and Han, Bing and Zheng, Xinhu and Liang, Wenrui and Li, Junjie and Zhang, Wei-Qiang and Qian, Yanmin and Chen, Xie and Lu, Cheng and Liu, Jia},
+  journal={arXiv preprint arXiv:2507.16696},
+  year={2025}
+}
+```

modules.py

@@ -215,19 +215,13 @@ class AltAttention(nn.Module):
         self.num_heads = num_heads
         head_dim = dim // num_heads
         self.scale = qk_scale or head_dim ** -0.5
+        assert cosine_attention is False
 
         self.qkv = nn.Linear(dim, dim * 3, bias=qkv_bias)
         self.attn_drop = nn.Dropout(attn_drop)
         self.proj = nn.Linear(dim, dim)
         self.proj_drop = nn.Dropout(proj_drop)
 
-        self.cosine_attention = cosine_attention
-
-        if cosine_attention:
-            self.logit_scale = nn.Parameter(
-                torch.log(10 * torch.ones((num_heads, 1, 1))), requires_grad=True
-            )
-
     def forward(self, x, padding_mask=None, alibi_bias=None):
         B, N, C = x.shape
         qkv = (
@@ -235,39 +229,23 @@ class AltAttention(nn.Module):
             .reshape(B, N, 3, self.num_heads, C // self.num_heads)
             .permute(2, 0, 3, 1, 4)  # qkv x B x H x L x D
         )
-        q, k, v = (
-            qkv[0],
-            qkv[1],
-            qkv[2],
-        )  # make torchscript happy (cannot use tensor as tuple)
-
-        dtype = q.dtype
-
-        if self.cosine_attention:
-            # cosine attention
-            attn = F.normalize(q, dim=-1) @ F.normalize(k, dim=-1).transpose(-2, -1)
-            logit_scale = torch.clamp(
-                self.logit_scale, max=torch.log(torch.tensor(1.0 / 0.01))
-            ).exp()
-            attn = attn * logit_scale
+        q, k, v = qkv[0], qkv[1], qkv[2]  # (B, H, N, D)
+
+        # key padding mask: True for preserve, False for padding
+        if padding_mask is not None and padding_mask.any():
+            key_padding_mask = ~padding_mask  # (B, N)
         else:
-            q = q * self.scale
-            attn = q @ k.transpose(-2, -1)
+            key_padding_mask = None
 
-        if alibi_bias is not None:
-            attn = attn.type_as(alibi_bias)
-            attn[:, : alibi_bias.size(1)] += alibi_bias
+        # use pytorch SDPA, auto select Flash Attention
+        x = F.scaled_dot_product_attention(
+            q, k, v,
+            attn_mask=key_padding_mask,
+            dropout_p=self.attn_drop.p if self.training else 0.0,
+            scale=self.scale,
+        )  # (B, H, N, D)
 
-        if padding_mask is not None and padding_mask.any():
-            attn = attn.masked_fill(
-                padding_mask.unsqueeze(1).unsqueeze(2).to(torch.bool),
-                float("-inf"),
-            )
-
-        attn = attn.softmax(dim=-1, dtype=torch.float32).to(dtype=dtype)
-        attn = self.attn_drop(attn)
-        x = (attn @ v).transpose(1, 2)  #
-        x = x.reshape(B, N, C)
+        x = x.transpose(1, 2).reshape(B, N, C)
         x = self.proj(x)
         x = self.proj_drop(x)
         return x