|
|
--- |
|
|
language: en |
|
|
license: mit |
|
|
tags: |
|
|
- pytorch |
|
|
--- |
|
|
|
|
|
<h1 align="center"> |
|
|
FISHER |
|
|
</h1> |
|
|
|
|
|
<div align="center"> |
|
|
<img src="assets/rmis_curve.png" alt="Model Performances on the RMIS Benchmark" style="width:80%; max-width: 1000px"> |
|
|
</div> |
|
|
|
|
|
|
|
|
## Introduction |
|
|
|
|
|
<div align="center"> |
|
|
<img src="assets/model_pipe.png" alt="Model Performances on the RMIS Benchmark" style="width:100%; max-width: 1500px"> |
|
|
</div> |
|
|
|
|
|
FISHER is a **F**oundation model for **I**ndustrial **S**ignal compre**HE**nsive **R**epresentation, which models heterogeneous industrial signals (sound, vibration, voltage, etc.) in a unified manner. FISHER accepts arbitrary sampling rates and models the increment of sampling rate as the concatenation of sub-band information, which first splits a STFT spectrogram into sub-bands before processsing it by the ViT encoder. FISHER is trained by teacher student EMA self-distillation. |
|
|
|
|
|
To evaluate the model, we develop the RMIS benchmark, which will also be open-sourced in the near future. FISHER achieves the SOTA performances on the RMIS benchmark with much more efficient scaling properties. |
|
|
|
|
|
## Inference |
|
|
|
|
|
Please use the following code to infer the signal representation by FISHER. |
|
|
|
|
|
```python |
|
|
import torch |
|
|
import torchaudio |
|
|
import torch.nn.functional as F |
|
|
from transformers import AutoModel |
|
|
|
|
|
model = AutoModel.from_pretrained('jiangab/FISHER-tiny-0723', trust_remote_code=True) |
|
|
model = model.cuda() |
|
|
model.eval() |
|
|
|
|
|
wav, sr = torchaudio.load('/path/to/local/signal.wav') |
|
|
# You can replace it with your custom loading function for other signals |
|
|
|
|
|
wav = wav - wav.mean() |
|
|
STFT = torchaudio.transforms.Spectrogram( |
|
|
n_fft=25 * sr // 1000, |
|
|
win_length=None, |
|
|
hop_length=10 * sr // 1000, |
|
|
power=1, |
|
|
center=False |
|
|
) |
|
|
spec = torch.log(torch.abs(STFT(wav)) + 1e-10) |
|
|
spec = spec.transpose(-2, -1) # [1, time, freq] |
|
|
spec = (spec + 3.017344307886898) / (2.1531635155379805 * 2) |
|
|
|
|
|
# time-wise cutoff |
|
|
if spec.shape[-2] > 1024: |
|
|
spec = spec[:, :1024] |
|
|
# freq-wise padding |
|
|
if spec.shape[-1] < model.cfg.band_width: |
|
|
spec = F.pad(spec, (0, model.cfg.band_width - spec.shape[-1])) |
|
|
spec = spec.unsqueeze(1).cuda() |
|
|
|
|
|
with torch.no_grad(): |
|
|
# Use autocast for mixed precision inference. You can disable it for full precision. |
|
|
with torch.autocast('cuda'): |
|
|
repre = model.extract_features(spec) |
|
|
print(repre.shape) |
|
|
``` |
|
|
|
|
|
## Acknowledgements |
|
|
|
|
|
FISHER is developed based on [EAT](https://github.com/cwx-worst-one/EAT) and [fairseq](https://github.com/facebookresearch/fairseq). We thank these authors for open-sourcing their works. |
|
|
|
|
|
## Citation |
|
|
|
|
|
If you find FISHER useful, please cite the following paper. |
|
|
|
|
|
```bibtex |
|
|
@article{fan2025fisher, |
|
|
title={FISHER: A Foundation Model for Multi-Modal Industrial Signal Comprehensive Representation}, |
|
|
author={Fan, Pingyi and Jiang, Anbai and Zhang, Shuwei and Lv, Zhiqiang and Han, Bing and Zheng, Xinhu and Liang, Wenrui and Li, Junjie and Zhang, Wei-Qiang and Qian, Yanmin and Chen, Xie and Lu, Cheng and Liu, Jia}, |
|
|
journal={arXiv preprint arXiv:2507.16696}, |
|
|
year={2025} |
|
|
} |
|
|
``` |
