Hugging Face's logo

kandinskylab
/
KVAE-Audio

Audio-to-Audio
Diffusers
Safetensors
KVAE-Audio
kvae-audio
vae
audio
Model card Files
xet
 Community
3

Add pipeline tag and paper link

#3
by nielsr HF Staff - opened
base: refs/heads/main
←
from: refs/pr/3
Discussion Files changed
+33
-32
Files changed (1) hide show
  1. README.md +33 -32
README.md CHANGED
@@ -1,9 +1,10 @@
1
  ---
2
  library_name: KVAE-Audio
 
 
3
  tags:
4
  - vae
5
  - audio
6
- license: mit
7
  ---
8
 
9
  <div align="center">
@@ -11,7 +12,7 @@ license: mit
11
  <img src="assets/kvae_audio.png">
12
  </picture>
13
 
14
- <a href="https://github.com/kandinskylab/kvae-audio">GitHub</a> | <a href="https://habr.com/ru/companies/sberbank/articles/1053410/">Habr article</a> | <a href="https://kandinskylab.ai/">Project Page</a> | Technical Report (soon)
15
  </div>
16
 
17
  <h1>KVAE-Audio</h1>
@@ -32,30 +33,30 @@ Generative quality is established under a **fixed generator** — same DiT archi
32
 
33
  ### AudioCaps test set
34
 
35
- | Model           | # Params | Latent dim | CLAP↑     | CE↑       | PQ↑       | FAD (PANNs)↓ | FAD (PASST)↓ | FAD (VGGIsh)↓ |
36
  | --------------- | -------- | ---------- | --------- | --------- | --------- | ------------ | ------------ | ------------- |
37
- | MMAudio 44.1kHz | 427.6M   | 40         | *0,336*   | *3,909*   | *6,192*     | *17,873*       | *195,910*      | 1,364         |
38
- | DACVAE MovieGen | 107.7M   | 128        | 0,313     | 3,772     | 6,167     | 20,558       | 234,312      | 1,700         |
39
- | SAME-L          | 852.1M   | 256        | 0,322     | 3,588     | 5,756     | 18,446       | 240,635      | *1,325*       |
40
- | KVAE-Audio      | 166.9M   | 64         | **0,344** | **3,982** | **6,242** | **15,381**   | **193,760**  | **1,210**     |
41
 
42
  #### Song Describer
43
 
44
- | Model           | # Params | Latent dim | CLAP↑     | CE↑       | PQ↑       | FAD (PANNs)↓ | FAD (PASST)↓ | FAD (VGGIsh)↓ |
45
  | --------------- | -------- | ---------- | --------- | --------- | --------- | ------------ | ------------ | ------------- |
46
- | MMAudio 44.1kHz | 427.6M   | 40         | **0,356** | *7,136*   | *7,707*   | **5,412**    | **158,599**  | **0,356**     |
47
- | DACVAE MovieGen | 107.7M   | 128        | 0,312     | 6,953     | 7,538     | 10,194       | 214,009      | 1,046         |
48
- | SAME-L          | 852.1M   | 256        | *0,345*   | 7,076     | 7,465     | 8,442        | 250,668      | 0,987         |
49
- | KVAE-Audio      | 166.9M   | 64         | 0,339     | **7,216** | **7,929** | *7,971*        | *189,427*      | *0,599*         |
50
 
51
  #### LibriSpeech test-clean
52
 
53
- | Model           | # Params | Latent dim | CLAP↑     | CE↑       | PQ↑       | FAD (PANNs)↓ | FAD (PASST)↓ | FAD (VGGIsh)↓ | WER↓      | CER↓      |
54
  | --------------- | -------- | ---------- | --------- | --------- | --------- | ------------ | ------------ | ------------- | --------- | --------- |
55
- | MMAudio 44.1kHz | 427.6M   | 40         | 0,368     | *5,704*   | 6,629     | 8,305        | **105,931**  | *2,001*       | *0,257*   | *0,593*   |
56
- | DACVAE MovieGen | 107.7M   | 128        | **0,413** | 5,482     | **7,052** | *5,008*      | 210,478      | **1,501**     | 0,911     | 1,048     |
57
- | SAME-L          | 852.1M   | 256        | 0,379     | 4,617     | 5,024     | 10,257       | 301,508      | 2,721         | 0,349     | 0,629     |
58
- | KVAE-Audio      | 166.9M   | 64         | *0,389*   | **5,906** | *6,940*   | **4,677**    | *185,609*    | 2,138         | **0,244** | **0,576** |
59
 
60
  ### Reconstructions
61
 
@@ -63,27 +64,27 @@ Reconstruction is evaluated on open datasets across domains (the released weight
63
 
64
  #### AudioSet eval
65
 
66
- | Model           | # Params | Latent dim | MEL↓      | STFT↓     | Waveform↓ | SI-SDR↑   | SDR↑       | SNR↑       |
67
  | --------------- | -------- | ---------- | --------- | --------- | --------- | --------- | ---------- | ---------- |
68
- | MMAudio 44.1kHz | 427.6M   | 40         | *0,636*   | *1,938*   | 0,106     | -32,080   | -2,682     | -2,686     |
69
- | DACVAE MovieGen | 107.7M   | 128        | 0,669     | 2,275     | 0,029     | 8,384     | 9,421      | 9,416      |
70
- | SAME-L          | 852.1M   | 256        | 0,986     | 2,726     | *0,027*   | **9,586** | **10,347** | **10,339** |
71
- | KVAE-Audio      | 166.9M   | 64         | **0,537** | **1,770** | **0,027** | *9,065*   | *9,920*    | *9,933*    |
72
 
73
  #### MUSDB18-HQ
74
 
75
- | Model           | # Params | Latent dim | MEL↓      | STFT↓     | Waveform↓ | SI-SDR↑    | SDR↑       | SNR↑       |
76
  | --------------- | -------- | ---------- | --------- | --------- | --------- | ---------- | ---------- | ---------- |
77
- | MMAudio 44.1kHz | 427.6M   | 40         | 0,681     | 1,865     | 0,114     | -40,204    | -3,274     | -3,273     |
78
- | DACVAE MovieGen | 107.7M   | 128        | *0,519*   | *1,762*   | 0,024     | 9,688      | 10,046     | 10,047     |
79
- | SAME-L          | 852.1M   | 256        | 0,668     | 1,786     | *0,023*   | *10,278*   | *10,648*   | *10,648*   |
80
- | KVAE-Audio      | 166.9M   | 64         | **0,516** | **1,725** | **0,022** | **10,390** | **10,675** | **10,677** |
81
 
82
  #### EARS
83
 
84
- | Model           | # Params | Latent dim | MEL↓      | STFT↓     | Waveform↓ | SI-SDR↑    | SDR↑       | SNR↑       | PESQ↑     |
85
  | --------------- | -------- | ---------- | --------- | --------- | --------- | ---------- | ---------- | ---------- | --------- |
86
- | MMAudio 44.1kHz | 427.6M   | 40         | 0,616     | 1,395     | 0,030     | -29,947    | -2,728     | -2,697     | 2,424     |
87
- | DACVAE MovieGen | 107.7M   | 128        | **0,453** | **1,310** | *0,006*   | **10,264** | **10,680** | **10,681** | *4,246*   |
88
- | SAME-L          | 852.1M   | 256        | 0,774     | 1,575     | 0,007     | 9,939      | 10,374     | 10,376     | 2,982     |
89
- | KVAE-Audio      | 166.9M   | 64         | *0,463*   | *1,314*   | **0,006** | *9,952*    | *10,377*   | *10,384*   | **4,266** |
 
1
  ---
2
  library_name: KVAE-Audio
3
+ license: mit
4
+ pipeline_tag: audio-to-audio
5
  tags:
6
  - vae
7
  - audio
 
8
  ---
9
 
10
  <div align="center">
 
12
  <img src="assets/kvae_audio.png">
13
  </picture>
14
 
15
+ <a href="https://github.com/kandinskylab/kvae-audio">GitHub</a> | <a href="https://habr.com/ru/companies/sberbank/articles/1053410/">Habr article</a> | <a href="https://kandinskylab.ai/">Project Page</a> | <a href="https://huggingface.co/papers/2608.05798">Paper</a> | Technical Report (soon)
16
  </div>
17
 
18
  <h1>KVAE-Audio</h1>
 
33
 
34
  ### AudioCaps test set
35
 
36
+ | Model | # Params | Latent dim | CLAP↑ | CE↑ | PQ↑ | FAD (PANNs)↓ | FAD (PASST)↓ | FAD (VGGIsh)↓ |
37
  | --------------- | -------- | ---------- | --------- | --------- | --------- | ------------ | ------------ | ------------- |
38
+ | MMAudio 44.1kHz | 427.6M | 40 | *0,336* | *3,909* | *6,192* | *17,873* | *195,910* | 1,364 |
39
+ | DACVAE MovieGen | 107.7M | 128 | 0,313 | 3,772 | 6,167 | 20,558 | 234,312 | 1,700 |
40
+ | SAME-L | 852.1M | 256 | 0,322 | 3,588 | 5,756 | 18,446 | 240,635 | *1,325* |
41
+ | KVAE-Audio | 166.9M | 64 | **0,344** | **3,982** | **6,242** | **15,381** | **193,760** | **1,210** |
42
 
43
  #### Song Describer
44
 
45
+ | Model | # Params | Latent dim | CLAP↑ | CE↑ | PQ↑ | FAD (PANNs)↓ | FAD (PASST)↓ | FAD (VGGIsh)↓ |
46
  | --------------- | -------- | ---------- | --------- | --------- | --------- | ------------ | ------------ | ------------- |
47
+ | MMAudio 44.1kHz | 427.6M | 40 | **0,356** | *7,136* | *7,707* | **5,412** | **158,599** | **0,356** |
48
+ | DACVAE MovieGen | 107.7M | 128 | 0,312 | 6,953 | 7,538 | 10,194 | 214,009 | 1,046 |
49
+ | SAME-L | 852.1M | 256 | *0,345* | 7,076 | 7,465 | 8,442 | 250,668 | 0,987 |
50
+ | KVAE-Audio | 166.9M | 64 | 0,339 | **7,216** | **7,929** | *7,971* | *189,427* | *0,599* |
51
 
52
  #### LibriSpeech test-clean
53
 
54
+ | Model | # Params | Latent dim | CLAP↑ | CE↑ | PQ↑ | FAD (PANNs)↓ | FAD (PASST)↓ | FAD (VGGIsh)↓ | WER↓ | CER↓ |
55
  | --------------- | -------- | ---------- | --------- | --------- | --------- | ------------ | ------------ | ------------- | --------- | --------- |
56
+ | MMAudio 44.1kHz | 427.6M | 40 | 0,368 | *5,704* | 6,629 | 8,305 | **105,931** | *2,001* | *0,257* | *0,593* |
57
+ | DACVAE MovieGen | 107.7M | 128 | **0,413** | 5,482 | **7,052** | *5,008* | 210,478 | **1,501** | 0,911 | 1,048 |
58
+ | SAME-L | 852.1M | 256 | 0,379 | 4,617 | 5,024 | 10,257 | 301,508 | 2,721 | 0,349 | 0,629 |
59
+ | KVAE-Audio | 166.9M | 64 | *0,389* | **5,906** | *6,940* | **4,677** | *185,609* | 2,138 | **0,244** | **0,576** |
60
 
61
  ### Reconstructions
62
 
 
64
 
65
  #### AudioSet eval
66
 
67
+ | Model | # Params | Latent dim | MEL↓ | STFT↓ | Waveform↓ | SI-SDR↑ | SDR↑ | SNR↑ |
68
  | --------------- | -------- | ---------- | --------- | --------- | --------- | --------- | ---------- | ---------- |
69
+ | MMAudio 44.1kHz | 427.6M | 40 | *0,636* | *1,938* | 0,106 | -32,080 | -2,682 | -2,686 |
70
+ | DACVAE MovieGen | 107.7M | 128 | 0,669 | 2,275 | 0,029 | 8,384 | 9,421 | 9,416 |
71
+ | SAME-L | 852.1M | 256 | 0,986 | 2,726 | *0,027* | **9,586** | **10,347** | **10,339** |
72
+ | KVAE-Audio | 166.9M | 64 | **0,537** | **1,770** | **0,027** | *9,065* | *9,920* | *9,933* |
73
 
74
  #### MUSDB18-HQ
75
 
76
+ | Model | # Params | Latent dim | MEL↓ | STFT↓ | Waveform↓ | SI-SDR↑ | SDR↑ | SNR↑ |
77
  | --------------- | -------- | ---------- | --------- | --------- | --------- | ---------- | ---------- | ---------- |
78
+ | MMAudio 44.1kHz | 427.6M | 40 | 0,681 | 1,865 | 0,114 | -40,204 | -3,274 | -3,273 |
79
+ | DACVAE MovieGen | 107.7M | 128 | *0,519* | *1,762* | 0,024 | 9,688 | 10,046 | 10,047 |
80
+ | SAME-L | 852.1M | 256 | 0,668 | 1,786 | *0,023* | *10,278* | *10,648* | *10,648* |
81
+ | KVAE-Audio | 166.9M | 64 | **0,516** | **1,725** | **0,022** | **10,390** | **10,675** | **10,677** |
82
 
83
  #### EARS
84
 
85
+ | Model | # Params | Latent dim | MEL↓ | STFT↓ | Waveform↓ | SI-SDR↑ | SDR↑ | SNR↑ | PESQ↑ |
86
  | --------------- | -------- | ---------- | --------- | --------- | --------- | ---------- | ---------- | ---------- | --------- |
87
+ | MMAudio 44.1kHz | 427.6M | 40 | 0,616 | 1,395 | 0,030 | -29,947 | -2,728 | -2,697 | 2,424 |
88
+ | DACVAE MovieGen | 107.7M | 128 | **0,453** | **1,310** | *0,006* | **10,264** | **10,680** | **10,681** | *4,246* |
89
+ | SAME-L | 852.1M | 256 | 0,774 | 1,575 | 0,007 | 9,939 | 10,374 | 10,376 | 2,982 |
90
+ | KVAE-Audio | 166.9M | 64 | *0,463* | *1,314* | **0,006** | *9,952* | *10,377* | *10,384* | **4,266** |