updated

README.md

@@ -10,14 +10,14 @@ license: apache-2.0
 # DashengTokenizer
 
 DashengTokenizer is a high-performance continious audio tokenizer designed for audio understanding and generation tasks.
-Compared to previous works, our framework simply trains a single linear layer to enable audio generation for semantically strong encoders.
+Compared to previous works, our framework trains a **single linear layer** to enable audio generation for semantically strong encoders.
 
 Achievements:
 
-* State-of-the-Art Audio Understanding: DashengTokenizer consistently outperforms most previous self-supervised and supervised audio encoders.
-* High-Fidelity Signal Reconstruction: Maintains exceptional signal integrity, ensuring that audio remains crisp and accurate after processing.
-* Accelerated Audio Generation Training: Achieves optimal performance significantly faster than standard VAE models, reducing training time and costs.
-* Superior Speech Enhancement: Provides a more robust encoding foundation for isolating and clarifying speech in noisy environments.
+* **State-of-the-Art** Audio Understanding: DashengTokenizer consistently outperforms most previous self-supervised and supervised audio encoders.
+* **High-Fidelity** Signal Reconstruction: Maintains exceptional signal integrity, ensuring that audio remains crisp and accurate after processing.
+* Accelerated **Audio Generation** Training: Achieves optimal performance significantly faster than standard VAE models, reducing training time and costs.
+* Superior **Speech Enhancement**: Provides a more robust encoding foundation for isolating and clarifying speech in noisy environments.
 
 
 ![Framework](./figures/framework.png)

notebook.ipynb

@@ -2,18 +2,115 @@
  "cells": [
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 1,
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Requirement already satisfied: transformers in ./.venv/lib/python3.11/site-packages (5.1.0)\n",
+      "Requirement already satisfied: torch in ./.venv/lib/python3.11/site-packages (2.10.0)\n",
+      "Requirement already satisfied: torchaudio in ./.venv/lib/python3.11/site-packages (2.10.0)\n",
+      "Requirement already satisfied: librosa in ./.venv/lib/python3.11/site-packages (0.11.0)\n",
+      "Collecting pandas\n",
+      "  Downloading pandas-3.0.0-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl.metadata (79 kB)\n",
+      "Requirement already satisfied: scikit-learn in ./.venv/lib/python3.11/site-packages (1.8.0)\n",
+      "Requirement already satisfied: tqdm in ./.venv/lib/python3.11/site-packages (4.67.3)\n",
+      "Requirement already satisfied: huggingface-hub<2.0,>=1.3.0 in ./.venv/lib/python3.11/site-packages (from transformers) (1.4.1)\n",
+      "Requirement already satisfied: numpy>=1.17 in ./.venv/lib/python3.11/site-packages (from transformers) (2.3.5)\n",
+      "Requirement already satisfied: packaging>=20.0 in ./.venv/lib/python3.11/site-packages (from transformers) (26.0)\n",
+      "Requirement already satisfied: pyyaml>=5.1 in ./.venv/lib/python3.11/site-packages (from transformers) (6.0.3)\n",
+      "Requirement already satisfied: regex!=2019.12.17 in ./.venv/lib/python3.11/site-packages (from transformers) (2026.1.15)\n",
+      "Requirement already satisfied: tokenizers<=0.23.0,>=0.22.0 in ./.venv/lib/python3.11/site-packages (from transformers) (0.22.2)\n",
+      "Requirement already satisfied: typer-slim in ./.venv/lib/python3.11/site-packages (from transformers) (0.23.0)\n",
+      "Requirement already satisfied: safetensors>=0.4.3 in ./.venv/lib/python3.11/site-packages (from transformers) (0.7.0)\n",
+      "Requirement already satisfied: filelock in ./.venv/lib/python3.11/site-packages (from huggingface-hub<2.0,>=1.3.0->transformers) (3.21.2)\n",
+      "Requirement already satisfied: fsspec>=2023.5.0 in ./.venv/lib/python3.11/site-packages (from huggingface-hub<2.0,>=1.3.0->transformers) (2026.2.0)\n",
+      "Requirement already satisfied: hf-xet<2.0.0,>=1.2.0 in ./.venv/lib/python3.11/site-packages (from huggingface-hub<2.0,>=1.3.0->transformers) (1.2.0)\n",
+      "Requirement already satisfied: httpx<1,>=0.23.0 in ./.venv/lib/python3.11/site-packages (from huggingface-hub<2.0,>=1.3.0->transformers) (0.28.1)\n",
+      "Requirement already satisfied: shellingham in ./.venv/lib/python3.11/site-packages (from huggingface-hub<2.0,>=1.3.0->transformers) (1.5.4)\n",
+      "Requirement already satisfied: typing-extensions>=4.1.0 in ./.venv/lib/python3.11/site-packages (from huggingface-hub<2.0,>=1.3.0->transformers) (4.15.0)\n",
+      "Requirement already satisfied: anyio in ./.venv/lib/python3.11/site-packages (from httpx<1,>=0.23.0->huggingface-hub<2.0,>=1.3.0->transformers) (4.12.1)\n",
+      "Requirement already satisfied: certifi in ./.venv/lib/python3.11/site-packages (from httpx<1,>=0.23.0->huggingface-hub<2.0,>=1.3.0->transformers) (2026.1.4)\n",
+      "Requirement already satisfied: httpcore==1.* in ./.venv/lib/python3.11/site-packages (from httpx<1,>=0.23.0->huggingface-hub<2.0,>=1.3.0->transformers) (1.0.9)\n",
+      "Requirement already satisfied: idna in ./.venv/lib/python3.11/site-packages (from httpx<1,>=0.23.0->huggingface-hub<2.0,>=1.3.0->transformers) (3.11)\n",
+      "Requirement already satisfied: h11>=0.16 in ./.venv/lib/python3.11/site-packages (from httpcore==1.*->httpx<1,>=0.23.0->huggingface-hub<2.0,>=1.3.0->transformers) (0.16.0)\n",
+      "Requirement already satisfied: sympy>=1.13.3 in ./.venv/lib/python3.11/site-packages (from torch) (1.14.0)\n",
+      "Requirement already satisfied: networkx>=2.5.1 in ./.venv/lib/python3.11/site-packages (from torch) (3.6.1)\n",
+      "Requirement already satisfied: jinja2 in ./.venv/lib/python3.11/site-packages (from torch) (3.1.6)\n",
+      "Requirement already satisfied: cuda-bindings==12.9.4 in ./.venv/lib/python3.11/site-packages (from torch) (12.9.4)\n",
+      "Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.8.93 in ./.venv/lib/python3.11/site-packages (from torch) (12.8.93)\n",
+      "Requirement already satisfied: nvidia-cuda-runtime-cu12==12.8.90 in ./.venv/lib/python3.11/site-packages (from torch) (12.8.90)\n",
+      "Requirement already satisfied: nvidia-cuda-cupti-cu12==12.8.90 in ./.venv/lib/python3.11/site-packages (from torch) (12.8.90)\n",
+      "Requirement already satisfied: nvidia-cudnn-cu12==9.10.2.21 in ./.venv/lib/python3.11/site-packages (from torch) (9.10.2.21)\n",
+      "Requirement already satisfied: nvidia-cublas-cu12==12.8.4.1 in ./.venv/lib/python3.11/site-packages (from torch) (12.8.4.1)\n",
+      "Requirement already satisfied: nvidia-cufft-cu12==11.3.3.83 in ./.venv/lib/python3.11/site-packages (from torch) (11.3.3.83)\n",
+      "Requirement already satisfied: nvidia-curand-cu12==10.3.9.90 in ./.venv/lib/python3.11/site-packages (from torch) (10.3.9.90)\n",
+      "Requirement already satisfied: nvidia-cusolver-cu12==11.7.3.90 in ./.venv/lib/python3.11/site-packages (from torch) (11.7.3.90)\n",
+      "Requirement already satisfied: nvidia-cusparse-cu12==12.5.8.93 in ./.venv/lib/python3.11/site-packages (from torch) (12.5.8.93)\n",
+      "Requirement already satisfied: nvidia-cusparselt-cu12==0.7.1 in ./.venv/lib/python3.11/site-packages (from torch) (0.7.1)\n",
+      "Requirement already satisfied: nvidia-nccl-cu12==2.27.5 in ./.venv/lib/python3.11/site-packages (from torch) (2.27.5)\n",
+      "Requirement already satisfied: nvidia-nvshmem-cu12==3.4.5 in ./.venv/lib/python3.11/site-packages (from torch) (3.4.5)\n",
+      "Requirement already satisfied: nvidia-nvtx-cu12==12.8.90 in ./.venv/lib/python3.11/site-packages (from torch) (12.8.90)\n",
+      "Requirement already satisfied: nvidia-nvjitlink-cu12==12.8.93 in ./.venv/lib/python3.11/site-packages (from torch) (12.8.93)\n",
+      "Requirement already satisfied: nvidia-cufile-cu12==1.13.1.3 in ./.venv/lib/python3.11/site-packages (from torch) (1.13.1.3)\n",
+      "Requirement already satisfied: triton==3.6.0 in ./.venv/lib/python3.11/site-packages (from torch) (3.6.0)\n",
+      "Requirement already satisfied: cuda-pathfinder~=1.1 in ./.venv/lib/python3.11/site-packages (from cuda-bindings==12.9.4->torch) (1.3.4)\n",
+      "Requirement already satisfied: audioread>=2.1.9 in ./.venv/lib/python3.11/site-packages (from librosa) (3.1.0)\n",
+      "Requirement already satisfied: numba>=0.51.0 in ./.venv/lib/python3.11/site-packages (from librosa) (0.63.1)\n",
+      "Requirement already satisfied: scipy>=1.6.0 in ./.venv/lib/python3.11/site-packages (from librosa) (1.17.0)\n",
+      "Requirement already satisfied: joblib>=1.0 in ./.venv/lib/python3.11/site-packages (from librosa) (1.5.3)\n",
+      "Requirement already satisfied: decorator>=4.3.0 in ./.venv/lib/python3.11/site-packages (from librosa) (5.2.1)\n",
+      "Requirement already satisfied: soundfile>=0.12.1 in ./.venv/lib/python3.11/site-packages (from librosa) (0.13.1)\n",
+      "Requirement already satisfied: pooch>=1.1 in ./.venv/lib/python3.11/site-packages (from librosa) (1.9.0)\n",
+      "Requirement already satisfied: soxr>=0.3.2 in ./.venv/lib/python3.11/site-packages (from librosa) (1.0.0)\n",
+      "Requirement already satisfied: lazy_loader>=0.1 in ./.venv/lib/python3.11/site-packages (from librosa) (0.4)\n",
+      "Requirement already satisfied: msgpack>=1.0 in ./.venv/lib/python3.11/site-packages (from librosa) (1.1.2)\n",
+      "Requirement already satisfied: python-dateutil>=2.8.2 in ./.venv/lib/python3.11/site-packages (from pandas) (2.9.0.post0)\n",
+      "Requirement already satisfied: threadpoolctl>=3.2.0 in ./.venv/lib/python3.11/site-packages (from scikit-learn) (3.6.0)\n",
+      "Requirement already satisfied: llvmlite<0.47,>=0.46.0dev0 in ./.venv/lib/python3.11/site-packages (from numba>=0.51.0->librosa) (0.46.0)\n",
+      "Requirement already satisfied: platformdirs>=2.5.0 in ./.venv/lib/python3.11/site-packages (from pooch>=1.1->librosa) (4.7.0)\n",
+      "Requirement already satisfied: requests>=2.19.0 in ./.venv/lib/python3.11/site-packages (from pooch>=1.1->librosa) (2.32.5)\n",
+      "Requirement already satisfied: six>=1.5 in ./.venv/lib/python3.11/site-packages (from python-dateutil>=2.8.2->pandas) (1.17.0)\n",
+      "Requirement already satisfied: charset_normalizer<4,>=2 in ./.venv/lib/python3.11/site-packages (from requests>=2.19.0->pooch>=1.1->librosa) (3.4.4)\n",
+      "Requirement already satisfied: urllib3<3,>=1.21.1 in ./.venv/lib/python3.11/site-packages (from requests>=2.19.0->pooch>=1.1->librosa) (2.6.3)\n",
+      "Requirement already satisfied: cffi>=1.0 in ./.venv/lib/python3.11/site-packages (from soundfile>=0.12.1->librosa) (2.0.0)\n",
+      "Requirement already satisfied: pycparser in ./.venv/lib/python3.11/site-packages (from cffi>=1.0->soundfile>=0.12.1->librosa) (3.0)\n",
+      "Requirement already satisfied: mpmath<1.4,>=1.1.0 in ./.venv/lib/python3.11/site-packages (from sympy>=1.13.3->torch) (1.3.0)\n",
+      "Requirement already satisfied: MarkupSafe>=2.0 in ./.venv/lib/python3.11/site-packages (from jinja2->torch) (3.0.3)\n",
+      "Requirement already satisfied: typer>=0.23.0 in ./.venv/lib/python3.11/site-packages (from typer-slim->transformers) (0.23.0)\n",
+      "Requirement already satisfied: click>=8.0.0 in ./.venv/lib/python3.11/site-packages (from typer>=0.23.0->typer-slim->transformers) (8.3.1)\n",
+      "Requirement already satisfied: rich>=10.11.0 in ./.venv/lib/python3.11/site-packages (from typer>=0.23.0->typer-slim->transformers) (14.3.2)\n",
+      "Requirement already satisfied: annotated-doc>=0.0.2 in ./.venv/lib/python3.11/site-packages (from typer>=0.23.0->typer-slim->transformers) (0.0.4)\n",
+      "Requirement already satisfied: markdown-it-py>=2.2.0 in ./.venv/lib/python3.11/site-packages (from rich>=10.11.0->typer>=0.23.0->typer-slim->transformers) (4.0.0)\n",
+      "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in ./.venv/lib/python3.11/site-packages (from rich>=10.11.0->typer>=0.23.0->typer-slim->transformers) (2.19.2)\n",
+      "Requirement already satisfied: mdurl~=0.1 in ./.venv/lib/python3.11/site-packages (from markdown-it-py>=2.2.0->rich>=10.11.0->typer>=0.23.0->typer-slim->transformers) (0.1.2)\n",
+      "Downloading pandas-3.0.0-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl (11.2 MB)\n",
+      "\u001b[2K   \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m11.2/11.2 MB\u001b[0m \u001b[31m10.7 MB/s\u001b[0m  \u001b[33m0:00:01\u001b[0m0.9 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m01\u001b[0m\n",
+      "\u001b[?25hInstalling collected packages: pandas\n",
+      "Successfully installed pandas-3.0.0\n"
+     ]
+    }
+   ],
    "source": [
     "!pip install transformers torch torchaudio librosa pandas scikit-learn tqdm"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 2,
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/home/richman/Programming/dashengtokenizer_hf/.venv/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
+      "  from .autonotebook import tqdm as notebook_tqdm\n"
+     ]
+    }
+   ],
    "source": [
     "import torch\n",
     "import torch.nn as nn\n",
@@ -30,7 +127,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 3,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -60,7 +157,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 4,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -82,7 +179,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 5,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -110,9 +207,61 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 6,
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Downloading ESC-50 dataset...\n",
+      "ESC-50 dataset downloaded and extracted\n",
+      "class_reference='configuration_dasheng_tokenizer.DashengTokenizerConfig'\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "A new version of the following files was downloaded from https://huggingface.co/mispeech/dashengtokenizer:\n",
+      "- configuration_dasheng_tokenizer.py\n",
+      ". Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "class_reference='modeling_dasheng_tokenizer.DashengTokenizerModel'\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "A new version of the following files was downloaded from https://huggingface.co/mispeech/dashengtokenizer:\n",
+      "- modeling_dasheng_encoder.py\n",
+      ". Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.\n",
+      "A new version of the following files was downloaded from https://huggingface.co/mispeech/dashengtokenizer:\n",
+      "- vocos.py\n",
+      ". Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.\n",
+      "A new version of the following files was downloaded from https://huggingface.co/mispeech/dashengtokenizer:\n",
+      "- modeling_dasheng_tokenizer.py\n",
+      "- modeling_dasheng_encoder.py\n",
+      "- vocos.py\n",
+      ". Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.\n",
+      "Loading weights: 100%|███████████████████████████████████████████████████████| 522/522 [00:00<00:00, 1545.80it/s, Materializing param=upsampler.weight]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Model embedding dimension: 1280\n",
+      "Using device: cpu\n"
+     ]
+    }
+   ],
    "source": [
     "# Download dataset\n",
     "download_esc50()\n",
@@ -140,9 +289,17 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 8,
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Train samples: 1600, Val samples: 400\n"
+     ]
+    }
+   ],
    "source": [
     "# Create datasets\n",
     "audio_dir = 'ESC-50/audio'\n",
@@ -165,7 +322,15 @@
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Epoch 1/10 Training:   5%|███▊                                                                           | 19/400 [00:53<15:40,  2.47s/it, loss=3.9561]"
+     ]
+    }
+   ],
    "source": [
     "# Training setup\n",
     "optimizer = torch.optim.Adam(classifier.parameters(), lr=1e-3)\n",
@@ -260,7 +425,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
@@ -274,9 +439,9 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.0"
+   "version": "3.11.11"
   }
  },
  "nbformat": 4,
  "nbformat_minor": 4
-}
+}