Correct readme
Browse files
README.md
CHANGED
|
@@ -26,7 +26,6 @@ tags:
|
|
| 26 |
- speech
|
| 27 |
- Whisper
|
| 28 |
---
|
| 29 |
-
|
| 30 |
# CarelessWhisper - Causal Whisper Streaming Model
|
| 31 |
Causal Whisper Streaming is a fine tuned version of OpenAI Whisper, which can handle causal data and perform real-time transcription.
|
| 32 |
|
|
@@ -34,11 +33,36 @@ Causal Whisper Streaming is a fine tuned version of OpenAI Whisper, which can ha
|
|
| 34 |
[](https://huggingface.co/spaces/MLSpeech/CarelessWhisper-causal-streaming)
|
| 35 |
|
| 36 |
|
| 37 |
-
## Setup
|
| 38 |
We used Python 3.9.16, PyTorch 2.6.0, and PyTorch-Lightning 2.5.0 to train and test our models.
|
| 39 |
-
All of the required dependencies are available on `requirements.txt` file. Make sure all of the packages are installed before running anything.
|
| 40 |
Portions of this code are adapted from [OpenAI's Whisper](https://github.com/openai/whisper).
|
| 41 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 42 |
## Available Models
|
| 43 |
We fine-tuned three different sizes of Whisper, all support english only transcription.
|
| 44 |
A `large-v2` that was fine tuned on multilingual data is available, and supports English, French, Spanish, German and Portuguese with chunk size of 300 miliseconds.
|
|
@@ -51,33 +75,57 @@ A `large-v2` that was fine tuned on multilingual data is available, and supports
|
|
| 51 |
|
| 52 |
|
| 53 |
## Running Inference
|
| 54 |
-
To run inference, download the repo content, and run accroding to following sections.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 55 |
### CLI Usage
|
| 56 |
The transcription model is easily activated using the next command:
|
| 57 |
```bash
|
| 58 |
# Using a local microphone for streaming transcription, dumping the recording to out.wav
|
| 59 |
-
python
|
| 60 |
-
--output_filename out.wav \
|
| 61 |
-
--channels 2 \
|
| 62 |
-
--model small \
|
| 63 |
-
--chunk_size 300 \
|
| 64 |
-
--device cuda \
|
| 65 |
-
--beam_size 5 \
|
| 66 |
-
--ca_kv_cache \
|
| 67 |
```
|
| 68 |
|
| 69 |
A simulation of a stream on a wav file is also available:
|
| 70 |
```bash
|
| 71 |
# Simulating a stream on a wav file
|
| 72 |
-
python
|
| 73 |
-
--model small \
|
| 74 |
-
--chunk_size 300 \
|
| 75 |
-
--device cuda \
|
| 76 |
-
--beam_size 5 \
|
| 77 |
-
--ca_kv_cache \
|
| 78 |
-
--wav_file /path/to/audio.wav \
|
| 79 |
-
--simulate_stream \
|
| 80 |
-
--use_latency
|
| 81 |
```
|
| 82 |
|
| 83 |
### Python Usage
|
|
@@ -85,14 +133,14 @@ If you prefer using python, a code sinppet utilizing a microphone or a wav file
|
|
| 85 |
|
| 86 |
```python
|
| 87 |
import torch
|
| 88 |
-
import
|
| 89 |
|
| 90 |
model_size = "small" # model size
|
| 91 |
chunk_size = 300 # chunk size in milliseconds
|
| 92 |
multilingual = False # currently on large-v2_300msec supports other languages than english.
|
| 93 |
-
device = "cuda" if torch.cuda.
|
| 94 |
|
| 95 |
-
model =
|
| 96 |
gran=chunk_size,
|
| 97 |
multilingual=multilingual,
|
| 98 |
device=device)
|
|
@@ -113,31 +161,46 @@ texts_wav_simulation = model.transcribe(simulate_stream=True,
|
|
| 113 |
## Training
|
| 114 |
In order to train using LoRA, you can use our existing code. Make sure all the requirements are installed.
|
| 115 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 116 |
### CLI Interface
|
| 117 |
```bash
|
| 118 |
-
#
|
| 119 |
python training_code/train.py \
|
| 120 |
-
--lora \
|
| 121 |
-
--streaming_train \
|
| 122 |
-
--simulate_stream \
|
| 123 |
-
--dataset LIBRI-960-ALIGNED \
|
| 124 |
-
--name example_training_base_model \
|
| 125 |
-
--size base \
|
| 126 |
-
--batch_size 32 \
|
| 127 |
-
--epochs 10 \
|
| 128 |
-
--learning_rate 1e-5 \
|
| 129 |
-
--rank 32 \
|
| 130 |
-
--gran 15 \
|
| 131 |
-
--extra_gran_blocks 1 \
|
| 132 |
-
--streaming_fraction 0.25 \
|
| 133 |
-
--top_k 5 \
|
| 134 |
```
|
| 135 |
|
| 136 |
For more options and training configurations, run:
|
| 137 |
```bash
|
| 138 |
python training_code/train.py --help
|
| 139 |
```
|
| 140 |
-
##
|
| 141 |
|
| 142 |
This project uses components from [OpenAI's Whisper](https://github.com/openai/whisper), licensed under the MIT License.
|
| 143 |
|
|
|
|
|
|
|
|
|
| 26 |
- speech
|
| 27 |
- Whisper
|
| 28 |
---
|
|
|
|
| 29 |
# CarelessWhisper - Causal Whisper Streaming Model
|
| 30 |
Causal Whisper Streaming is a fine tuned version of OpenAI Whisper, which can handle causal data and perform real-time transcription.
|
| 31 |
|
|
|
|
| 33 |
[](https://huggingface.co/spaces/MLSpeech/CarelessWhisper-causal-streaming)
|
| 34 |
|
| 35 |
|
| 36 |
+
## 🔧 Setup
|
| 37 |
We used Python 3.9.16, PyTorch 2.6.0, and PyTorch-Lightning 2.5.0 to train and test our models.
|
|
|
|
| 38 |
Portions of this code are adapted from [OpenAI's Whisper](https://github.com/openai/whisper).
|
| 39 |
|
| 40 |
+
To set up the project environment using `conda`, follow these steps:
|
| 41 |
+
|
| 42 |
+
1. **Clone the repository**
|
| 43 |
+
```bash
|
| 44 |
+
git clone https://github.com/tomer9080/CarelessWhisper-streaming
|
| 45 |
+
cd CarelessWhisper-streaming
|
| 46 |
+
```
|
| 47 |
+
|
| 48 |
+
> 💡 Make sure you have [Miniconda](https://docs.conda.io/en/latest/miniconda.html) or [Anaconda](https://www.anaconda.com/products/distribution) installed before proceeding.
|
| 49 |
+
|
| 50 |
+
2. **Create the conda environment**
|
| 51 |
+
```bash
|
| 52 |
+
conda env create -f environment.yml
|
| 53 |
+
```
|
| 54 |
+
|
| 55 |
+
3. **Activate The environment**
|
| 56 |
+
```bash
|
| 57 |
+
conda activate careless_whisper
|
| 58 |
+
```
|
| 59 |
+
|
| 60 |
+
4. **Install the appropriate PyTorch version**
|
| 61 |
+
Depending on your hardware and CUDA version, install PyTorch by following the instructions at [https://pytorch.org/get-started/locally](https://pytorch.org/get-started/locally).
|
| 62 |
+
This project was tested with CUDA 12.4, but it should also work with compatible earlier or later versions.
|
| 63 |
+
|
| 64 |
+
After installing all of the dependencies, you can try to run inference.
|
| 65 |
+
|
| 66 |
## Available Models
|
| 67 |
We fine-tuned three different sizes of Whisper, all support english only transcription.
|
| 68 |
A `large-v2` that was fine tuned on multilingual data is available, and supports English, French, Spanish, German and Portuguese with chunk size of 300 miliseconds.
|
|
|
|
| 75 |
|
| 76 |
|
| 77 |
## Running Inference
|
| 78 |
+
To run inference, download the repo content, and run from the repository root accroding to following sections.
|
| 79 |
+
|
| 80 |
+
> **Note:** The models are hosted on the [Hugging Face Hub](https://huggingface.co/), which requires an access token.
|
| 81 |
+
> Make sure you are logged in with your token to access the models.
|
| 82 |
+
|
| 83 |
+
### How to Apply Your Hugging Face Access Token
|
| 84 |
+
|
| 85 |
+
1. **Create a Hugging Face account** (if you don’t have one) at [https://huggingface.co/join](https://huggingface.co/join).
|
| 86 |
+
|
| 87 |
+
2. **Generate an access token:**
|
| 88 |
+
- Go to your Hugging Face account settings: [https://huggingface.co/settings/tokens](https://huggingface.co/settings/tokens)
|
| 89 |
+
- Click on **"New token"**, give it a name, select the appropriate scopes (usually `read` is enough), and create it.
|
| 90 |
+
|
| 91 |
+
3. **Login using the Hugging Face CLI:**
|
| 92 |
+
Install the CLI if you don’t have it:
|
| 93 |
+
```bash
|
| 94 |
+
pip install huggingface_hub
|
| 95 |
+
```
|
| 96 |
+
Then login:
|
| 97 |
+
```bash
|
| 98 |
+
huggingface-cli login
|
| 99 |
+
```
|
| 100 |
+
Paste your token when prompted.
|
| 101 |
+
|
| 102 |
+
|
| 103 |
### CLI Usage
|
| 104 |
The transcription model is easily activated using the next command:
|
| 105 |
```bash
|
| 106 |
# Using a local microphone for streaming transcription, dumping the recording to out.wav
|
| 107 |
+
python transcribe.py \
|
| 108 |
+
--output_filename out.wav \
|
| 109 |
+
--channels 2 \
|
| 110 |
+
--model small \
|
| 111 |
+
--chunk_size 300 \
|
| 112 |
+
--device cuda \
|
| 113 |
+
--beam_size 5 \
|
| 114 |
+
--ca_kv_cache \
|
| 115 |
```
|
| 116 |
|
| 117 |
A simulation of a stream on a wav file is also available:
|
| 118 |
```bash
|
| 119 |
# Simulating a stream on a wav file
|
| 120 |
+
python transcribe.py \
|
| 121 |
+
--model small \
|
| 122 |
+
--chunk_size 300 \
|
| 123 |
+
--device cuda \
|
| 124 |
+
--beam_size 5 \
|
| 125 |
+
--ca_kv_cache \
|
| 126 |
+
--wav_file /path/to/audio.wav \
|
| 127 |
+
--simulate_stream \
|
| 128 |
+
--use_latency
|
| 129 |
```
|
| 130 |
|
| 131 |
### Python Usage
|
|
|
|
| 133 |
|
| 134 |
```python
|
| 135 |
import torch
|
| 136 |
+
import careless_whisper_stream
|
| 137 |
|
| 138 |
model_size = "small" # model size
|
| 139 |
chunk_size = 300 # chunk size in milliseconds
|
| 140 |
multilingual = False # currently on large-v2_300msec supports other languages than english.
|
| 141 |
+
device = "cuda" if torch.cuda.is_available() else "cpu"
|
| 142 |
|
| 143 |
+
model = careless_whisper_stream.load_streaming_model(name=model_size,
|
| 144 |
gran=chunk_size,
|
| 145 |
multilingual=multilingual,
|
| 146 |
device=device)
|
|
|
|
| 161 |
## Training
|
| 162 |
In order to train using LoRA, you can use our existing code. Make sure all the requirements are installed.
|
| 163 |
|
| 164 |
+
### Dataset Structure
|
| 165 |
+
|
| 166 |
+
Before starting model training using the command-line interface provided below, you must first configure your dataset dictionary file located at `training_code/ds_dict.py`.
|
| 167 |
+
|
| 168 |
+
This file defines a Python dictionary named `ds_paths`, where you should specify paths to the `train`, `val`, and `test` partitions of your dataset. Each partition should be a CSV file with the following three columns:
|
| 169 |
+
|
| 170 |
+
1. `wav_path` — Path to the WAV audio file.
|
| 171 |
+
2. `tg_path` — Path to the corresponding `.TextGrid` file containing forced alignment.
|
| 172 |
+
3. `raw_text` — Ground truth transcription.
|
| 173 |
+
|
| 174 |
+
> **Note:** The dictionary key (i.e., the name of the dataset) will be used by the training script to identify and load the dataset correctly.
|
| 175 |
+
|
| 176 |
+
You can find an example entry in `training_code/ds_dict.py`.
|
| 177 |
+
|
| 178 |
### CLI Interface
|
| 179 |
```bash
|
|
|
|
| 180 |
python training_code/train.py \
|
| 181 |
+
--lora \
|
| 182 |
+
--streaming_train \
|
| 183 |
+
--simulate_stream \
|
| 184 |
+
--dataset LIBRI-960-ALIGNED \
|
| 185 |
+
--name example_training_base_model \
|
| 186 |
+
--size base \
|
| 187 |
+
--batch_size 32 \
|
| 188 |
+
--epochs 10 \
|
| 189 |
+
--learning_rate 1e-5 \
|
| 190 |
+
--rank 32 \
|
| 191 |
+
--gran 15 \
|
| 192 |
+
--extra_gran_blocks 1 \
|
| 193 |
+
--streaming_fraction 0.25 \
|
| 194 |
+
--top_k 5 \
|
| 195 |
```
|
| 196 |
|
| 197 |
For more options and training configurations, run:
|
| 198 |
```bash
|
| 199 |
python training_code/train.py --help
|
| 200 |
```
|
| 201 |
+
## 🙏 Acknowledgements
|
| 202 |
|
| 203 |
This project uses components from [OpenAI's Whisper](https://github.com/openai/whisper), licensed under the MIT License.
|
| 204 |
|
| 205 |
+
|
| 206 |
+
|