MLSpeech commited on
Commit
37634ed
·
verified ·
1 Parent(s): 1f34f23

Correct readme

Browse files
Files changed (1) hide show
  1. README.md +103 -40
README.md CHANGED
@@ -26,7 +26,6 @@ tags:
26
  - speech
27
  - Whisper
28
  ---
29
-
30
  # CarelessWhisper - Causal Whisper Streaming Model
31
  Causal Whisper Streaming is a fine tuned version of OpenAI Whisper, which can handle causal data and perform real-time transcription.
32
 
@@ -34,11 +33,36 @@ Causal Whisper Streaming is a fine tuned version of OpenAI Whisper, which can ha
34
  [![Demo on Hugging Face](https://img.shields.io/badge/🤗%20Demo-Hugging%20Face-blueviolet?logo=huggingface&logoColor=white)](https://huggingface.co/spaces/MLSpeech/CarelessWhisper-causal-streaming)
35
 
36
 
37
- ## Setup
38
  We used Python 3.9.16, PyTorch 2.6.0, and PyTorch-Lightning 2.5.0 to train and test our models.
39
- All of the required dependencies are available on `requirements.txt` file. Make sure all of the packages are installed before running anything.
40
  Portions of this code are adapted from [OpenAI's Whisper](https://github.com/openai/whisper).
41
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
42
  ## Available Models
43
  We fine-tuned three different sizes of Whisper, all support english only transcription.
44
  A `large-v2` that was fine tuned on multilingual data is available, and supports English, French, Spanish, German and Portuguese with chunk size of 300 miliseconds.
@@ -51,33 +75,57 @@ A `large-v2` that was fine tuned on multilingual data is available, and supports
51
 
52
 
53
  ## Running Inference
54
- To run inference, download the repo content, and run accroding to following sections.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
55
  ### CLI Usage
56
  The transcription model is easily activated using the next command:
57
  ```bash
58
  # Using a local microphone for streaming transcription, dumping the recording to out.wav
59
- python causal_whisper_stream/streaming_transcribe.py \
60
- --output_filename out.wav \ # path to dump recording to - optional
61
- --channels 2 \ # number of channels
62
- --model small \ # model size
63
- --chunk_size 300 \ # chunk size in milliseconds
64
- --device cuda \ # inference device
65
- --beam_size 5 \ # run inference with beam of size 5
66
- --ca_kv_cache \ # run inference with CA kv cache
67
  ```
68
 
69
  A simulation of a stream on a wav file is also available:
70
  ```bash
71
  # Simulating a stream on a wav file
72
- python causal_whisper_stream/streaming_transcribe.py \
73
- --model small \ # model size
74
- --chunk_size 300 \ # chunk size in milliseconds
75
- --device cuda \ # inference device
76
- --beam_size 5 \ # run inference with beam of size 5
77
- --ca_kv_cache \ # run inference with CA kv cache
78
- --wav_file /path/to/audio.wav \ # path to audio file to transcribe
79
- --simulate_stream \ # a must for a simulation to occur
80
- --use_latency \ # simulating real latency of the chunk size - optional.
81
  ```
82
 
83
  ### Python Usage
@@ -85,14 +133,14 @@ If you prefer using python, a code sinppet utilizing a microphone or a wav file
85
 
86
  ```python
87
  import torch
88
- import causal_whisper_stream
89
 
90
  model_size = "small" # model size
91
  chunk_size = 300 # chunk size in milliseconds
92
  multilingual = False # currently on large-v2_300msec supports other languages than english.
93
- device = "cuda" if torch.cuda.is_avaialable() else "cpu"
94
 
95
- model = causal_whisper_stream.load_streaming_model(name=model_size,
96
  gran=chunk_size,
97
  multilingual=multilingual,
98
  device=device)
@@ -113,31 +161,46 @@ texts_wav_simulation = model.transcribe(simulate_stream=True,
113
  ## Training
114
  In order to train using LoRA, you can use our existing code. Make sure all the requirements are installed.
115
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
116
  ### CLI Interface
117
  ```bash
118
- #
119
  python training_code/train.py \
120
- --lora \ # use LoRA fine-tuning, currently only this way is supported.
121
- --streaming_train \ # streaming traing
122
- --simulate_stream \ # simulate a stream of data, use streaming mel
123
- --dataset LIBRI-960-ALIGNED \ # dataset name from a dict
124
- --name example_training_base_model \ # name for differentiation
125
- --size base \ # model size to train on
126
- --batch_size 32 \ # batch size
127
- --epochs 10 \ # maximal epochs
128
- --learning_rate 1e-5 \ # initial learning rate
129
- --rank 32 \ # LoRA rank
130
- --gran 15 \ # Chunk size in encoder frames units. 15 encoder frames = 300msec
131
- --extra_gran_blocks 1 \ # extra blocks for inital chunk. 1 means that an extra block is added for the beginning. i.e., the inital chunk will be 600msec.
132
- --streaming_fraction 0.25 \ # fraction of samples out of the possible subsequences in the streaming process to train on
133
- --top_k 5 \ # how many ckpts to save.
134
  ```
135
 
136
  For more options and training configurations, run:
137
  ```bash
138
  python training_code/train.py --help
139
  ```
140
- ### 🙏 Acknowledgements
141
 
142
  This project uses components from [OpenAI's Whisper](https://github.com/openai/whisper), licensed under the MIT License.
143
 
 
 
 
26
  - speech
27
  - Whisper
28
  ---
 
29
  # CarelessWhisper - Causal Whisper Streaming Model
30
  Causal Whisper Streaming is a fine tuned version of OpenAI Whisper, which can handle causal data and perform real-time transcription.
31
 
 
33
  [![Demo on Hugging Face](https://img.shields.io/badge/🤗%20Demo-Hugging%20Face-blueviolet?logo=huggingface&logoColor=white)](https://huggingface.co/spaces/MLSpeech/CarelessWhisper-causal-streaming)
34
 
35
 
36
+ ## 🔧 Setup
37
  We used Python 3.9.16, PyTorch 2.6.0, and PyTorch-Lightning 2.5.0 to train and test our models.
 
38
  Portions of this code are adapted from [OpenAI's Whisper](https://github.com/openai/whisper).
39
 
40
+ To set up the project environment using `conda`, follow these steps:
41
+
42
+ 1. **Clone the repository**
43
+ ```bash
44
+ git clone https://github.com/tomer9080/CarelessWhisper-streaming
45
+ cd CarelessWhisper-streaming
46
+ ```
47
+
48
+ > 💡 Make sure you have [Miniconda](https://docs.conda.io/en/latest/miniconda.html) or [Anaconda](https://www.anaconda.com/products/distribution) installed before proceeding.
49
+
50
+ 2. **Create the conda environment**
51
+ ```bash
52
+ conda env create -f environment.yml
53
+ ```
54
+
55
+ 3. **Activate The environment**
56
+ ```bash
57
+ conda activate careless_whisper
58
+ ```
59
+
60
+ 4. **Install the appropriate PyTorch version**
61
+ Depending on your hardware and CUDA version, install PyTorch by following the instructions at [https://pytorch.org/get-started/locally](https://pytorch.org/get-started/locally).
62
+ This project was tested with CUDA 12.4, but it should also work with compatible earlier or later versions.
63
+
64
+ After installing all of the dependencies, you can try to run inference.
65
+
66
  ## Available Models
67
  We fine-tuned three different sizes of Whisper, all support english only transcription.
68
  A `large-v2` that was fine tuned on multilingual data is available, and supports English, French, Spanish, German and Portuguese with chunk size of 300 miliseconds.
 
75
 
76
 
77
  ## Running Inference
78
+ To run inference, download the repo content, and run from the repository root accroding to following sections.
79
+
80
+ > **Note:** The models are hosted on the [Hugging Face Hub](https://huggingface.co/), which requires an access token.
81
+ > Make sure you are logged in with your token to access the models.
82
+
83
+ ### How to Apply Your Hugging Face Access Token
84
+
85
+ 1. **Create a Hugging Face account** (if you don’t have one) at [https://huggingface.co/join](https://huggingface.co/join).
86
+
87
+ 2. **Generate an access token:**
88
+ - Go to your Hugging Face account settings: [https://huggingface.co/settings/tokens](https://huggingface.co/settings/tokens)
89
+ - Click on **"New token"**, give it a name, select the appropriate scopes (usually `read` is enough), and create it.
90
+
91
+ 3. **Login using the Hugging Face CLI:**
92
+ Install the CLI if you don’t have it:
93
+ ```bash
94
+ pip install huggingface_hub
95
+ ```
96
+ Then login:
97
+ ```bash
98
+ huggingface-cli login
99
+ ```
100
+ Paste your token when prompted.
101
+
102
+
103
  ### CLI Usage
104
  The transcription model is easily activated using the next command:
105
  ```bash
106
  # Using a local microphone for streaming transcription, dumping the recording to out.wav
107
+ python transcribe.py \
108
+ --output_filename out.wav \
109
+ --channels 2 \
110
+ --model small \
111
+ --chunk_size 300 \
112
+ --device cuda \
113
+ --beam_size 5 \
114
+ --ca_kv_cache \
115
  ```
116
 
117
  A simulation of a stream on a wav file is also available:
118
  ```bash
119
  # Simulating a stream on a wav file
120
+ python transcribe.py \
121
+ --model small \
122
+ --chunk_size 300 \
123
+ --device cuda \
124
+ --beam_size 5 \
125
+ --ca_kv_cache \
126
+ --wav_file /path/to/audio.wav \
127
+ --simulate_stream \
128
+ --use_latency
129
  ```
130
 
131
  ### Python Usage
 
133
 
134
  ```python
135
  import torch
136
+ import careless_whisper_stream
137
 
138
  model_size = "small" # model size
139
  chunk_size = 300 # chunk size in milliseconds
140
  multilingual = False # currently on large-v2_300msec supports other languages than english.
141
+ device = "cuda" if torch.cuda.is_available() else "cpu"
142
 
143
+ model = careless_whisper_stream.load_streaming_model(name=model_size,
144
  gran=chunk_size,
145
  multilingual=multilingual,
146
  device=device)
 
161
  ## Training
162
  In order to train using LoRA, you can use our existing code. Make sure all the requirements are installed.
163
 
164
+ ### Dataset Structure
165
+
166
+ Before starting model training using the command-line interface provided below, you must first configure your dataset dictionary file located at `training_code/ds_dict.py`.
167
+
168
+ This file defines a Python dictionary named `ds_paths`, where you should specify paths to the `train`, `val`, and `test` partitions of your dataset. Each partition should be a CSV file with the following three columns:
169
+
170
+ 1. `wav_path` — Path to the WAV audio file.
171
+ 2. `tg_path` — Path to the corresponding `.TextGrid` file containing forced alignment.
172
+ 3. `raw_text` — Ground truth transcription.
173
+
174
+ > **Note:** The dictionary key (i.e., the name of the dataset) will be used by the training script to identify and load the dataset correctly.
175
+
176
+ You can find an example entry in `training_code/ds_dict.py`.
177
+
178
  ### CLI Interface
179
  ```bash
 
180
  python training_code/train.py \
181
+ --lora \
182
+ --streaming_train \
183
+ --simulate_stream \
184
+ --dataset LIBRI-960-ALIGNED \
185
+ --name example_training_base_model \
186
+ --size base \
187
+ --batch_size 32 \
188
+ --epochs 10 \
189
+ --learning_rate 1e-5 \
190
+ --rank 32 \
191
+ --gran 15 \
192
+ --extra_gran_blocks 1 \
193
+ --streaming_fraction 0.25 \
194
+ --top_k 5 \
195
  ```
196
 
197
  For more options and training configurations, run:
198
  ```bash
199
  python training_code/train.py --help
200
  ```
201
+ ## 🙏 Acknowledgements
202
 
203
  This project uses components from [OpenAI's Whisper](https://github.com/openai/whisper), licensed under the MIT License.
204
 
205
+
206
+