WhisperLv3-PT

This repository contains the WhisperLv3-X-PT-All model with the generation_config file updated according to the whisper-large-v3 model. This update is necessary to avoid errors in certain customization parameters, such as beam_search, task, language, etc. This rep also contains WhisperLv3-X-PT-All quantized to 4 Bits and 8 Bits using BitsAndBytes.
llama.cpp does not support quantization for ASR models.

Description

This model is a fine-tuned WhisperX variant of openai/whisper-large-v3, trained for Portuguese (pt) (European Portuguese, Brazilian and African and Asian Portuguese varieties) automatic speech recognition (ASR). From CAMÕES work.

Model Description

• Base model: openai/whisper-large-v3
• Architecture: Transformer encoder–decoder
• Training: Fine-tuned on around 800 hours of Portuguese speech
• Task: Transcription (task="transcribe")
• Compute type: float16 (recommended)

Quantization Files

This model was quantized using BitsAndBytes, allowing for a drastic reduction in memory, which enables inference on more modest systems.

File	Quantization	Size
8BIT	Q8	1.78 GB
4BIT_NF	Q4	1.06 GB

Inference

If you want to use Hugging Face transformers to generate text, you can do something like this.

from transformers import AutoProcessor, AutoModelForSpeechSeq2Seq
from transformes.audio_utils import load_audio

MODEL = "EliezerCarvalho/WhisperLv3-PT"

PROCESSOR_ASR = AutoProcessor.from_pretrained (MODEL)
MODEL_ASR = AutoModelForSpeechSeq2Seq.from_pretrained (MODEL)

WAV = load_audio (path, sampling_rate = PROCESSOR_ASR.feature_extractor.sampling_rate)

input = PROCESSOR_ASR (wav, sampling_rate = PROCESSOR_ASR.feature_extractor.sampling_rate, return_tensors = "pt")

inference = MODEL_ASR.generate (input["input_features"], return_timestamps = True, task = "transcribe", language = "pt", return_segments = True)