Update README.md

README.md

@@ -37,9 +37,76 @@ model-index:
 ---
 
 
+# Voxtral small LoRA finetuned on CoRaL release 1
 
+This is a Danish state of the art automatic speech recognition (ASR) model, which combines the decoder and audio-adapter of [**Voxtral-Small-24B-2507**](mistralai/Voxtral-Small-24B-2507) with the encoder from [**roest-whisper-large-v1**](CoRal-project/roest-whisper-large-v1). The decoder and audio-adapter were finetuned using LoRA for 2 epochs on the Danish [coral dataset](CoRal-project/coral) for automatic speech recognition (ASR).
 
-# hinge/voxtral
+## Evaluation Results
 
-This is a fine-tuned version of the **Voxtral-Small-24B-2507** model, with its audio encoder replaced by the highly-performant Danish Whisper encoder from **roest-whisper-large-v1**. The combined model was further fine-tuned using LoRA on the Danish `CoRal-project/coral` dataset for automatic speech recognition (ASR).
+| Model | Number of parameters | [CoRal](https://huggingface.co/datasets/alexandrainst/coral/viewer/read_aloud/test) CER | [CoRal](https://huggingface.co/datasets/alexandrainst/coral/viewer/read_aloud/test) WER |
+|:---|---:|---:|---:|
+| [hinge/danstral-v1](hinge/danstral-v1) | 24B | **4.2% ± 0.2%** | **9.7% ± 0.3%** | 
+| [Alvenir/coral-1-whisper-large](https://huggingface.co/Alvenir/coral-1-whisper-large) | 1.540B | 4.3% ± 0.2% | 10.4% ± 0.3% | 
+| [alexandrainst/roest-315m](https://huggingface.co/alexandrainst/roest-315m) | 0.315B | 6.6% ± 0.2% | 17.0% ± 0.4% | 
+| [mhenrichsen/hviske-v2](https://huggingface.co/syvai/hviske-v2) | 1.540B | 4.7% ± 0.07% | 11.8% ± 0.3% |
+| [openai/whisper-large-v3](https://hf.co/openai/whisper-large-v3) | 1.540B | 11.4% ± 0.3% | 28.3% ± 0.6% |
 
+
+## Limitations
+danstral-v1:
+- is a finetune of voxtral-small-24b, whose encoder is a finetune of mistral-small-24b - a highly capable multilingual LLM currently ranked 13 on the [Danish generative leaderboard](https://euroeval.com/leaderboards/Monolingual/danish/#__tabbed_1_1). Although Mistral does not [disclose their datasets used for training](https://help.mistral.ai/en/articles/347390-does-mistral-ai-communicate-on-the-training-datasets), it is likely that Danish wikipedia articles were used in the training cohort of mistral small. Since the CoRaL test split also contains read aloud samples from Danish wikipedia, there is a risk of data leakage, which would influence the test scores.   
+- is huge, 16x the size of whisper large with only moddest performance improvements. However, the LoRA adapter amount to just 25mil parameters. 
+- is finetuned solely on the coral v1 dataset and performance  may deterioate significantly for other data sources.
+
+## Future work and ideas
+- SOTA performance was achieved using a LoRA adapter with 25M parameters. A full finetune on larger GPU's and bigger datasets will likely give even better results
+- Using danstral-v1 for knowledge distillation to train smaller models
+
+
+## How to use 
+
+```python
+from transformers import VoxtralForConditionalGeneration, AutoProcessor, WhisperForConditionalGeneration
+import torch
+from peft import PeftModel
+from datasets import load_dataset, Audio
+import os
+
+repo_id = "mistralai/Voxtral-Small-24B-2507"
+
+processor = AutoProcessor.from_pretrained(repo_id)
+model = VoxtralForConditionalGeneration.from_pretrained(repo_id, torch_dtype=torch.bfloat16, device_map="auto",attn_implementation="flash_attention_2")
+
+# Load audio encoder
+whisper_model = WhisperForConditionalGeneration.from_pretrained(
+    "CoRal-project/roest-whisper-large-v1",
+    torch_dtype=torch.bfloat16,
+    attn_implementation="flash_attention_2"
+)
+
+whisper_encoder_state_dict = whisper_model.model.encoder.state_dict()
+model.audio_tower.load_state_dict(whisper_encoder_state_dict)
+
+# Load LoRA adapters
+model = PeftModel.from_pretrained(model, "hinge/danstral-v1")
+
+coral = load_dataset("CoRal-project/coral", "read_aloud")
+coral = coral.cast_column("audio", Audio(sampling_rate=16000))
+
+
+for i in range(10):
+    sample = coral["test"][i]
+    audio_data = sample['audio']
+    ground_truth = sample['text']
+    
+    inputs = processor.apply_transcription_request(language="da", audio=audio_data['array'], format=["WAV"], model_id=repo_id)
+    inputs = inputs.to("cuda:0", dtype=torch.bfloat16)
+    
+    outputs = model.generate(**inputs, max_new_tokens=256,do_sample=False)
+    decoded_outputs = processor.batch_decode(outputs[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)
+    
+    print(f"Ground Truth: {ground_truth}")
+    print(f"Prediction:   {decoded_outputs[0]}")
+    print("-" * 40)
+
+```