happyman11/TacRFC-whisper-base

Whisper Base — Fine-tuned for Tactical Radio ASR (Hindi-English Code-Switched)

This is openai/whisper-base fine-tuned with LoRA (then merged into the base weights) to transcribe code-switched Hindi-English tactical radio communications over VHF/UHF push-to-talk channels, decoded to Latin/Roman-script text. It is one of eight architectures (FastConformer, SeamlessM4T v2 Large, IndicConformer, Whisper base/small/medium/large-v3, and wav2vec2-XLSR-53) benchmarked side-by-side on the same TacRFC dataset and evaluation protocol.

This repository contains the fully merged model (base weights + LoRA delta already folded in) — it loads and runs like any standard WhisperForConditionalGeneration checkpoint; no PEFT/adapter loading code is required.

Model Details

• Base model: openai/whisper-base (74M parameters)
• Architecture: Whisper encoder-decoder Transformer — 6 encoder layers, 6 decoder layers, d_model=512, 8 attention heads
• Fine-tuning method: LoRA, rank=16, alpha=32, dropout=0.05, applied to the q_proj, k_proj, v_proj, out_proj attention projections in both encoder and decoder, then merged (merge_and_unload()) into the base weights before export
• Decode language forced to: en (English task token), so output stays Latin/Roman script even for the romanized-Hindi portions of speech — this matches how the training labels themselves were encoded
• Precision: fp16
• Task: transcribe (speech-to-text, not translation)

Intended Uses

• Research and benchmarking of ASR robustness on noisy, code-switched, tactical radio speech.
• A starting checkpoint for further fine-tuning on related radio-communications or code-switched Hindi-English speech tasks.

Out of scope: general-purpose English transcription (use the stock openai/whisper-base checkpoint instead — this fine-tune has specialized toward, and on this small a dataset likely narrowed its behavior toward, the tactical-radio domain).

Training Data

This model was fine-tuned on TacRFC — "TacRFC: A Hindi-English Code-Mixed Tactical Radio Frequency Communication Corpus for Low-Resource Domain ASR" (internally referred to during data collection as merged-data-augmented-radio-bg) — this checkpoint is not trained on any public speech dataset beyond its base checkpoint's original pretraining.

• Domain: simulated/exercise military tactical voice traffic over VHF and UHF push-to-talk radio nets (e.g. "Delta one, report to DS that the net is through, over").
• Language: code-switched Hindi–English, transcribed entirely in Latin/Roman script (romanized Hindi + English) — there is no Devanagari text anywhere in the training labels or vocabulary.
• Size: 804 transcribed recordings matched to audio (out of 805 ground-truth rows; one could not be matched to a file), drawn from a working folder of ~834 audio files (a handful of filename/format variants exist alongside the annotated set). Audio is augmented with realistic radio background noise/static to reflect real net conditions.
• Annotation: every recording is labeled with a channel type (VHF/UHF; fraction VHF ≈ 0.92, UHF ≈ 0.08); a speaker count is available for a fraction ≈ 0.89 of recordings and an English translation for ≈ 0.92. The source spreadsheet also defines a 30-category tactical operation taxonomy (Ambush, Contact with enemy, Casevac/medevac, etc.) as columns, but these are not populated in the released data — treat that taxonomy as aspirational schema, not usable labels, unless you annotate it yourself.
• Official split: an 80/10/10 train/validation/test split, stratified by channel type, and a 5-fold stratified cross-validation assignment, are published with the dataset itself (seed 42) — see the dataset card for the exact methodology. Note this specific training run may have computed its own internal split independently rather than loading the dataset's published one; check the Training Procedure section below.
• Availability: released as a private Hugging Face dataset alongside this model (link above). It was collected/simulated for tactical-communications ASR research; it is not derived from real intercepted or classified traffic.

Training Procedure

• Framework: Hugging Face transformers Seq2SeqTrainer, LoRA via peft
• Epochs trained: 12 (best validation-WER checkpoint at epoch 2, selected via load_best_model_at_end / metric_for_best_model="wer")
• Peak learning rate: ~5e-6 (linear schedule, 50 warmup steps)
• Precision: fp16 mixed precision
• Data split: 80/10/10 train/val/test, stratified by VHF/UHF channel type, seed 42
• Max transcript length: 448 tokens (Whisper's generation cap); longer transcripts were excluded from training rather than truncated

Evaluation Results

Metric (best checkpoint, validation split)	Value
Word Error Rate (WER)	1.7478
Character Error Rate (CER)	1.4865
BERTScore F1	0.6488
Best checkpoint	epoch 2 (of 12 trained)

Error rates are fractions (0–1 scale, lower is better); values above 1 mean more errors than reference words or characters.

This checkpoint did not converge to usable transcription quality — validation WER exceeds 1.0 (1.7478) even at its best epoch, and got worse (WER 2.118) by epoch 12 as training continued, indicating divergence/overfitting rather than learning. At only 74M parameters, Whisper-base likely lacks the capacity to adapt to this noisy, code-switched, low-resource task. This model is included as a small-model baseline for comparison and is not recommended for actual transcription use — see whisper-small or whisper-large-v3 in this series for usable results.

How to Use

import torch
import librosa
from transformers import WhisperProcessor, WhisperForConditionalGeneration

model_id = "happyman11/TacRFC-whisper-base"
processor = WhisperProcessor.from_pretrained(model_id, language="en", task="transcribe")
model = WhisperForConditionalGeneration.from_pretrained(model_id, torch_dtype=torch.float16)
model.eval()

audio, _ = librosa.load("your_radio_clip.wav", sr=16000, mono=True)
inputs = processor(audio, sampling_rate=16000, return_tensors="pt").input_features.to(model.dtype)

with torch.no_grad():
    ids = model.generate(
        inputs,
        language="en",
        task="transcribe",
        repetition_penalty=1.3,   # helps avoid single-token loops on repetitive radio phrasing
        max_new_tokens=256,
    )
print(processor.batch_decode(ids, skip_special_tokens=True)[0])

Limitations and Ethical Considerations

• Research artifact, not a production system. This checkpoint was fine-tuned on only 804 recordings — a very small dataset by ASR standards — specifically to benchmark how different architectures adapt to noisy, code-switched tactical radio speech. See Evaluation Results above for this checkpoint's actual measured performance and caveats before relying on it for anything beyond research comparison.
• Narrow domain. Performance is expected to degrade sharply outside VHF/UHF radio traffic, tactical/military phraseology, and Hindi-English code-switching in Latin script. It is not tuned for conversational, broadcast, or clean studio speech.
• Dual-use / sensitive domain. This model transcribes simulated military tactical radio communications. It is released for defense-communications ASR research and benchmarking purposes. Do not use it to process real operational, classified, or intercepted communications, and do not deploy it in any surveillance or targeting system without appropriate legal authorization and human oversight.
• No PII/OPSEC review guarantee. While the underlying dataset is a research corpus rather than real intercepted traffic, no formal OPSEC/PII audit of model outputs has been performed — outputs should not be assumed safe to publish verbatim in downstream applications without review.
• Evaluation caveat. The metrics below are computed on this run's held-out validation split (the same split used for checkpoint selection during training), not an independently held-out test set. Treat them as an internal model-selection signal rather than a final benchmark number.

Citation

If you use this model, please cite the corresponding paper.

@misc{tacrfc_whisper_base,
  title  = {TacRFC: A Hindi-English Code-Mixed Tactical Radio Frequency Communication Corpus for Low-Resource Domain ASR},
  author = {[Your name / lab], IIIT-Delhi},
  year   = {2026},
  note   = {Fine-tuned Whisper Base checkpoint, TacRFC corpus}
}

(Replace the author field above with the correct attribution before publishing.)