README.md

metadata

language:
  - en
  - zh
  - id
  - ms
base_model: openai/whisper-large-v3
tags:
  - language-identification
  - code-switching
  - audio-classification
  - whisper
pipeline_tag: audio-classification

CS-LID-4

Utterance-level code-switch detection and 4-language identification ({English, Mandarin, Indonesian, Malay}) for conversational speech.

The model is a pair of linear heads on pooled openai/whisper-large-v3 encoder features:

• cs head — 1 logit (sigmoid): probability the utterance contains code-switching.
• lang4 head — 4-way softmax over {en, zh, id, ms}: the (dominant) language of the utterance. Trained on monolingual utterances.

Each head comes in two pooling variants stored in the same checkpoint:

• real (recommended) — mean over the encoder frames that correspond to actual audio (50 frames/s, up to 1500).
• pad — mean over all 1500 encoder frames of the padded 30 s window.

Training data spans conversational and read speech in the four languages plus en–zh, en–id, and en–ms code-switching corpora (see yyhenggg/CS-LID-Dataset), with utterance-level labels.

Usage

pip install torch transformers soundfile huggingface_hub
python inference.py audio1.wav audio2.flac

Audio should be mono; 16 kHz is expected (other rates are resampled when torchaudio is installed) and input longer than 30 s is truncated. Output, one line per file:

audio1.wav  P(cs)=0.912  lang=zh  p_en=0.031 p_zh=0.952 p_id=0.009 p_ms=0.008

Python API

import torch
from huggingface_hub import hf_hub_download
from transformers import WhisperProcessor, WhisperForConditionalGeneration

ck = torch.load(hf_hub_download('yyhenggg/CS-LID-Model', 'cslid4_head.pt'),
                map_location='cpu', weights_only=True)
sd, langs = ck['head_state_dict'], ck['langs']   # langs = ['en','zh','id','ms']

proc = WhisperProcessor.from_pretrained('openai/whisper-large-v3')
enc = WhisperForConditionalGeneration.from_pretrained(
    'openai/whisper-large-v3').eval().model.encoder

audio = ...  # float32 mono 16 kHz numpy array, <= 30 s
feats = proc([audio], sampling_rate=16000, return_tensors='pt',
             padding='max_length').input_features
with torch.no_grad():
    h = enc(feats, return_dict=True).last_hidden_state.float()[0]
n_real = min(int(len(audio) / 16000 * 50), 1500)          # 'real' pooling
x = h[:max(n_real, 1)].mean(0)

p_cs = torch.sigmoid(x @ sd['cs_real.weight'].T + sd['cs_real.bias']).item()
p_lang = torch.softmax(x @ sd['lang4_real.weight'].T
                       + sd['lang4_real.bias'], dim=-1)
print(p_cs, dict(zip(langs, p_lang.tolist())))

Checkpoint format

cslid4_head.pt is a plain torch.save dict:

head_state_dict:
  cs_real.weight    [1, 1280]     cs_real.bias    [1]
  cs_pad.weight     [1, 1280]     cs_pad.bias     [1]
  lang4_real.weight [4, 1280]     lang4_real.bias [4]
  lang4_pad.weight  [4, 1280]     lang4_pad.bias  [4]
langs: ['en', 'zh', 'id', 'ms']
design: short provenance string