Nishanthini03/speech_to_speech_model

Tamil Speech-to-Speech Pipeline

This repository provides automatic speech recognition (ASR) and text-to-speech (TTS) models for the Tamil language, designed for end-to-end spoken dialogue systems. The pipeline is demonstrated with the Mother Care Clinic example, where patient audio in Tamil is recognized, replied to conversationally using GPT OSS (Groq API), and synthesized back to natural Tamil speech audio.

---

What’s Inside

• ASR Model
  • Fine-tuned Whisper with LoRA adapter for Tamil speech transcription.
• TTS Model
  • MMS Tamil TTS (facebook/mms-tts-tam) packed for natural speech synthesis.

---

How the Pipeline Works

1. Speech-to-Text (ASR): User speaks in Tamil – recognized and transcribed using local ASR.
2. Dialogue Generation (Groq API): Transcription is sent to GPT OSS via Groq API for a natural, context-aware Tamil reply.
3. Text-to-Speech (TTS): The Tamil reply is synthesized as speech using the local MMS TTS model.

---

Usage Instructions

1. Clone this repository or download model files.

2. Install dependencies

   pip install transformers peft groq soundfile librosa torch
   

3. Load both models from Hub using the correct subfolder (for example: asr_model, tts_model).

4. Authenticate

   • For private repos, log in to Hugging Face via from huggingface_hub import login.

---

Mother Care Clinic Example: Sample Code

from transformers import WhisperForConditionalGeneration, WhisperProcessor, VitsModel, AutoTokenizer
from peft import PeftModel
from groq import Groq
import torch, librosa, soundfile as sf

# Repo IDs and subfolders
asr_repo_id = "Nishanthini03/speech_to_speech_model"
asr_subfolder = "asr_model"
tts_repo_id = "Nishanthini03/speech_to_speech_model"
tts_subfolder = "tts_model"

# Load ASR (Whisper + LoRA)
base_asr = "vasista22/whisper-tamil-large-v2"
asr_model = WhisperForConditionalGeneration.from_pretrained(base_asr)
asr_model = PeftModel.from_pretrained(asr_model, asr_repo_id, subfolder=asr_subfolder)
processor = WhisperProcessor.from_pretrained(base_asr)
asr_model.to("cpu").eval()

# Load TTS (MMS Tamil VITS)
tts_model = VitsModel.from_pretrained(tts_repo_id, subfolder=tts_subfolder)
tts_tokenizer = AutoTokenizer.from_pretrained(tts_repo_id, subfolder=tts_subfolder)
tts_model.to("cpu").eval()

# Groq GPT OSS for NLU/NLG
client = Groq(api_key="your-groq-api-key")
system_msg = "You are a polite and friendly customer care assistant for a Mother Care Clinic in Sri Lanka. Reply in spoken-style Sri Lankan Tamil."
history = [{"role": "system", "content": system_msg}]

def transcribe(audio_path):
    audio, _ = librosa.load(audio_path, sr=16000)
    inputs = processor(audio, sampling_rate=16000, return_tensors="pt").input_features
    ids = asr_model.generate(inputs)
    return processor.batch_decode(ids, skip_special_tokens=True)[0]

def generate_reply(text):
    history.append({"role": "user", "content": text})
    response = client.chat.completions.create(
        model="openai/gpt-oss-120b", messages=history
    )
    reply = response.choices[0].message.content
    history.append({"role": "assistant", "content": reply})
    return reply

def text_to_speech(text, out_path="reply.wav"):
    tts_inp = tts_tokenizer(text, return_tensors="pt")
    with torch.no_grad():
        audio = tts_model(**tts_inp).waveform
    sf.write(out_path, audio.cpu().numpy().squeeze(), tts_model.config.sampling_rate)

# Example usage:
# audio_path = "user_input.wav"
# tamil_text = transcribe(audio_path)
# reply = generate_reply(tamil_text)
# text_to_speech(reply, "clinic_reply.wav")

---

Notes

• This repo does not include the GPT OSS model weights. Dialog is generated by calling Groq's GPT OSS API in real time.
• Only Tamil is supported in this workflow.
• Example provided is for the Mother Care Clinic, but any Tamil conversation context can be supported.

---

License

This repository is released for research and non-commercial use. See LICENSE file for details.

---

Let me know if you want the markdown source or any customization for your specific model/repo names!