Initial commit

README.md

@@ -1,12 +1,192 @@
 ---
-title: Talk Globe
-emoji: 🐠
-colorFrom: green
-colorTo: purple
+title: TalkGlobe (Gradio UI)
+emoji: 🗣️
+colorFrom: purple
+colorTo: red
 sdk: gradio
-sdk_version: 5.27.0
-app_file: app.py
+sdk_version: 5.26.0
+app_file: src/app.py
 pinned: false
+license: mit
+short_description: Real-time translator with multilang support (Gradio UI)
+tags: [webrtc, gradio]
 ---
 
 Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+
+# TalkGlobe: Real-Time Speech Translation
+
+TalkGlobe is an AI-powered application that enables seamless, real-time speech-to-speech translation. Using the state-of-the-art Seamless-M4T-v2 model from Meta, it delivers:
+
+- **🎙️ 101 input languages** for speech recognition
+
+- **🔊 35 output languages** for natural-sounding translated speech
+
+- **⚡ Instant translation** with low latency
+
+- **🖥️ Intuitive interface** for effortless language selection
+
+Simply speak in your native language, choose a target language, and TalkGlobe generates the translated audio in real time. Ideal for travel, business, or multilingual conversations.
+
+## Supported Languages:
+
+Listed below, are the languages supported (either as source or target) by TalkGlobe (according to facebook/seamless-m4t-v2-large model card).
+
+| code     | language               | Source | Target |
+| -------- | ---------------------- | :----: | :----: |
+| afr      | Afrikaans              |   ✅   |   ❌   |
+| amh      | Amharic                |   ✅   |   ❌   |
+| arb      | Modern Standard Arabic |   ✅   |   ✅   |
+| ary      | Moroccan Arabic        |   ✅   |   ❌   |
+| arz      | Egyptian Arabic        |   ✅   |   ❌   |
+| asm      | Assamese               |   ✅   |   ❌   |
+| ast      | Asturian               |   ✅   |   ❌   |
+| azj      | North Azerbaijani      |   ✅   |   ❌   |
+| bel      | Belarusian             |   ✅   |   ❌   |
+| ben      | Bengali                |   ✅   |   ✅   |
+| bos      | Bosnian                |   ✅   |   ❌   |
+| bul      | Bulgarian              |   ✅   |   ❌   |
+| cat      | Catalan                |   ✅   |   ✅   |
+| ceb      | Cebuano                |   ✅   |   ❌   |
+| ces      | Czech                  |   ✅   |   ✅   |
+| ckb      | Central Kurdish        |   ✅   |   ❌   |
+| cmn      | Mandarin Chinese       |   ✅   |   ✅   |
+| cmn_Hant | Mandarin Chinese       |   ✅   |   ✅   |
+| cym      | Welsh                  |   ✅   |   ✅   |
+| dan      | Danish                 |   ✅   |   ✅   |
+| deu      | German                 |   ✅   |   ✅   |
+| ell      | Greek                  |   ✅   |   ❌   |
+| eng      | English                |   ✅   |   ✅   |
+| est      | Estonian               |   ✅   |   ✅   |
+| eus      | Basque                 |   ✅   |   ❌   |
+| fin      | Finnish                |   ✅   |   ✅   |
+| fra      | French                 |   ✅   |   ✅   |
+| fuv      | Nigerian Fulfulde      |   ✅   |   ❌   |
+| gaz      | West Central Oromo     |   ✅   |   ❌   |
+| gle      | Irish                  |   ✅   |   ❌   |
+| glg      | Galician               |   ✅   |   ❌   |
+| guj      | Gujarati               |   ✅   |   ❌   |
+| heb      | Hebrew                 |   ✅   |   ❌   |
+| hin      | Hindi                  |   ✅   |   ✅   |
+| hrv      | Croatian               |   ✅   |   ❌   |
+| hun      | Hungarian              |   ✅   |   ❌   |
+| hye      | Armenian               |   ✅   |   ❌   |
+| ibo      | Igbo                   |   ✅   |   ❌   |
+| ind      | Indonesian             |   ✅   |   ✅   |
+| isl      | Icelandic              |   ✅   |   ❌   |
+| ita      | Italian                |   ✅   |   ✅   |
+| jav      | Javanese               |   ✅   |   ❌   |
+| jpn      | Japanese               |   ✅   |   ✅   |
+| kam      | Kamba                  |   ✅   |   ❌   |
+| kan      | Kannada                |   ✅   |   ❌   |
+| kat      | Georgian               |   ✅   |   ❌   |
+| kaz      | Kazakh                 |   ✅   |   ❌   |
+| kea      | Kabuverdianu           |   ✅   |   ❌   |
+| khk      | Halh Mongolian         |   ✅   |   ❌   |
+| khm      | Khmer                  |   ✅   |   ❌   |
+| kir      | Kyrgyz                 |   ✅   |   ❌   |
+| kor      | Korean                 |   ✅   |   ✅   |
+| lao      | Lao                    |   ✅   |   ❌   |
+| lit      | Lithuanian             |   ✅   |   ❌   |
+| ltz      | Luxembourgish          |   ✅   |   ❌   |
+| lug      | Ganda                  |   ✅   |   ❌   |
+| luo      | Luo                    |   ✅   |   ❌   |
+| lvs      | Standard Latvian       |   ✅   |   ❌   |
+| mai      | Maithili               |   ✅   |   ❌   |
+| mal      | Malayalam              |   ✅   |   ❌   |
+| mar      | Marathi                |   ✅   |   ❌   |
+| mkd      | Macedonian             |   ✅   |   ❌   |
+| mlt      | Maltese                |   ✅   |   ✅   |
+| mni      | Meitei                 |   ✅   |   ❌   |
+| mya      | Burmese                |   ✅   |   ❌   |
+| nld      | Dutch                  |   ✅   |   ✅   |
+| nno      | Norwegian Nynorsk      |   ✅   |   ❌   |
+| nob      | Norwegian Bokmål       |   ✅   |   ❌   |
+| npi      | Nepali                 |   ✅   |   ❌   |
+| nya      | Nyanja                 |   ✅   |   ❌   |
+| oci      | Occitan                |   ✅   |   ❌   |
+| ory      | Odia                   |   ✅   |   ❌   |
+| pan      | Punjabi                |   ✅   |   ❌   |
+| pbt      | Southern Pashto        |   ✅   |   ❌   |
+| pes      | Western Persian        |   ✅   |   ✅   |
+| pol      | Polish                 |   ✅   |   ✅   |
+| por      | Portuguese             |   ✅   |   ✅   |
+| ron      | Romanian               |   ✅   |   ✅   |
+| rus      | Russian                |   ✅   |   ✅   |
+| slk      | Slovak                 |   ✅   |   ✅   |
+| slv      | Slovenian              |   ✅   |   ❌   |
+| sna      | Shona                  |   ✅   |   ❌   |
+| snd      | Sindhi                 |   ✅   |   ❌   |
+| som      | Somali                 |   ✅   |   ❌   |
+| spa      | Spanish                |   ✅   |   ✅   |
+| srp      | Serbian                |   ✅   |   ❌   |
+| swe      | Swedish                |   ✅   |   ✅   |
+| swh      | Swahili                |   ✅   |   ✅   |
+| tam      | Tamil                  |   ✅   |   ❌   |
+| tel      | Telugu                 |   ✅   |   ✅   |
+| tgk      | Tajik                  |   ✅   |   ❌   |
+| tgl      | Tagalog                |   ✅   |   ✅   |
+| tha      | Thai                   |   ✅   |   ✅   |
+| tur      | Turkish                |   ✅   |   ✅   |
+| ukr      | Ukrainian              |   ✅   |   ✅   |
+| urd      | Urdu                   |   ✅   |   ✅   |
+| uzn      | Northern Uzbek         |   ✅   |   ✅   |
+| vie      | Vietnamese             |   ✅   |   ✅   |
+| xho      | Xhosa                  |   ✅   |   ❌   |
+| yor      | Yoruba                 |   ✅   |   ❌   |
+| yue      | Cantonese              |   ✅   |   ❌   |
+| zlm      | Colloquial Malay       |   ✅   |   ❌   |
+| zul      | Zulu                   |   ✅   |   ❌   |
+
+## Getting Started
+
+This guide provides step-by-step instructions to set up and run the project on your local machine for development and testing purposes. For details on deploying the project to a production environment, refer to the Deployment section.
+
+### Prerequisites
+
+To set up and run this project, ensure the following software and tools are installed on your system:
+
+- **Python**: Version `3.10.12` or higher is required. Verify your Python version by running:
+
+  ```bash
+  python3 --version
+  ```
+
+- **Dependencies**: Install the required Python packages listed in requirements.txt using pip. Run the following command in your terminal:
+
+  ```bash
+  pip install -r requirements.txt
+  ```
+
+### Local Development and Testing
+
+To run the application locally for development and testing purposes, execute the following command in your terminal:
+
+```bash
+python app.py
+```
+
+> [!WARNING]
+> Ensure you are in the project's **src** directory before running the script or adapt running path.
+
+## Deployment
+
+### Deployment on Hugging Face Spaces
+
+To deploy the project on Hugging Face Spaces, follow these steps:
+
+1. Create an account on [Hugging Face](https://huggingface.co) if you don’t already have one.
+
+2. Refer to the official [Spaces Overview](https://huggingface.co/docs/hub/en/spaces-overview) documentation for detailed instructions on setting up and deploying your project.
+
+### Deployment on Other Cloud Platforms
+
+For deployment on other cloud or live systems, consult the documentation provided by your chosen service provider. Each platform may have specific requirements and steps for deploying Python-based applications.
+
+## Built With
+
+- [Python 3.10.12](http://www.python.org/) - Developing with the best programming language
+
+## Authors
+
+**Vlasios Dimitriadis** - _Initial work:_ [TalkGlobe](https://huggingface.co/spaces/blasisd/talk-globe)

requirements.txt

@@ -0,0 +1,8 @@
+fastrtc
+fastrtc[vad]
+openpyxl
+protobuf 
+scipy
+sentencepiece
+torchaudio
+transformers

src/app.py

@@ -0,0 +1,207 @@
+from pathlib import Path
+
+import pandas as pd
+
+import torchaudio
+import torch
+import numpy as np
+
+import gradio as gr
+
+from fastrtc import WebRTC, ReplyOnPause
+from transformers import AutoProcessor, SeamlessM4Tv2Model
+
+
+parent_dir = Path(__file__).parents[1]
+config_path = Path(parent_dir, "configs")
+
+processor = AutoProcessor.from_pretrained("facebook/seamless-m4t-v2-large")
+model = SeamlessM4Tv2Model.from_pretrained("facebook/seamless-m4t-v2-large")
+default_sampling_rate = 16_000
+
+
+def translate_audio(
+    audio: tuple[int, np.ndarray], tgt_language: str
+) -> tuple[int, np.ndarray]:
+    """Translate the audio that is captured through the streaming component.
+    Source language of the audio has to be one of the supported languages to be successful.
+
+    :param audio: the captured audio
+    :type audio: tuple[int, np.ndarray]
+    :param tgt_language: the target language for translation
+    :type tgt_language: str
+    :yield: the tuple containing the sampling rate and the audio array
+    :rtype: tuple[int, np.ndarray]
+    """
+    orig_freq, np_array = audio
+    waveform = torch.from_numpy(np_array)
+    waveform = waveform.to(torch.float32)
+    waveform = waveform / 32768.0  # normalize int16 to [-1, 1]
+
+    audio = torchaudio.functional.resample(
+        waveform, orig_freq=orig_freq, new_freq=default_sampling_rate
+    )  # must be a 16 kHz waveform array
+
+    audio_inputs = processor(
+        audios=audio,
+        return_tensors="pt",
+        sampling_rate=default_sampling_rate,
+    )
+
+    audio_array_from_audio = (
+        model.generate(**audio_inputs, tgt_lang=tgt_language)[0].cpu().numpy().squeeze()
+    )
+
+    yield (default_sampling_rate, audio_array_from_audio)
+
+
+# Supported target languages for speech
+supported_langs_df = pd.read_excel(Path(config_path, "supported_languages.xlsx"))
+supported_speech_langs_df = supported_langs_df[
+    supported_langs_df["Target"].str.contains("Sp")
+]
+
+# Labels and values for supported speech languages dropdown
+supported_speech_langs = list(
+    zip(supported_speech_langs_df["language"], supported_speech_langs_df["code"])
+)
+
+# Sort by the first element of the tuple (full language name)
+supported_speech_langs.sort()
+
+css = """
+#componentsContainer {
+    width: 70%;
+    display: block;
+    margin-left: auto;
+    margin-right: auto;
+}
+
+#langDropdown .container .wrap {
+    width: 230px;
+}
+
+.audio-container {
+    padding-bottom: 2rem !important;
+    margin-bottom: 2rem !important;
+}
+
+.vspace-sm { margin-bottom: 20px !important; }
+.vspace-md { margin-bottom: 40px !important; }
+.vspace-lg { margin-bottom: 60px !important; }
+
+.tagline {
+    color: #4a5568;
+}
+.tagline-emphasis {
+    font-family: 'Playfair Display', serif;
+    font-style: italic;
+    color: #718096;
+    position: relative;
+    display: inline-block;
+}
+.tagline-emphasis:after {
+    content: "";
+    position: absolute;
+    bottom: -5px;
+    left: 0;
+    width: 100%;
+    height: 2px;
+    background: linear-gradient(90deg, transparent, #6a11cb, transparent);
+}
+
+.gradio-footer {
+    position: fixed;
+    bottom: 0;
+    left: 0;
+    right: 0;
+    text-align: center;
+    padding: 12px;
+    background: var(--background-fill-secondary);
+    border-top: 1px solid var(--border-color-primary);
+    font-size: 0.9em;
+    z-index: 100;
+    display: flex;
+    justify-content: center;
+    align-items: center;
+    gap: 6px;
+}
+.gradio-footer a {
+    display: inline-flex;
+    align-items: center;
+    gap: 4px;
+    color: var(--link-text-color);
+    text-decoration: none;
+}
+
+.fastrtc-icon {
+    height: 24px;
+    width: 24px;
+}
+"""
+
+with gr.Blocks(
+    theme=gr.themes.Glass(),
+    css=css,
+) as demo:
+    gr.HTML(
+        """
+        <div style='display: flex; align-items: center; justify-content: center; gap: 20px'>
+            <div style="background-color: var(--block-background-fill); border-radius: 8px">
+                <img src="https://images.icon-icons.com/3975/PNG/512/translation_language_translator_icon_251869.png" style="width: 100px; height: 100px;">
+            </div>
+            <div>
+                <h1>TalkGlobe</h1>
+                <p class="tagline">
+                    Break language barriers in real-time <span class="globe-icon">🌍</span><br>
+                    <span class="tagline-emphasis">no more lost in translation</span> <span class="globe-icon">✨</span>
+                </p>
+            </div>
+        </div>
+        """,
+        elem_classes="vspace-sm",
+    )
+
+    # The main components (translation language dropdown and streaming capture component)
+    with gr.Group(elem_id="componentsContainer"):
+        with gr.Row(equal_height=True, min_height="11rem"):
+            with gr.Column(scale=5, elem_id="langCol"):
+                target_lang = gr.Dropdown(
+                    choices=supported_speech_langs,
+                    value="eng",
+                    label="Supported Languages",
+                    info="Select one of the supported languages for translation",
+                    elem_id="langDropdown",
+                )
+
+            with gr.Column(scale=5, elem_id="micCol"):
+                audio = WebRTC(
+                    modality="audio",
+                    mode="send-receive",
+                    label="Audio Stream",
+                )
+
+                # Trigger on pause
+                audio.stream(
+                    ReplyOnPause(translate_audio),
+                    inputs=[audio, target_lang],
+                    outputs=[audio],
+                )
+
+    # Sticky footer (will stay at bottom on all screen sizes)
+    gr.HTML(
+        """
+        <div class="gradio-footer">
+            Powered by 
+            <a href="https://gradio.app/" target="_blank">
+                Gradio <img class="gradio-icon" src="https://www.gradio.app/_app/immutable/assets/gradio.CHB5adID.svg" alt="GradioIcon" style="height:24px; width:auto;">
+            </a>
+            • 
+            <a href="https://freddyaboulton.github.io/gradio-webrtc/" target="_blank">
+                FastRTC <img class="fastrtc-icon" src="https://fastrtc.org/fastrtc_logo.png" alt="FastRTCIcon">
+            </a>
+        </div>
+        """
+    )
+
+demo.launch()