Buckets:
| { | |
| "cells": [ | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "1fgVWTMK9SNz" | |
| }, | |
| "source": [ | |
| "~~~\n", | |
| "Copyright 2025 Google LLC\n", | |
| "\n", | |
| "Licensed under the Apache License, Version 2.0 (the \"License\");\n", | |
| "you may not use this file except in compliance with the License.\n", | |
| "You may obtain a copy of the License at\n", | |
| "\n", | |
| " https://www.apache.org/licenses/LICENSE-2.0\n", | |
| "\n", | |
| "Unless required by applicable law or agreed to in writing, software\n", | |
| "distributed under the License is distributed on an \"AS IS\" BASIS,\n", | |
| "WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", | |
| "See the License for the specific language governing permissions and\n", | |
| "limitations under the License.\n", | |
| "~~~\n", | |
| "\n", | |
| "# Quick start with Hugging Face\n", | |
| "\n", | |
| "<table><tbody><tr>\n", | |
| " <td style=\"text-align: center\">\n", | |
| " <a href=\"https://colab.research.google.com/github/google-health/medasr/blob/main/notebooks/quick_start_with_hugging_face.ipynb\">\n", | |
| " <img alt=\"Google Colab logo\" src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" width=\"32px\"><br> Run in Google Colab\n", | |
| " </a>\n", | |
| " </td>\n", | |
| " <td style=\"text-align: center\">\n", | |
| " <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2Fgoogle-health%2Fmedasr%2Fmain%2Fnotebooks%2Fquick_start_with_hugging_face.ipynb\">\n", | |
| " <img alt=\"Google Cloud Colab Enterprise logo\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" width=\"32px\"><br> Run in Colab Enterprise\n", | |
| " </a>\n", | |
| " </td>\n", | |
| " <td style=\"text-align: center\">\n", | |
| " <a href=\"https://github.com/google-health/medasr/blob/main/notebooks/quick_start_with_hugging_face.ipynb\">\n", | |
| " <img alt=\"GitHub logo\" src=\"https://github.githubassets.com/assets/GitHub-Mark-ea2971cee799.png\" width=\"32px\"><br> View on GitHub\n", | |
| " </a>\n", | |
| " </td>\n", | |
| " <td style=\"text-align: center\">\n", | |
| " <a href=\"https://huggingface.co/google/medasr\">\n", | |
| " <img alt=\"Hugging Face logo\" src=\"https://huggingface.co/front/assets/huggingface_logo-noborder.svg\" width=\"32px\"><br> View on Hugging Face\n", | |
| " </a>\n", | |
| " </td>\n", | |
| "</tr></tbody></table>\n", | |
| "\n", | |
| "This notebook provides a basic demo of using MedASR (Medical Automatic Speech Recognition), an automatic speech recognition model trained for medical terms. MedASR is intended to accelerate building healthcare-based AI applications that require audio input.\n", | |
| "\n", | |
| "Learn more about the model on the [HAI-DEF developer site](https://developers.google.com/health-ai-developer-foundations/medasr).\n", | |
| "\n", | |
| "This notebook is for educational purposes only and does not represent a finished or approved product." | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "t9xt2XZgaaH2" | |
| }, | |
| "source": [ | |
| "## Setup\n", | |
| "\n", | |
| "For an optimal experience, we recommend running this tutorial on a system equipped with an NVIDIA Tesla T4 GPU or higher. While the tutorial can run on a CPU, performance will be significantly slower.\n", | |
| "\n", | |
| "You can try running the notebook on Google Colab with a free T4 GPU:\n", | |
| "\n", | |
| "1. In the upper-right of the Colab window where it reads **Connect**, select **▾ (Additional connection options)**.\n", | |
| "2. Select **Change runtime type**.\n", | |
| "3. Under Hardware accelerator, select **T4 GPU**." | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "L9ITcQtdal7J" | |
| }, | |
| "source": [ | |
| "### Get access to MedASR\n", | |
| "\n", | |
| "Before you get started, make sure that you have access to MedASR models on Hugging Face:\n", | |
| "\n", | |
| "1. If you don't already have a Hugging Face account, you can create one for free by clicking [here](https://huggingface.co/join).\n", | |
| "2. Head over to the [MedASR model page](https://huggingface.co/google/medasr) and accept the usage conditions." | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "W7xTbWg6pY4W" | |
| }, | |
| "source": [ | |
| "### Install dependencies" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "CulOXOrhpY4W" | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "! uv pip install accelerate bitsandbytes git+https://github.com/huggingface/transformers.git@65dc261512cbdb1ee72b88ae5b222f2605aad8e5 levenshtein jiwer librosa" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "source": [ | |
| "**Important**: Please restart your runtime after reinstalling transformers from Git.\n", | |
| "" | |
| ], | |
| "metadata": { | |
| "id": "HWWViEDHUR_5" | |
| } | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "qRFQnPL2a9Dj" | |
| }, | |
| "source": [ | |
| "### Authenticate with Hugging Face\n", | |
| "\n", | |
| "Generate a Hugging Face `read` access token by going to [settings](https://huggingface.co/settings/tokens).\n", | |
| "\n", | |
| "If you are using Google Colab, add your access token to the Colab Secrets manager to securely store it. If not, proceed to run the cell below to authenticate with Hugging Face.\n", | |
| "\n", | |
| "1. Open your Google Colab notebook and click on the 🔑 Secrets tab in the left panel. <img src=\"https://storage.googleapis.com/generativeai-downloads/images/secrets.jpg\" alt=\"The Secrets tab is found on the left panel.\" width=50%>\n", | |
| "2. Create a new secret with the name `HF_TOKEN`.\n", | |
| "3. Copy/paste your token key into the Value input box of `HF_TOKEN`.\n", | |
| "4. Toggle the button on the left to allow notebook access to the secret." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "ZwUUIY0gpY4W" | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "import os\n", | |
| "import sys\n", | |
| "\n", | |
| "google_colab = \"google.colab\" in sys.modules and not os.environ.get(\"VERTEX_PRODUCT\")\n", | |
| "\n", | |
| "if google_colab:\n", | |
| " # Use secret if running in Google Colab\n", | |
| " from google.colab import userdata\n", | |
| " os.environ[\"HF_TOKEN\"] = userdata.get(\"HF_TOKEN\")\n", | |
| "else:\n", | |
| " # Store Hugging Face data under `/content` if running in Colab Enterprise\n", | |
| " if os.environ.get(\"VERTEX_PRODUCT\") == \"COLAB_ENTERPRISE\":\n", | |
| " os.environ[\"HF_HOME\"] = \"/content/hf\"\n", | |
| " # Authenticate with Hugging Face\n", | |
| " ! uv pip install huggingface_hub\n", | |
| " from huggingface_hub import get_token\n", | |
| " if get_token() is None:\n", | |
| " from huggingface_hub import notebook_login\n", | |
| " notebook_login()" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "source": [ | |
| "### Retrieve sample data\n" | |
| ], | |
| "metadata": { | |
| "id": "HC1wQReWHBlx" | |
| } | |
| }, | |
| { | |
| "cell_type": "code", | |
| "source": [ | |
| "import huggingface_hub\n", | |
| "from IPython.display import Audio, display\n", | |
| "\n", | |
| "audio = huggingface_hub.hf_hub_download('google/medasr', 'test_audio.wav')\n", | |
| "sample_transcript = \"Exam type CT chest PE protocol period. Indication 54 year old female, shortness of breath, evaluate for PE period. Technique standard protocol period. Findings colon. Pulmonary vasculature colon. The main PA is patent period. There are filling defects in the segmental branches of the right lower lobe comma compatible with acute PE period. No saddle embolus period. Lungs colon. No pneumothorax period. Small bilateral effusions comma right greater than left period. New paragraph. Impression colon Acute segmental PE right lower lobe period.\"\n", | |
| "display(Audio(audio, autoplay=False))\n", | |
| "model_id = \"google/medasr\"\n" | |
| ], | |
| "metadata": { | |
| "id": "I7n1Fm0GHBsT" | |
| }, | |
| "execution_count": null, | |
| "outputs": [] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "source": [ | |
| "### Define utility functions to evaluate Word Error Rate (WER)" | |
| ], | |
| "metadata": { | |
| "id": "RM9legibu-Lk" | |
| } | |
| }, | |
| { | |
| "cell_type": "code", | |
| "source": [ | |
| "# @title\n", | |
| "import re\n", | |
| "import jiwer\n", | |
| "import Levenshtein\n", | |
| "\n", | |
| "def normalize(s: str) -> str:\n", | |
| " s = s.lower()\n", | |
| " s = s.replace('</s>', '')\n", | |
| " s = re.sub(r\"[^ a-z0-9']\", ' ', s)\n", | |
| " s = ' '.join(s.split())\n", | |
| " return s\n", | |
| "\n", | |
| "def _colored(text, color):\n", | |
| " if color == 'red':\n", | |
| " return f\"\\033[91m{text}\\033[0m\"\n", | |
| " elif color == 'green':\n", | |
| " return f\"\\033[92m{text}\\033[0m\"\n", | |
| " return text\n", | |
| "\n", | |
| "def evaluate(\n", | |
| " ref_text: str,\n", | |
| " hyp_text: str,\n", | |
| " delete_color: str = 'red',\n", | |
| " insert_color: str = 'green',\n", | |
| ") -> None:\n", | |
| " print('HYP:', hyp_text)\n", | |
| " normalized_ref = normalize(ref_text)\n", | |
| " normalized_hyp = normalize(hyp_text)\n", | |
| "\n", | |
| " # Calculate word lists early so we can use them for both jiwer and diffs\n", | |
| " ref_words = normalized_ref.split()\n", | |
| " hyp_words = normalized_hyp.split()\n", | |
| "\n", | |
| " # jiwer.process_words expects a list of strings (sentences) or list of list of words\n", | |
| " measures = jiwer.process_words([normalized_ref], [normalized_hyp])\n", | |
| "\n", | |
| " # Calculate edit operations using Levenshtein for the colored diff\n", | |
| " edits = Levenshtein.editops(ref_words, hyp_words)\n", | |
| "\n", | |
| " r = 0 # Index for the reference words for diff building\n", | |
| " diff = ''\n", | |
| "\n", | |
| " for op, i, j in edits:\n", | |
| " # Add matched words before the current edit\n", | |
| " if r < i:\n", | |
| " diff += ' ' + ' '.join(ref_words[r:i])\n", | |
| " r = i # Update reference index for next iteration\n", | |
| "\n", | |
| " if op == 'replace':\n", | |
| " diff += (\n", | |
| " f' {_colored(f\"{{-{ref_words[i]}-}}\", delete_color)}'\n", | |
| " f' {_colored(f\"{{+{hyp_words[j]}+}}\", insert_color)}'\n", | |
| " )\n", | |
| " r += 1 # Advance reference index after replacement\n", | |
| " elif op == 'insert':\n", | |
| " diff += f' {_colored(f\"{{+{hyp_words[j]}+}}\", insert_color)}'\n", | |
| " # Reference index `r` does not advance for an insertion\n", | |
| " elif op == 'delete':\n", | |
| " diff += f' {_colored(f\"{{-{ref_words[i]}-}}\", delete_color)}'\n", | |
| " r += 1 # Advance reference index after deletion\n", | |
| "\n", | |
| " # Add any remaining matched words from the reference\n", | |
| " if r < len(ref_words):\n", | |
| " diff += ' ' + ' '.join(ref_words[r:])\n", | |
| "\n", | |
| " print(\n", | |
| " f'WER: {measures.wer * 100:.2f}%: '\n", | |
| " f'insertions {measures.insertions}, deletions {measures.deletions}, substitutions {measures.substitutions}, '\n", | |
| " f'ref tokens {len(ref_words)}'\n", | |
| " )\n", | |
| " print(diff)" | |
| ], | |
| "metadata": { | |
| "id": "72oc8l3bfMb4", | |
| "cellView": "form" | |
| }, | |
| "execution_count": null, | |
| "outputs": [] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "GRN9Yg_kpY4X" | |
| }, | |
| "source": [ | |
| "## Run inference on sample audio" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "oPhEFjiOTpcM" | |
| }, | |
| "source": [ | |
| "The following sections contain standalone examples demonstrating how to use the model both directly and with the [`pipeline`](https://huggingface.co/docs/transformers/en/main_classes/pipelines) API. The `pipeline` API provides a simple way to use the model for inference while abstracting away complex details, while directly using the model gives you complete control over the inference process, including preprocessing and postprocessing. In practice, you should select the method that is best suited for your use case.\n", | |
| "\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "T2Of7LKuT_Sz" | |
| }, | |
| "source": [ | |
| "**Run model with the `pipeline` API**" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "YORs_sDfpY4X" | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "from transformers import pipeline\n", | |
| "\n", | |
| "pipe = pipeline(\"automatic-speech-recognition\", model=model_id)\n", | |
| "result = pipe(audio,chunk_length_s=20, stride_length_s=2)\n", | |
| "# the chunk length is how long in seconds MedASR segments audio and the stride length is the overlap between chunks.\n", | |
| "print(result)\n", | |
| "evaluate(ref_text=sample_transcript, hyp_text=result['text'])" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "source": [ | |
| "**Use a language model to further improve MedASR**\n", | |
| "\n", | |
| "We also include a n-gram SentencePiece language model for improving the output quality. Here's an example of how it can be used." | |
| ], | |
| "metadata": { | |
| "id": "9K2G-kOJxXns" | |
| } | |
| }, | |
| { | |
| "cell_type": "code", | |
| "source": [ | |
| "# We must install a modified version of pyctcdecode which is compatible with\n", | |
| "# newer versions of `transformers`.\n", | |
| "#\n", | |
| "# Please use Python 3.12 (e.g. `uv venv --python=3.12`) if kenlm cannot be\n", | |
| "# installed under newer Python versions [https://github.com/kpu/kenlm/pull/473].\n", | |
| "! uv pip install kenlm==0.3.0 git+https://github.com/mediacatch/pyctcdecode.git@ff49fc562bf8fc5d6697d4dcd34188dd630cc977\n", | |
| "\n", | |
| "import dataclasses\n", | |
| "import pyctcdecode\n", | |
| "import transformers\n", | |
| "\n", | |
| "def _restore_text(text: str) -> str:\n", | |
| " return text.replace(\" \", \"\").replace(\"#\", \" \").replace(\"</s>\", \"\").strip()\n", | |
| "\n", | |
| "\n", | |
| "class LasrCtcBeamSearchDecoder:\n", | |
| "\n", | |
| " def __init__(\n", | |
| " self,\n", | |
| " tokenizer: transformers.LasrTokenizer,\n", | |
| " kenlm_model_path=None,\n", | |
| " **kwargs,\n", | |
| " ):\n", | |
| " vocab = [None for _ in range(tokenizer.vocab_size)]\n", | |
| " for k, v in tokenizer.vocab.items():\n", | |
| " if v < tokenizer.vocab_size:\n", | |
| " vocab[v] = k\n", | |
| " assert not [i for i in vocab if i is None]\n", | |
| " # pyctcdecode also expect the blank label to map to the empty string.\n", | |
| " vocab[0] = \"\"\n", | |
| " # Replace '▁' with '#' and prefix each token with a '▁'. This way, pyctcdecode\n", | |
| " # treats each token as a \"word\".\n", | |
| " for i in range(1, len(vocab)):\n", | |
| " piece = vocab[i]\n", | |
| " if not piece.startswith(\"<\") and not piece.endswith(\">\"):\n", | |
| " piece = \"▁\" + piece.replace(\"▁\", \"#\")\n", | |
| " vocab[i] = piece\n", | |
| " self._decoder = pyctcdecode.build_ctcdecoder(\n", | |
| " vocab, kenlm_model_path, **kwargs\n", | |
| " )\n", | |
| "\n", | |
| " def decode_beams(self, *args, **kwargs):\n", | |
| " beams = self._decoder.decode_beams(*args, **kwargs)\n", | |
| " return [dataclasses.replace(i, text=_restore_text(i.text)) for i in beams]\n", | |
| "\n", | |
| "\n", | |
| "def beam_search_pipe(model: str, lm: str):\n", | |
| " feature_extractor = transformers.LasrFeatureExtractor.from_pretrained(model)\n", | |
| " feature_extractor._processor_class = \"LasrProcessorWithLM\"\n", | |
| " pipe = transformers.pipeline(\n", | |
| " task=\"automatic-speech-recognition\",\n", | |
| " model=model,\n", | |
| " feature_extractor=feature_extractor,\n", | |
| " decoder=LasrCtcBeamSearchDecoder(\n", | |
| " transformers.AutoTokenizer.from_pretrained(model), lm\n", | |
| " ),\n", | |
| " )\n", | |
| " assert pipe.type == \"ctc_with_lm\"\n", | |
| " return pipe\n", | |
| "\n", | |
| "pipe_with_lm = beam_search_pipe(\n", | |
| " model_id,\n", | |
| " huggingface_hub.hf_hub_download(model_id, filename='lm_6.kenlm'),\n", | |
| ")\n", | |
| "\n", | |
| "result_with_lm = pipe_with_lm(\n", | |
| " audio,\n", | |
| " chunk_length_s=20,\n", | |
| " stride_length_s=2,\n", | |
| " decoder_kwargs=dict(beam_width=8),\n", | |
| ")\n", | |
| "evaluate(ref_text=sample_transcript, hyp_text=result_with_lm[\"text\"])" | |
| ], | |
| "metadata": { | |
| "id": "dzxdX-QAxWyt" | |
| }, | |
| "execution_count": null, | |
| "outputs": [] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "b9F5HEr6UMqO" | |
| }, | |
| "source": [ | |
| "**Run model directly**" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "source": [ | |
| "import librosa\n", | |
| "import torch\n", | |
| "from transformers import AutoModelForCTC, AutoProcessor\n", | |
| "\n", | |
| "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n", | |
| "\n", | |
| "# 1. Setup\n", | |
| "processor = AutoProcessor.from_pretrained(model_id)\n", | |
| "model = AutoModelForCTC.from_pretrained(model_id).to(device)\n", | |
| "\n", | |
| "# 2. Load audio with librosa\n", | |
| "# We use the 'audio' variable from the earlier cell which contains the local path to the downloaded file.\n", | |
| "# We explicitly set sr=16000 because most speech models expect 16kHz audio.\n", | |
| "speech, sample_rate = librosa.load(audio, sr=16000)\n", | |
| "\n", | |
| "# 3. Preprocess\n", | |
| "inputs = processor(speech, sampling_rate=sample_rate)\n", | |
| "inputs = inputs.to(device)\n", | |
| "\n", | |
| "# 4. Inference\n", | |
| "# Here we do not segment audio at all.\n", | |
| "outputs = model.generate(**inputs)\n", | |
| "decoded_text = processor.batch_decode(outputs)[0]\n", | |
| "evaluate(ref_text=sample_transcript, hyp_text=decoded_text)" | |
| ], | |
| "metadata": { | |
| "id": "DVhxeWCBGqKn" | |
| }, | |
| "execution_count": null, | |
| "outputs": [] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "_3M0Hyl3pY4X" | |
| }, | |
| "source": [ | |
| "## Run inference on your own audio recording\n", | |
| "\n", | |
| "This section demonstrates how you can record your own speech and test the model.\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "source": [ | |
| "### Write transcript of recording\n", | |
| "\n", | |
| "Start by editing the transcript field below to reflect the speech you'd like to record." | |
| ], | |
| "metadata": { | |
| "id": "NG0IN3WPxsfv" | |
| } | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "qunAkiKspY4X" | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "import pprint\n", | |
| "\n", | |
| "transcript = \"FINDINGS: LUNGS AND PLEURA: There is a focal airspace opacity in the right lower lobe, consistent with airspace disease. The remaining lung parenchyma is clear. There is no evidence of pneumothorax or significant pleural effusion. HEART AND MEDIASTINUM: The cardiac silhouette is normal in size. The mediastinal and hilar contours are unremarkable. BONES: The visualized osseous structures, including the ribs, clavicles, and thoracic spine, are unremarkable for acute fracture or lytic lesion. SOFT TISSUES: The overlying soft tissues are unremarkable.\" # @param {type: \"string\"}\n", | |
| "pprint.pprint(transcript)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "source": [ | |
| "### Record audio\n" | |
| ], | |
| "metadata": { | |
| "id": "o9ovgI6Vd8AP" | |
| } | |
| }, | |
| { | |
| "cell_type": "code", | |
| "metadata": { | |
| "id": "208abe81", | |
| "cellView": "form" | |
| }, | |
| "source": [ | |
| "# @title #### Define helper functions to record audio\n", | |
| "\n", | |
| "import base64\n", | |
| "from google.colab import output\n", | |
| "\n", | |
| "def receive_audio_data_simplified(base64_audio, filename):\n", | |
| " # Decode the base64 string\n", | |
| " audio_bytes = base64.b64decode(base64_audio)\n", | |
| "\n", | |
| " # Save the file to the Colab local filesystem\n", | |
| " with open(filename, 'wb') as f:\n", | |
| " f.write(audio_bytes)\n", | |
| "\n", | |
| " print(f\"Saved audio file to: {filename}\")\n", | |
| "\n", | |
| "# Register the callback so JavaScript can call it\n", | |
| "output.register_callback('receive_audio_data_simplified', receive_audio_data_simplified)" | |
| ], | |
| "execution_count": null, | |
| "outputs": [] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "source": [ | |
| "Now press play in the next code cell and dictate the words in the transcript above and we'll see how good MedASR is at recognizing your voice." | |
| ], | |
| "metadata": { | |
| "id": "tkUDGUuw_pTQ" | |
| } | |
| }, | |
| { | |
| "cell_type": "code", | |
| "source": [ | |
| "# @title\n", | |
| "%%javascript\n", | |
| "(async () => {\n", | |
| " const display = (msg) => {\n", | |
| " const div = document.createElement('div');\n", | |
| " div.innerHTML = msg;\n", | |
| " document.body.appendChild(div);\n", | |
| " };\n", | |
| "\n", | |
| " const html = `\n", | |
| " <div>\n", | |
| " <button id=\"record\">Start Recording</button>\n", | |
| " <button id=\"stop\" disabled>Stop Recording</button>\n", | |
| " <button id=\"play\" disabled>Play Recording</button>\n", | |
| " <audio id=\"audioPlayback\" controls style=\"margin-top: 10px;\"></audio>\n", | |
| " <div id=\"status\" style=\"margin-top: 10px;\">Click \"Start Recording\" to begin.</div>\n", | |
| " </div>\n", | |
| " `;\n", | |
| " display(html);\n", | |
| "\n", | |
| " const recordButton = document.getElementById('record');\n", | |
| " const stopButton = document.getElementById('stop');\n", | |
| " const playButton = document.getElementById('play');\n", | |
| " const audioPlayback = document.getElementById('audioPlayback');\n", | |
| " const statusDiv = document.getElementById('status');\n", | |
| "\n", | |
| " let mediaRecorder;\n", | |
| " let audioChunks = [];\n", | |
| " let audioBlob;\n", | |
| "\n", | |
| " const enableButtons = (record, stop, play) => {\n", | |
| " recordButton.disabled = !record;\n", | |
| " stopButton.disabled = !stop;\n", | |
| " playButton.disabled = !play;\n", | |
| " };\n", | |
| "\n", | |
| " const sendAudioToPython = async (blob, filename) => {\n", | |
| " if (!blob) {\n", | |
| " statusDiv.innerText = 'No audio recorded to save.';\n", | |
| " return;\n", | |
| " }\n", | |
| " enableButtons(false, false, false);\n", | |
| " statusDiv.innerText = 'Sending audio data to Python...';\n", | |
| " const reader = new FileReader();\n", | |
| " reader.onloadend = async () => {\n", | |
| " const base64data = reader.result.split(',')[1];\n", | |
| " try {\n", | |
| " await google.colab.kernel.invokeFunction('receive_audio_data_simplified', [base64data, filename], {});\n", | |
| " statusDiv.innerText = 'Audio data sent to Python successfully.';\n", | |
| " enableButtons(true, false, true);\n", | |
| " } catch (e) {\n", | |
| " statusDiv.innerText = 'Error sending audio data to Python: ' + e.message;\n", | |
| " enableButtons(true, false, true);\n", | |
| " }\n", | |
| " };\n", | |
| " reader.readAsDataURL(blob);\n", | |
| " };\n", | |
| "\n", | |
| " recordButton.onclick = async () => {\n", | |
| " statusDiv.innerText = 'Requesting microphone access...';\n", | |
| " try {\n", | |
| " if (!mediaRecorder) {\n", | |
| " const stream = await navigator.mediaDevices.getUserMedia({ audio: true });\n", | |
| " mediaRecorder = new MediaRecorder(stream);\n", | |
| "\n", | |
| " mediaRecorder.ondataavailable = event => {\n", | |
| " audioChunks.push(event.data);\n", | |
| " };\n", | |
| "\n", | |
| " mediaRecorder.onstop = async () => {\n", | |
| " audioBlob = new Blob(audioChunks, { type: 'audio/webm' });\n", | |
| " const audioUrl = URL.createObjectURL(audioBlob);\n", | |
| " audioPlayback.src = audioUrl;\n", | |
| " statusDiv.innerText = 'Recording stopped. Saving audio...';\n", | |
| " enableButtons(false, false, false);\n", | |
| " const filename = \"test_audio_1.webm\";\n", | |
| " sendAudioToPython(audioBlob, filename);\n", | |
| " audioChunks = [];\n", | |
| " };\n", | |
| " }\n", | |
| "\n", | |
| " audioChunks = [];\n", | |
| " mediaRecorder.start();\n", | |
| " statusDiv.innerText = 'Recording... Click \"Stop Recording\" to finish.';\n", | |
| " enableButtons(false, true, false);\n", | |
| " } catch (err) {\n", | |
| " statusDiv.innerText = 'Error accessing microphone: ' + err.message;\n", | |
| " console.error(err);\n", | |
| " }\n", | |
| " };\n", | |
| "\n", | |
| " stopButton.onclick = () => {\n", | |
| " mediaRecorder.stop();\n", | |
| " statusDiv.innerText = 'Processing recording...';\n", | |
| " enableButtons(false, false, false);\n", | |
| " };\n", | |
| "\n", | |
| " playButton.onclick = () => {\n", | |
| " audioPlayback.play();\n", | |
| " statusDiv.innerText = 'Playing recorded audio.';\n", | |
| " };\n", | |
| "\n", | |
| "})();" | |
| ], | |
| "metadata": { | |
| "id": "bC3DPYO5cnIB", | |
| "cellView": "form" | |
| }, | |
| "execution_count": null, | |
| "outputs": [] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "_-4LriNOpY4X" | |
| }, | |
| "source": [ | |
| "### Run inference on recorded audio" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "fgL2JLlGpY4X" | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "hyp_text = pipe(\"test_audio_1.webm\")['text']\n", | |
| "print(hyp_text)\n", | |
| "evaluate(ref_text=transcript, hyp_text=hyp_text)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "PHTxQttKYNpa" | |
| }, | |
| "source": [ | |
| "## Next steps\n", | |
| "\n", | |
| "Explore the other [notebooks](https://github.com/google-health/medasr/blob/main/notebooks) to learn what else you can do with the model." | |
| ] | |
| } | |
| ], | |
| "metadata": { | |
| "accelerator": "GPU", | |
| "colab": { | |
| "gpuType": "T4", | |
| "toc_visible": true, | |
| "private_outputs": true | |
| }, | |
| "kernelspec": { | |
| "display_name": "Python 3", | |
| "name": "python3" | |
| } | |
| }, | |
| "nbformat": 4, | |
| "nbformat_minor": 0 | |
| } | |
Xet Storage Details
- Size:
- 118 kB
- Xet hash:
- 19904425a69d07f1461912cac5e9c45967182aab28b4ce63293735a48dbe03ac
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.