Add custom inference handler for Maya1 TTS

README.md

@@ -0,0 +1,119 @@
+# text-to-speech-maya
+
+A small playground for the Maya text-to-speech inference handler and a FastAPI test server.
+
+## Run the FastAPI test server (local)
+
+1. Install dependencies. Follow the official PyTorch instructions for your platform first (CPU/MPS/CUDA), then install the remaining requirements:
+
+```bash
+# Example: install torch from https://pytorch.org/ for your system first, then:
+pip install -r inference_endpoints/requirements.txt
+```
+
+2. Start the server (set `MAYA_MODEL_PATH` to your model path if needed):
+
+```bash
+export MAYA_MODEL_PATH="/path/to/maya_model"
+uvicorn server.app:app --reload --host 0.0.0.0 --port 8000
+```
+
+3. Test the synthesis endpoint (saves `output.wav`):
+
+```bash
+curl -X POST "http://localhost:8000/synthesize" -H "Content-Type: application/json" -d '{
+  "description": "neutral female voice",
+  "text": "Hello world from Maya TTS"
+}' --output output.wav
+```
+
+Notes
+
+- The `EndpointHandler` expects the Maya model to be compatible with the SNAC decoder flow; you may need to adapt token decoding depending on the actual model outputs.
+- `torch` often requires platform-specific wheels; if `pip install -r` fails for `torch`, install the appropriate wheel from pytorch.org first then re-run the requirements install for the remaining packages.
+
+---
+
+## VSCode setup (recommended)
+
+The repo includes helper files to make development in VSCode convenient: a workspace `.venv`, editor settings, tasks, and launch configurations.
+
+1. Create and activate a workspace venv (the Makefile has a helper):
+
+```bash
+make venv
+source .venv/bin/activate
+```
+
+2. Install project requirements into the venv:
+
+```bash
+pip install -r inference_endpoints/requirements.txt
+# dev tools for formatting & linting
+pip install black flake8
+```
+
+3. Open this folder in VSCode. The workspace settings point to `${workspaceFolder}/.venv/bin/python` and the Python extension will activate the venv in the terminal.
+
+4. Useful VSCode features included:
+
+- Formatting: `Black` is configured and will run on save (line length 88).
+- Linting: `flake8` is enabled; run it using the Tasks panel or via the command palette.
+- Tasks: Run `Format: black`, `Lint: flake8`, or `Run server (run.sh)` from the Tasks menu.
+- Launch: Use the `Run FastAPI (uvicorn)` launch configuration to start the server with debugging enabled. It will load `server/.env` for environment variables.
+
+5. Running the server from VSCode:
+
+- Use the Run panel and choose `Run FastAPI (uvicorn)` to start with the debugger.
+- Or open the integrated terminal, ensure the venv is active, and run:
+
+```bash
+./run.sh
+```
+
+6. Poetry users
+
+If you prefer Poetry for the server package, there's a `server/pyproject.toml`. From `server/` run:
+
+```bash
+poetry install
+poetry run uvicorn server.app:app --reload --host 0.0.0.0 --port 8000
+```
+
+---
+
+If you want I can add a short checklist or developer guide in `DEVELOPING.md` with these steps and troubleshooting tips.
+
+# text-to-speech-maya
+
+A small playground for the Maya text-to-speech inference handler and a FastAPI test server.
+
+## Run the FastAPI test server (local)
+
+1. Install dependencies. Follow the official PyTorch instructions for your platform first (CPU/MPS/CUDA), then install the remaining requirements:
+
+```bash
+# Example: install torch from https://pytorch.org/ for your system first, then:
+pip install -r inference_endpoints/requirements.txt
+```
+
+2. Start the server (set `MAYA_MODEL_PATH` to your model path if needed):
+
+```bash
+export MAYA_MODEL_PATH="/path/to/maya_model"
+uvicorn server.app:app --reload --host 0.0.0.0 --port 8000
+```
+
+3. Test the synthesis endpoint (saves `output.wav`):
+
+```bash
+curl -X POST "http://localhost:8000/synthesize" -H "Content-Type: application/json" -d '{
+  "description": "neutral female voice",
+  "text": "Hello world from Maya TTS"
+}' --output output.wav
+```
+
+Notes
+
+- The `EndpointHandler` expects the Maya model to be compatible with the SNAC decoder flow; you may need to adapt token decoding depending on the actual model outputs.
+- `torch` often requires platform-specific wheels; if `pip install -r` fails for `torch`, install the appropriate wheel from pytorch.org first then re-run the requirements install for the remaining packages.

handler.py

@@ -0,0 +1,210 @@
+import base64
+import io
+import os
+import struct
+import wave
+
+
+class EndpointHandler:
+    def __init__(self, path=""):
+        # allow overriding device and dtype via environment variables for local CPU testing
+        # `MAYA_DEVICE`: 'cpu' or 'auto' (default 'auto')
+        # `MAYA_TORCH_DTYPE`: 'bf16' or 'fp32' (default 'bf16')
+        # `MAYA_USE_FAKE`: if '1', use a tiny fake pipeline for smoke testing (no HF downloads)
+        device_override = os.getenv("MAYA_DEVICE", "auto")
+        dtype_override = os.getenv("MAYA_TORCH_DTYPE", "bf16")
+        use_fake = os.getenv("MAYA_USE_FAKE", "0") == "1"
+
+        # map string to torch dtype will be set after importing torch
+
+        if use_fake:
+            # Minimal fake components for quick local smoke tests. Generates a short sine tone.
+            self.model = None
+            self.tokenizer = None
+            # keep device as plain string for fake mode
+            self.device = "cpu"
+            self.snac = None
+            # no external libs required for fake mode
+            self.sf = None
+            return
+
+        # import heavy inference dependencies lazily so fake-mode doesn't require them
+        try:
+            import soundfile as sf
+            import torch
+            from snac import SNAC
+            from transformers import AutoModelForCausalLM, AutoTokenizer
+        except Exception as e:
+            raise RuntimeError(
+                f"Failed to import inference dependencies: {e}.\n"
+                "Install the packages listed in `inference_endpoints/requirements.txt`."
+            )
+
+        # map string to torch dtype
+        torch_dtype = torch.bfloat16 if dtype_override == "bf16" else torch.float32
+
+        # load Maya1 text-to-voice model
+        # force CPU device_map when requested to avoid trying to use GPUs
+        device_map_arg = "auto"
+        if device_override == "cpu":
+            device_map_arg = "cpu"
+
+        self.model = AutoModelForCausalLM.from_pretrained(
+            path, torch_dtype=torch_dtype, device_map=device_map_arg
+        )
+        self.tokenizer = AutoTokenizer.from_pretrained(path)
+
+        # determine device from model parameters (safer than using `model.device`)
+        try:
+            self.device = next(self.model.parameters()).device
+        except StopIteration:
+            # fallback to CPU if model has no parameters
+            self.device = torch.device("cpu")
+
+        # load SNAC model (audio decoder) 24 kHz
+        self.snac = (
+            SNAC.from_pretrained("hubertsiuzdak/snac_24khz").eval().to(self.device)
+        )
+        self.sf = sf
+
+    def __call__(self, data):
+        """
+        Expect `data` dict like:
+        {
+          "description": "... voice description ...",
+          "text": "... text to speak ...",
+          "generation_args": { optional dict for text generation params }
+        }
+        Returns dict with base64 audio:
+        {
+          "audio_base64": "<base64-encoded WAV data>",
+          "sampling_rate": 24000
+        }
+        """
+        description = data.get("description", "")
+        text = data.get("text", "")
+        if not text:
+            return {"error": "No text provided."}
+        prompt = description + "\n" + text
+
+        # If running in fake mode (quick smoke test), synthesize a sine tone
+        if getattr(self, "snac", None) is None and getattr(self, "model", None) is None:
+            # generate a 1-second 24kHz sine wave and write a 16-bit WAV using stdlib
+            sr = 24000
+            duration = 1.0
+            n_samples = int(sr * duration)
+            freq = 220.0
+            # generate samples without numpy
+            waveform = [
+                int(
+                    0.1
+                    * 32767
+                    * __import__("math").sin(
+                        2 * __import__("math").pi * freq * (i / sr)
+                    )
+                )
+                for i in range(n_samples)
+            ]
+
+            buf = io.BytesIO()
+            with wave.open(buf, "wb") as wf:
+                wf.setnchannels(1)
+                wf.setsampwidth(2)  # 16-bit
+                wf.setframerate(sr)
+                # pack samples as little-endian signed 16-bit
+                frames = struct.pack("<" + ("h" * len(waveform)), *waveform)
+                wf.writeframes(frames)
+            wav_bytes = buf.getvalue()
+            b64 = base64.b64encode(wav_bytes).decode("utf-8")
+
+            return {"audio_base64": b64, "sampling_rate": sr}
+
+        inputs = self.tokenizer(prompt, return_tensors="pt").to(self.device)
+        # generate token ids with default or custom params
+        gen_args = data.get("generation_args", {})
+        outputs = self.model.generate(**inputs, **gen_args)
+        token_ids = outputs[0]
+
+        # decode tokens to intermediate representation (for Maya1)
+        # assuming model outputs token ids for audio generation — adjust as per model spec
+        audio_feats = (
+            token_ids  # may need further decoding depending on how Maya1 works
+        )
+
+        # pass features to SNAC to synthesize waveform
+        waveform = self.snac.decode(audio_feats).cpu().numpy()  # shape (n_samples,)
+
+        # convert waveform to bytes (e.g. WAV) using soundfile loaded into self.sf
+        buf = io.BytesIO()
+        self.sf.write(buf, waveform, 24000, format="WAV")
+        wav_bytes = buf.getvalue()
+        b64 = base64.b64encode(wav_bytes).decode("utf-8")
+
+        return {
+            "audio_base64": b64,
+            "sampling_rate": 24000,
+        }
+
+
+# Module-level convenience functions for hosting platforms (Hugging Face Endpoints)
+# The platform typically expects top-level `init` and `predict` (or `run`) callables
+# so we provide thin wrappers around the EndpointHandler class.
+_HANDLER = None
+
+
+def init(model_id: str = None):
+    """Initialize the module-level handler.
+
+    model_id: optional path or model identifier to pass to EndpointHandler.
+    If omitted the handler will look for environment variables `HF_MODEL_ID` or
+    `MODEL_ID` and otherwise instantiate with a blank path (which may be
+    appropriate when the model files are bundled with the repo).
+    """
+    global _HANDLER
+    if _HANDLER is not None:
+        return
+
+    model_path = model_id or os.getenv("HF_MODEL_ID") or os.getenv("MODEL_ID") or ""
+    _HANDLER = EndpointHandler(path=model_path)
+
+
+def predict(payload):
+    """Predict/predict wrapper for hosted endpoints.
+
+    Accepts a dict (recommended) or a JSON-ish payload. If a list is passed,
+    the first element is used. If a plain string is provided it is treated as
+    the `text` field.
+    Returns the same dict structure produced by EndpointHandler.__call__.
+    """
+    global _HANDLER
+    if _HANDLER is None:
+        init()
+
+    data = payload
+    # handle list payloads (common in some platform wrappers)
+    if isinstance(payload, (list, tuple)) and len(payload) > 0:
+        data = payload[0]
+
+    # allow raw JSON string -> dict
+    if isinstance(data, str):
+        try:
+            import json
+
+            data = json.loads(data)
+        except Exception:
+            # treat plain string as the text to synthesize
+            data = {"text": data}
+
+    if not isinstance(data, dict):
+        # best-effort normalization
+        data = {"text": str(data)}
+
+    try:
+        return _HANDLER(data)
+    except Exception as e:
+        return {"error": str(e)}
+
+
+def run(payload):
+    """Alias for predict for platforms that expect `run` entrypoint."""
+    return predict(payload)

requirements.txt

@@ -0,0 +1,5 @@
+torch>=2.0
+transformers>=4.30
+snac
+soundfile
+accelerate