Update README.md

a6de3c1 verified about 2 months ago

4.33 kB

---
license: apache-2.0
language:
- en
- hi
- ta
- te
- ml
- bn
- mr
- kn
base_model:
- Aratako/MioCodec-25Hz-24kHz
pipeline_tag: audio-to-audio
---
**Overview**
- **What:** Lightweight toolkit to run MioCodec decoding in two modes: streaming (real‑time token→audio from an LLM websocket) and offline (JSONL of embeddings → WAV).
- **Why:** You can avoid pushing the entire MioCodec source to your HF model repo — push only checkpoints and a few small helper files instead.

**Requirements**
- See [MioCodec-stream/requirements.txt](MioCodec-stream/requirements.txt#L1) for the base list. Minimal packages:
  - `torch`, `numpy`
  - `git+https://github.com/Aratako/MioCodec.git` (or `miocodec` package if available)
  - `websockets` (streaming)
  - `soundfile` (offline WAV write) or `sounddevice` (optional playback)

**Repository contents (what to push to HF)**
- `weights/` — your `lora_windowed` checkpoint files (use Git LFS).
- `streaming_istft_head.py` — streaming-capable ISTFT head (small helper).
- `windowed_streamer.py` — windowed streamer implementation.
- `mio_codec_wrapper.py` — wrapper script for streaming and offline decode. See [mio_codec_wrapper.py](mydrive2/Indic-Mio/MioCodec/mio_codec_wrapper.py#L1).
- `requirements.txt` — environment dependencies. See [MioCodec-stream/requirements.txt](MioCodec-stream/requirements.txt#L1).

**Install (inference host)**
1. Create a fresh venv and activate it:

```bash
python3 -m venv ~/mio_env
source ~/mio_env/bin/activate
```

2. Install dependencies:

```bash
pip install -r MioCodec-stream/requirements.txt
```

If you prefer to install MioCodec from the GitHub repo directly:

```bash
pip install git+https://github.com/Aratako/MioCodec.git
```

**Usage — Streaming (LLM websocket → audio)**
- Start/point your LLM (vLLM or other) to stream embeddings to a websocket endpoint (JSON messages). Expected message format (per message):

```json
{"embedding": [0.12, 0.34, ...]}
```

Send a final message `{ "finished": true }` when the utterance completes.

Run the wrapper in streaming mode (plays audio locally if `--play`):

```bash
python mydrive2/Indic-Mio/MioCodec/mio_codec_wrapper.py --mode stream \
  --uri ws://localhost:8000/stream --device cpu --play
```

Notes:
- The wrapper prefers local helper files (`windowed_streamer.py`, `streaming_istft_head.py`) when present; otherwise it will attempt to use the installed `miocodec` package and swap in the streaming head if available.
- Adjust `--chunk-size`, `--left`, `--right` options to tune streaming latency/quality.

**Usage — Offline (JSONL tokens → WAV)**
- Prepare a JSONL file where each line is a JSON object with an `embedding` array, e.g.:

```jsonl
{"embedding": [ ... ]}
{"embedding": [ ... ]}
```

Run offline decode:

```bash
python MioCodec-stream/mio_codec_wrapper.py --mode offline --input tokens.jsonl --out out.wav --device cpu
```

The wrapper will prefer the `windowed_streamer` workflow (exact frame accounting). If the streamer is not available it falls back to a best-effort one-shot decode (less exact).

**Pushing weights to Hugging Face**
- Only push the small helper files and the `weights/` folder to your HF repo. Use Git LFS for large checkpoint files:

```bash
git lfs install
git lfs track "weights/*"
git add .gitattributes weights/ streaming_istft_head.py windowed_streamer.py mio_codec_wrapper.py requirements.txt
git commit -m "Add weights + streaming helpers"
git push
```

When calling `MioCodecModel.from_pretrained()` in the wrapper, you can pass the HF repo id (for example `PRIYANSHUDHAKED/Mio-Codec-Stream`) so the model code downloads the weights from the hub automatically.

**Troubleshooting**
- Import errors (e.g. missing `BloomPreTrainedModel`) usually come from incompatible `transformers` versions — use the version recommended by the model package or install inside a fresh venv.
- If audio shows seams, increase `--right` lookahead or ensure `windowed_streamer.py` is present so exact frame trimming is used.
- If playback is silent but files exist, inspect `out.wav` in a player; the wrapper may write raw float32 if `soundfile` is unavailable.

**Next steps / options**
- I can add a tiny `README` section that shows how to configure a vLLM or other LLM server to emit the expected websocket JSON format. Want that added?

	---
	license: apache-2.0
	language:
	- en
	- hi
	- ta
	- te
	- ml
	- bn
	- mr
	- kn
	base_model:
	- Aratako/MioCodec-25Hz-24kHz
	pipeline_tag: audio-to-audio
	---
	Overview
	- What: Lightweight toolkit to run MioCodec decoding in two modes: streaming (real‑time token→audio from an LLM websocket) and offline (JSONL of embeddings → WAV).
	- Why: You can avoid pushing the entire MioCodec source to your HF model repo — push only checkpoints and a few small helper files instead.

	Requirements
	- See [MioCodec-stream/requirements.txt](MioCodec-stream/requirements.txt#L1) for the base list. Minimal packages:
	- `torch`, `numpy`
	- `git+https://github.com/Aratako/MioCodec.git` (or `miocodec` package if available)
	- `websockets` (streaming)
	- `soundfile` (offline WAV write) or `sounddevice` (optional playback)

	Repository contents (what to push to HF)
	- `weights/` — your `lora_windowed` checkpoint files (use Git LFS).
	- `streaming_istft_head.py` — streaming-capable ISTFT head (small helper).
	- `windowed_streamer.py` — windowed streamer implementation.
	- `mio_codec_wrapper.py` — wrapper script for streaming and offline decode. See [mio_codec_wrapper.py](mydrive2/Indic-Mio/MioCodec/mio_codec_wrapper.py#L1).
	- `requirements.txt` — environment dependencies. See [MioCodec-stream/requirements.txt](MioCodec-stream/requirements.txt#L1).

	Install (inference host)
	1. Create a fresh venv and activate it:

	```bash
	python3 -m venv ~/mio_env
	source ~/mio_env/bin/activate
	```

	2. Install dependencies:

	```bash
	pip install -r MioCodec-stream/requirements.txt
	```

	If you prefer to install MioCodec from the GitHub repo directly:

	```bash
	pip install git+https://github.com/Aratako/MioCodec.git
	```

	Usage — Streaming (LLM websocket → audio)
	- Start/point your LLM (vLLM or other) to stream embeddings to a websocket endpoint (JSON messages). Expected message format (per message):

	```json
	{"embedding": [0.12, 0.34, ...]}
	```

	Send a final message `{ "finished": true }` when the utterance completes.

	Run the wrapper in streaming mode (plays audio locally if `--play`):

	```bash
	python mydrive2/Indic-Mio/MioCodec/mio_codec_wrapper.py --mode stream \
	--uri ws://localhost:8000/stream --device cpu --play
	```

	Notes:
	- The wrapper prefers local helper files (`windowed_streamer.py`, `streaming_istft_head.py`) when present; otherwise it will attempt to use the installed `miocodec` package and swap in the streaming head if available.
	- Adjust `--chunk-size`, `--left`, `--right` options to tune streaming latency/quality.

	Usage — Offline (JSONL tokens → WAV)
	- Prepare a JSONL file where each line is a JSON object with an `embedding` array, e.g.:

	```jsonl
	{"embedding": [ ... ]}
	{"embedding": [ ... ]}
	```

	Run offline decode:

	```bash
	python MioCodec-stream/mio_codec_wrapper.py --mode offline --input tokens.jsonl --out out.wav --device cpu
	```

	The wrapper will prefer the `windowed_streamer` workflow (exact frame accounting). If the streamer is not available it falls back to a best-effort one-shot decode (less exact).

	Pushing weights to Hugging Face
	- Only push the small helper files and the `weights/` folder to your HF repo. Use Git LFS for large checkpoint files:

	```bash
	git lfs install
	git lfs track "weights/*"
	git add .gitattributes weights/ streaming_istft_head.py windowed_streamer.py mio_codec_wrapper.py requirements.txt
	git commit -m "Add weights + streaming helpers"
	git push
	```

	When calling `MioCodecModel.from_pretrained()` in the wrapper, you can pass the HF repo id (for example `PRIYANSHUDHAKED/Mio-Codec-Stream`) so the model code downloads the weights from the hub automatically.

	Troubleshooting
	- Import errors (e.g. missing `BloomPreTrainedModel`) usually come from incompatible `transformers` versions — use the version recommended by the model package or install inside a fresh venv.
	- If audio shows seams, increase `--right` lookahead or ensure `windowed_streamer.py` is present so exact frame trimming is used.
	- If playback is silent but files exist, inspect `out.wav` in a player; the wrapper may write raw float32 if `soundfile` is unavailable.

	Next steps / options
	- I can add a tiny `README` section that shows how to configure a vLLM or other LLM server to emit the expected websocket JSON format. Want that added?