Initial release

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,92 @@
+---
+language:
+- da
+license: cc-by-4.0
+pipeline_tag: automatic-speech-recognition
+library_name: mlx
+base_model: KlangAI/pianissimo-da
+base_model_relation: quantized
+tags:
+- mlx
+- speech
+- audio
+- fastconformer
+- tdt
+- danish
+---
+
+# Klang Pianissimo for Danish: MLX 8-bit
+
+A version of [Klang Pianissimo for Danish](https://huggingface.co/KlangAI/pianissimo-da), our Danish speech recognition model, for Macs with Apple silicon (M1 or later). It runs on the Mac's GPU using [MLX](https://github.com/ml-explore/mlx) and [parakeet-mlx](https://github.com/senstella/parakeet-mlx), without PyTorch or NeMo.
+
+This repository contains the 8-bit version, with almost the same accuracy as the full-size model at less than a third of the download size.
+
+## Versions
+
+| Repository | Weights | Download | FLEURS test | Speed | Memory, 5 min |
+|---|---|---:|---:|---:|---:|
+| [pianissimo-da](https://huggingface.co/KlangAI/pianissimo-da) | Original fp32 | 2.51 GB | **7.43** |  |  |
+| [pianissimo-da-mlx-8bit](https://huggingface.co/KlangAI/pianissimo-da-mlx-8bit) | 8-bit | 759 MB | 7.49 | 151× | **2.2 GB** |
+| [pianissimo-da-mlx-4bit](https://huggingface.co/KlangAI/pianissimo-da-mlx-4bit) | 4-bit | **528 MB** | 7.80 | 126× | **2.2 GB** |
+| [pianissimo-da-mlx](https://huggingface.co/KlangAI/pianissimo-da-mlx) | bf16 | 1.25 GB | 7.48 | 140× | 2.7 GB |
+| [pianissimo-da-mlx-fp32](https://huggingface.co/KlangAI/pianissimo-da-mlx-fp32) | fp32 | 2.51 GB | **7.43** | **173×** | 3.8 GB |
+
+The FLEURS column shows word error rate (WER, %) on all 930 Danish test clips, scored with our evaluation normaliser; lower is better. Speed is audio duration divided by processing time (RTFx), measured on a 30-second clip.
+Speed and memory were measured on the Swedish model, which has the same architecture, on an Apple M5 Pro GPU (24 GB). Memory is peak GPU memory use while transcribing a 5-minute recording.
+
+For other computers, use the [ONNX versions](https://huggingface.co/KlangAI/pianissimo-da-onnx).
+
+## Usage
+
+```bash
+pip install parakeet-mlx==0.5.2 huggingface_hub sentencepiece
+brew install ffmpeg   # to read audio files
+```
+
+```python
+import sys
+from huggingface_hub import snapshot_download
+
+path = snapshot_download("KlangAI/pianissimo-da-mlx-8bit")
+sys.path.append(path)
+import pianissimo_mlx
+
+model = pianissimo_mlx.load(path)
+result = pianissimo_mlx.transcribe(model, "audio.wav")
+print(result.text)
+```
+
+Recordings longer than 2 minutes are transcribed in 2-minute chunks with a 15-second overlap. Set `chunk_duration` to change the chunk length in seconds, or to `None` for a single pass. `transcribe` also accepts a 16 kHz mono NumPy array. Timestamps are available in `result.sentences` and `sentence.tokens`.
+
+Use the `pianissimo_mlx.py` loader included in this repository. It supports the 8-bit and 4-bit weights and reproduces the audio preprocessing used during training. Calling parakeet-mlx directly does not support these quantized weights and uses slightly different preprocessing, adding about 0.2 percentage points of WER.
+
+### Vocabulary
+
+Our implementation of NeMo's phrase boosting supports custom vocabulary and matches NeMo's results on the original model. List names and terms to help the model spell them correctly.
+
+```python
+vocabulary = ["Klang AI", "Pianissimo", "Folketinget", "Aarhus Universitet"]
+result = pianissimo_mlx.transcribe(model, "audio.wav", boost=vocabulary)
+```
+
+See the [main model card](https://huggingface.co/KlangAI/pianissimo-da#vocabulary) for vocabulary boosting with NeMo.
+
+## How they were made
+
+All versions are converted from the released checkpoint and retain its local attention (256 frames on each side). The 8-bit and 4-bit versions use MLX's built-in quantization for the encoder's linear layers, with groups of 64 weights for 8-bit and 32 for 4-bit. The remaining weights in those two versions are stored in bfloat16.
+
+FLEURS WER was measured with this code on Linux. In a separate Mac check, this version matched the normalised original fp32 reference on 187 of 200 FLEURS clips. See the [main model card](https://huggingface.co/KlangAI/pianissimo-da) for comparisons with other models.
+
+## License and citation
+
+Released under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), like the original model.
+
+```bibtex
+@misc{klang2026pianissimoda,
+  title = {Klang Pianissimo for Danish},
+  author = {{Klang}},
+  year = {2026},
+  howpublished = {Hugging Face model repository},
+  url = {https://huggingface.co/KlangAI/pianissimo-da}
+}
+```

manifest.json

@@ -0,0 +1,20 @@
+{
+ "artifact": "pianissimo-da-mlx-8bit",
+ "checkpoint_sha256": "ef346b91fc125926ed8a5a69091b591bf59d0bc32e2d5d197f112a2a0b12d41b",
+ "mlx": "0.32.2",
+ "quantization": {
+  "group_size": 64,
+  "bits": 8,
+  "modules": "nn.Linear in encoder.layers"
+ },
+ "files": {
+  "config.json": {
+   "bytes": 200943,
+   "sha256": "93895461c005ccae88cffbec31c7ad05b9f44635cf3958670123c53a370013c8"
+  },
+  "model.safetensors": {
+   "bytes": 758783094,
+   "sha256": "61b8fec4b0a9d134812eb59f4482ffcca9ec03877b97df65189236513072b4f1"
+  }
+ }
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:61b8fec4b0a9d134812eb59f4482ffcca9ec03877b97df65189236513072b4f1
+size 758783094

phrase_boost.py

@@ -0,0 +1,84 @@
+"""Phrase boosting for Klang Pianissimo. A numpy port of NeMo's GPU-PB boosting tree
+(ContextGraph and GPUBoostingTreeModel, NeMo 2.7, Apache 2.0)."""
+import math
+from collections import deque
+
+import numpy as np
+import sentencepiece as spm
+
+
+class _Node:
+    __slots__ = ("id", "children", "token_score", "node_score", "is_end", "fail")
+
+    def __init__(self, node_id, token_score, node_score, is_end):
+        self.id = node_id
+        self.children = {}
+        self.token_score = token_score
+        self.node_score = node_score
+        self.is_end = is_end
+        self.fail = None
+
+
+class PhraseBoost:
+    def __init__(self, phrases, tokenizer_path, vocab_size, context_score=1.0, depth_scaling=2.0):
+        sp = spm.SentencePieceProcessor(model_file=str(tokenizer_path))
+        self.vocab_size = vocab_size
+        self.root = _Node(0, 0.0, 0.0, False)
+        self.root.fail = self.root
+        self.nodes = [self.root]
+        for phrase in dict.fromkeys(phrases):
+            self._add(sp.encode(phrase), context_score, depth_scaling)
+        self._fill_fail()
+        self._cache = {}
+
+    def _add(self, tokens, context_score, depth_scaling):
+        node = self.root
+        for i, token in enumerate(tokens):
+            last = i == len(tokens) - 1
+            child = node.children.get(token)
+            if child is None:
+                token_score = context_score * depth_scaling + math.log(i + 1) if i > 0 else context_score
+                child = _Node(len(self.nodes), token_score, node.node_score + token_score, last)
+                node.children[token] = child
+                self.nodes.append(child)
+            else:
+                child.token_score = max(context_score, child.token_score)
+                child.node_score = node.node_score + child.token_score
+                child.is_end = last or child.is_end
+            node = child
+
+    def _fill_fail(self):
+        queue = deque()
+        for child in self.root.children.values():
+            child.fail = self.root
+            queue.append(child)
+        while queue:
+            node = queue.popleft()
+            for token, child in node.children.items():
+                fail = node.fail
+                while token not in fail.children and fail is not self.root:
+                    fail = fail.fail
+                child.fail = fail.children.get(token, self.root)
+                queue.append(child)
+
+    def scores(self, state):
+        """Bonus and next state for every token from `state`."""
+        if state not in self._cache:
+            bonus = np.zeros(self.vocab_size, dtype=np.float32)
+            nxt = np.full(self.vocab_size, -1, dtype=np.int64)
+            node, taken_back = self.nodes[state], 0.0
+            while True:
+                for token, child in node.children.items():
+                    if nxt[token] < 0:
+                        bonus[token] = taken_back + child.token_score
+                        nxt[token] = child.id
+                if node is self.root:
+                    unset = nxt < 0
+                    bonus[unset] = taken_back
+                    nxt[unset] = 0
+                    break
+                if not node.is_end:
+                    taken_back += node.fail.node_score - node.node_score
+                node = node.fail
+            self._cache[state] = (bonus, nxt)
+        return self._cache[state]

pianissimo_mlx.py

@@ -0,0 +1,147 @@
+"""Klang Pianissimo on Apple silicon with parakeet-mlx: loads the quantized versions, computes NeMo's
+audio features, splits long audio into chunks and supports phrase boosting.
+
+    model = pianissimo_mlx.load("KlangAI/pianissimo-da-mlx-8bit")
+    print(pianissimo_mlx.transcribe(model, "audio.wav", boost=["Klang AI", "Pianissimo", "RixVox", "Common Voice"]).text)
+"""
+import json
+import os
+import shutil
+import subprocess
+
+import mlx.core as mx
+import mlx.nn as nn
+import numpy as np
+from parakeet_mlx import tokenizer
+from parakeet_mlx.alignment import (AlignedToken, merge_longest_common_subsequence, merge_longest_contiguous,
+                                    sentences_to_result, tokens_to_sentences)
+from parakeet_mlx.utils import from_config
+
+from phrase_boost import PhraseBoost
+
+SAMPLE_RATE = 16000
+
+
+def load(repo_or_path):
+    """A Hugging Face repo id or a local directory with config.json and model.safetensors."""
+    path = repo_or_path
+    if not os.path.isdir(path):
+        from huggingface_hub import snapshot_download
+        path = snapshot_download(repo_or_path, allow_patterns=["config.json", "model.safetensors", "tokenizer.model"])
+    cfg = json.load(open(os.path.join(path, "config.json")))
+    model = from_config(cfg)
+    q = cfg.get("quantization")
+    if q:
+        nn.quantize(model, group_size=q["group_size"], bits=q["bits"],
+                    class_predicate=lambda p, m: isinstance(m, nn.Linear) and p.startswith("encoder.layers."))
+    model.load_weights(os.path.join(path, "model.safetensors"))
+    model.tokenizer_path = os.path.join(path, "tokenizer.model")
+    return model
+
+
+def read_audio(path):
+    """Any format ffmpeg reads, as 16 kHz mono float32."""
+    if shutil.which("ffmpeg") is None:
+        raise RuntimeError("reading audio files needs ffmpeg (brew install ffmpeg), or pass a 16 kHz numpy array")
+    out = subprocess.run(["ffmpeg", "-nostdin", "-v", "error", "-i", str(path), "-f", "s16le", "-ac", "1",
+                          "-ar", str(SAMPLE_RATE), "-"], capture_output=True, check=True).stdout
+    return np.frombuffer(out, np.int16).astype(np.float32) / 32768.0
+
+
+def features(audio, args):
+    """NeMo's log-mel featurizer, ported to MLX."""
+    n_fft, win = args.n_fft, int(args.window_size * args.sample_rate)
+    hop = int(args.window_stride * args.sample_rate)
+    x = mx.concatenate([audio[:1], audio[1:] - 0.97 * audio[:-1]])
+    x = mx.pad(x, [(n_fft // 2, n_fft // 2)])
+    frames = mx.as_strided(x, shape=((x.shape[0] - n_fft) // hop + 1, n_fft), strides=(hop, 1))
+    window = mx.array(np.pad(np.hanning(win), (n_fft - win) // 2).astype(np.float32))
+    power = mx.abs(mx.fft.rfft(frames * window)) ** 2
+    logmel = mx.log(mx.matmul(power, args._filterbanks.T.astype(mx.float32)) + 2 ** -24)
+    valid = audio.shape[0] // hop
+    logmel = logmel[:valid]
+    mean = logmel.mean(axis=0, keepdims=True)
+    std = mx.sqrt(((logmel - mean) ** 2).sum(axis=0, keepdims=True) / (valid - 1))
+    return mx.expand_dims((logmel - mean) / (std + 1e-5), 0)
+
+
+def transcribe(model, audio, chunk_duration=120.0, overlap_duration=15.0, boost=None, boost_alpha=1.0):
+    """audio is a path or a 16 kHz mono numpy array. Audio longer than chunk_duration seconds is
+    transcribed in overlapping chunks (None for one pass). boost lists phrases to spell as written."""
+    if isinstance(audio, (str, os.PathLike)):
+        audio = read_audio(audio)
+    audio = np.asarray(audio, dtype=np.float32)
+    tree = PhraseBoost(boost, model.tokenizer_path, vocab_size=len(model.vocabulary)) if boost else None
+    if chunk_duration is None or len(audio) <= chunk_duration * SAMPLE_RATE:
+        return _transcribe_piece(model, audio, tree, boost_alpha)
+
+    step = int((chunk_duration - overlap_duration) * SAMPLE_RATE)
+    size = int(chunk_duration * SAMPLE_RATE)
+    tokens = []
+    for start in range(0, len(audio), step):
+        result = _transcribe_piece(model, audio[start:start + size], tree, boost_alpha)
+        offset = start / SAMPLE_RATE
+        for sentence in result.sentences:
+            for token in sentence.tokens:
+                token.start += offset
+                token.end = token.start + token.duration
+        if not tokens:
+            tokens = result.tokens
+        else:
+            try:
+                tokens = merge_longest_contiguous(tokens, result.tokens, overlap_duration=overlap_duration)
+            except RuntimeError:
+                tokens = merge_longest_common_subsequence(tokens, result.tokens, overlap_duration=overlap_duration)
+        mx.clear_cache()
+        if start + size >= len(audio):
+            break
+    return sentences_to_result(tokens_to_sentences(tokens))
+
+
+def _transcribe_piece(model, audio, tree, alpha):
+    mel = features(mx.array(audio), model.preprocessor_config)
+    if tree is None:
+        return model.generate(mel)[0]
+    encoded, lengths = model.encoder(mel)
+    mx.eval(encoded, lengths)
+    return sentences_to_result(tokens_to_sentences(_boosted_greedy(model, encoded, int(lengths[0]), tree, alpha)))
+
+
+def _boosted_greedy(model, encoded, length, tree, alpha):
+    """parakeet-mlx's greedy TDT decoding with the phrase bonus added."""
+    blank = len(model.vocabulary)
+    hypothesis, last_token, hidden_state = [], None, None
+    boost_state, step, new_symbols = 0, 0, 0
+    while step < length:
+        decoder_out, (hidden, cell) = model.decoder(
+            mx.array([[last_token]]) if last_token is not None else None, hidden_state)
+        decoder_out = decoder_out.astype(encoded.dtype)
+        decoder_hidden = (hidden.astype(encoded.dtype), cell.astype(encoded.dtype))
+        joint_out = model.joint(encoded[:, step:step + 1], decoder_out)
+
+        token_logits = joint_out[0, 0, :, :blank + 1]
+        pred_token = int(mx.argmax(token_logits))
+        token_probs = mx.softmax(token_logits, axis=-1)
+        entropy = -mx.sum(token_probs * mx.log(token_probs + 1e-10), axis=-1)
+        confidence = float(1.0 - entropy / mx.log(mx.array(blank + 1, dtype=token_probs.dtype)))
+        decision = int(mx.argmax(joint_out[0, 0, :, blank + 1:]))
+
+        if pred_token != blank:
+            bonus, next_states = tree.scores(boost_state)
+            logits = np.array(token_logits[0, :blank].astype(mx.float32))
+            pred_token = int(np.argmax(logits + alpha * bonus))
+            boost_state = int(next_states[pred_token])
+            hypothesis.append(AlignedToken(
+                pred_token, start=step * model.time_ratio, duration=model.durations[decision] * model.time_ratio,
+                confidence=confidence, text=tokenizer.decode([pred_token], model.vocabulary)))
+            last_token = pred_token
+            hidden_state = decoder_hidden
+
+        step += model.durations[decision]
+        new_symbols += 1
+        if model.durations[decision] != 0:
+            new_symbols = 0
+        elif model.max_symbols is not None and model.max_symbols <= new_symbols:
+            step += 1
+            new_symbols = 0
+    return hypothesis