Download README.md from codebucketsolutions/edge-punct-casing-en-onnx: direct link, hf CLI and curl.
- Browser
- Download file 3.4 kB
-
https://huggingface.co/codebucketsolutions/edge-punct-casing-en-onnx/resolve/main/README.md
- Command line
-
hf download hf://codebucketsolutions/edge-punct-casing-en-onnx/README.md
-
curl -L -o README.md https://huggingface.co/codebucketsolutions/edge-punct-casing-en-onnx/resolve/main/README.md
license: apache-2.0
library_name: onnx
language:
- en
base_model: frankyoujian/Edge-Punct-Casing
tags:
- punctuation
- truecasing
- sherpa-onnx
- onnx
inference: false
Edge-Punct-Casing (English), ONNX, for the browser
English punctuation and word casing for transcripts that have neither: a CNN-BiLSTM with a 5,000-piece unigram vocabulary by Jian You (Cisco Systems), from the paper "A lightweight and efficient punctuation and word casing prediction model for on-device streaming ASR" (code, weights). It adds commas, full stops and question marks after words and predicts lower case, capitalised or all-capital words.
This repository holds the two files of the sherpa-onnx package of that model, so that a web page can fetch them
(the package is only published as a .tar.bz2 archive on GitHub releases, which a page cannot read). It is not
an official release of the author or of sherpa-onnx.
What was changed
Nothing in the files. They were taken from
sherpa-onnx-online-punct-en-2024-08-06.tar.bz2
(SHA-256 9f5e5a72c7d2829635bd074fce92b6bbd5b78da8a52e7ad8ed1be933f366b99d), which sherpa-onnx built from the
author's sherpa-onnx-cnn-bilstm-unigram-bpe-en.7z. Only the 8-bit model is copied; the archive's float32
model.onnx (29.7 MB) is left out.
| File | Bytes | SHA-256 |
|---|---|---|
model.int8.onnx |
7,490,500 | 9d611f445fe4a46186080fe161be6059d87d72eb88d3a8cb00c1a06e83a6067e |
bpe.vocab |
149,430 | e118b7ad88c54db562517df49e1cffd4836d166c34fb190fd311d7f34eb238f5 |
How it runs
The steps are sherpa-onnx's OnlinePunctuationCNNBiLSTMImpl and simple-sentencepiece:
- Split the text on ASCII white space. Cut each word, with
▁in front, into vocabulary pieces: the cut with the highest sum of the pieces' scores (float32), the shorter first piece on a tie,<unk>(0) for a byte no piece starts with. - Chunks of at most 200 tokens:
<s>(1), the pieces,</s>(2), padded with 0.valid_idsis 1 for<s>, the first piece of each word and</s>, else 0;label_lensis the number of ones. token_idsandvalid_idsint32[N, 200],label_lensint32[N]giveactive_case_logitsandactive_punct_logits, float32[words, 4]: one row per word, chunks one after the other.- Case (argmax): 0 as written, 1 all capitals, 2 first letter capital, 3 mixed (left as written). Punctuation
(argmax): 0 none, 1
,, 2., 3?, written after the word. Only ASCII letters change case.
A word longer than one chunk crashes sherpa-onnx 2.0.1; keep words under 190 bytes.
bolo runs it on its server through sherpa-onnx (punct-en-cnn-bilstm)
and in its browser demo with ONNX Runtime Web, with a TypeScript port of these steps that gives the same text as
sherpa-onnx on its test transcripts.
Licence and attribution
Apache-2.0 (see LICENSE, the licence of Edge-Punct-Casing
and of its weights on the Hugging Face Hub). Model and training by Jian You (Cisco Systems); ONNX packaging by the
sherpa-onnx project (Apache-2.0).