|
Download README.md from WhissleAI/arc-tagger: direct link, hf CLI and curl.
- Browser
- Download file 21.5 kB
-
https://huggingface.co/WhissleAI/arc-tagger/resolve/main/README.md
- Command line
-
hf download hf://WhissleAI/arc-tagger/README.md
-
curl -L -o README.md https://huggingface.co/WhissleAI/arc-tagger/resolve/main/README.md
21.5 kB
| language: | |
| - en | |
| license: cc-by-nc-4.0 | |
| library_name: pytorch | |
| pipeline_tag: token-classification | |
| tags: | |
| - streaming | |
| - asr | |
| - ner | |
| - pii | |
| - redaction | |
| - intent | |
| - dialogue-act | |
| # Arc Tagger (10M), v0.4.1 | |
| Arc Tagger is a **text-only streaming tagger** for live speech transcripts. It reads the word stream of any ASR (Deepgram, | |
| Speechmatics, Whisper, NeMo, your own ASR, or plain text). As each word arrives, it emits: | |
| * entity and number spans; | |
| * PII spans for redaction; | |
| * profanity and toxicity; | |
| * discourse acts, intents and safety flags, including prompt-injection / jailbreak attempts; | |
| * speaker roles; | |
| * events, such as speaker change, turn yield, sports, legal and meeting events; | |
| * content state; | |
| * affect: emotion, arousal, valence and intensity (experimental, text only). | |
| The model has about 10 M parameters and runs on CPU. Interim ASR hypotheses can be revised: only the revised words are | |
| re-processed. | |
| ## Usage terms | |
| * **Inference use only.** This repository provides the model and the inference code, nothing else. | |
| * **Non-commercial.** The weights and code are licensed under CC BY-NC 4.0 plus the Whissle terms (see [LICENSE](LICENSE)). | |
| * **Commercial licensing:** contact WhissleAI (ksingla@whissle.ai). | |
| ## Tag families | |
| 284 tags across 10 read-outs (365 `tag@readout` columns). `ontology/` holds the full definitions, and `config.json["columns"]` | |
| lists every column. | |
| | family | tags (examples) | read-out | | |
| |---|---|---| | |
| | Entities | `entity.mention.{person,organization,team,place,event_name,legal_ref,term,...}` (10) | word spans | | |
| | Numbers | `entity.number.{date,time_of_day,duration,count,money,percent,measure,score,clock,ordinal,id_pii,...}` (16), `entity.numeral` | word spans | | |
| | PII / redaction | `safety.pii.*` (11): `{private_name,phone_number,email,street_address,card_account_number,gov_id,date_of_birth,url_handle,public_contact}` spans; `safety.pii.{request,leak}` per utterance | spans and utterance | | |
| | Profanity / toxicity | `safety.profanity.{mild,strong}`, `safety.toxicity.*` (6) | spans and utterance | | |
| | Safety | `safety.jailbreak.attempt` (experimental), `safety.scam.*` (6), `safety.crisis.self_harm`, `safety.threat.escalation`, `safety.child.*` (2), `safety.solicitation.dating_romantic` | utterance | | |
| | Discourse act | `discourse.act.*` (7: statement, question, command_request, ...), `discourse.answer`, `discourse.backchannel`, `discourse.assistant.*` (2) | utterance | | |
| | Intent | 26 cross-domain `intent.*` (request_information, schedule_arrange, complain_criticize, ...), 60 assistant `discourse.intent.*` (alarm_set, weather_query, ...) | utterance | | |
| | Speaker role | `speaker.role.*` (19, including contact-centre agent and customer) | per speaker (needs speaker ids on the words) | | |
| | Events | `speaker.change`, `discourse.turn_yield`, `discourse.interruption`, `event.sports.*` (18), `event.legal.*` (3), `event.meeting.{decision,action_item}` | per word | | |
| | Content state | `content.mode` (8), `content.subject` (12), `content.game_state` (6), `content.legal_phase` (9), `content.meeting_phase` (6), `content.sports_focus` (7) | per word | | |
| | Affect (experimental) | `affect.emotion.*` (5), `affect.arousal.{low,medium,high}`, `affect.valence.{neg,neu,pos,mixed}`, `affect.intensity.{1,2,3}` | utterance | | |
| ## Quickstart | |
| ```bash | |
| pip install huggingface_hub | |
| huggingface-cli download WhissleAI/arc-tagger --local-dir arc-tagger | |
| cd arc-tagger | |
| pip install -r requirements.txt # CPU torch is enough: pip install torch --index-url https://download.pytorch.org/whl/cpu | |
| python example.py | |
| ``` | |
| ### Offline | |
| ```python | |
| import sys; sys.path.insert(0, "arc-tagger") # the arctag package ships in the repo | |
| from arctag import ArcTagger | |
| tg = ArcTagger.from_pretrained("arc-tagger") # a local dir, or the repo id "WhissleAI/arc-tagger" (the default) | |
| out = tg.tag("Hi, this is Maria from Northwind Bank. Can I get your phone number?", domain="talk") | |
| print(out["redacted_text"]) # Hi, this is [NAME] from Northwind Bank. ... | |
| print(out["spans"]) # entity / number / PII / profanity spans with probabilities | |
| print(out["utterances"]) # per-utterance intent, act, safety, role, state, affect | |
| ``` | |
| ### Streaming | |
| Push words as the ASR emits them. An interim hypothesis is replaced with `revise_from=k` (it replaces `words[k:]`). | |
| ```python | |
| st = tg.stream(domain="assistant") | |
| st.push_words(["set", "an", "alarm", "for"], final=False) # interim | |
| up = st.push_words(["set", "an", "alarm", "for", "seven", "thirty."], revise_from=0, final=True) | |
| up = st.endpoint() # end of utterance: utterance read-outs arrive in up["utterances"] | |
| res = st.finalize() # the full result (same schema as tag()) | |
| ``` | |
| ## Live ASR scripts (low latency) | |
| Two scripts connect a live ASR to the tagger. Each takes a wav file or the microphone, and prints the live tags and redacted text | |
| as JSON lines. The API key is read **only** from an environment variable. | |
| ```bash | |
| pip install websockets # plus sounddevice for --mic | |
| export DEEPGRAM_API_KEY=... | |
| python stream_deepgram.py --wav call.wav --domain talk | |
| python stream_deepgram.py --mic --domain assistant --seconds 30 | |
| export SPEECHMATICS_API_KEY=... | |
| python stream_speechmatics.py --wav call.wav --domain talk | |
| python stream_speechmatics.py --mic --domain meeting --url wss://eu2.rt.speechmatics.com/v2 | |
| ``` | |
| How provider messages map to the stream: | |
| | provider message | tagger call | | |
| |---|---| | |
| | Deepgram `Results`, `is_final=false` | `push_words(words, final=False, revise_from=n_final)` | | |
| | Deepgram `Results`, `is_final=true` | `push_words(words, final=True, revise_from=n_final)` | | |
| | Deepgram `speech_final=true` or `UtteranceEnd` | `endpoint()` | | |
| | Speechmatics `AddPartialTranscript` | `push_words(words, final=False, revise_from=n_final)` | | |
| | Speechmatics `AddTranscript` | `push_words(words, final=True, revise_from=n_final)` | | |
| | Speechmatics `EndOfUtterance` | `endpoint()` | | |
| * `n_final` is the number of words the provider has already finalised, so interim words never rewrite final ones. | |
| * An end of utterance that arrives while interim words are pending is applied right after those words become final. | |
| * Speaker ids from diarization (Deepgram `diarize=true`, Speechmatics `diarization: "speaker"`) are passed on as `spk`. | |
| They enable speaker roles. | |
| * Useful options: | |
| * `--no-interim`: print only final updates. | |
| * `--result out.json`: write the full result at the end. | |
| * `--tier T1`: delayed, more stable labels. | |
| * Each line is `{"type": "update", "source": "interim" | "final" | "endpoint", "spans", "events", "utterances", "running", "redacted_text"}`. | |
| The last line is `{"type": "done", ...}`. | |
| **Status.** | |
| * Deepgram: the adapter is tested offline against recorded-format message sequences. | |
| * Speechmatics: tested against the documented message format; not yet tested live. | |
| The fixtures and tests are in `tests/` (`python tests/test_adapters.py`). The adapters themselves (`arctag/asr.py`) have no | |
| network code, so you can reuse them with any client. | |
| ## Input | |
| `tag(x, domain="auto", tier="T0")` | |
| `x` can be any of: | |
| * a `str`; | |
| * a list of words; | |
| * a list of word dicts `{"w": str, "t0": s, "t1": s, "conf": float, "spk": str|int, "final": bool}`. Every field except `w` is optional. | |
| Offline, utterances end after words ending in `. ? !`. In streaming, call `endpoint()` at the ASR's end of utterance. | |
| `domain` is one of `auto | talk | meeting | court | assistant | sports | news`. | |
| * It selects the per-domain behaviour and thresholds. | |
| * `talk` covers contact centre, podcasts and chat. | |
| * In `sports` and `court`, the specific league or court type is detected from the words. | |
| * `auto` is weaker. **Set the domain when you know it.** | |
| Streaming: `push_words(words, final=True, revise_from=None, endpoint=False)`, `endpoint()`, `finalize()`. | |
| ## Output (JSON) | |
| ```jsonc | |
| { | |
| "meta": {"model": "arc-tagger", "version": "0.4.1", "domain": "talk", "fine_domain": "talk", "tier": "T0", | |
| "thresholds": "display + default"}, | |
| "words": ["Hi,", "this", "is", "Maria", ...], | |
| "spans": [ // word indices [start, end) | |
| {"tag": "safety.pii.private_name", "family": "pii", "start": 3, "end": 4, "text": "Maria", "p": 0.97, "thr": 0.42, | |
| "alt": {"tag": "...", "p": 0.01}, // runner-up type | |
| "ver": 0.99, // span verifier probability | |
| "src": "head"} | |
| ], | |
| "entities": [...], "pii": [...], "profanity_toxicity": [...], // spans split by family | |
| "redacted_text": "Hi, this is [NAME] from ...", // PII, profanity and toxicity spans masked (public_contact is not) | |
| "utterances": [ | |
| {"start": 0, "end": 8, "text": "...", | |
| "intent": ["intent.request_information"], "act": ["discourse.act.question"], "safety": ["safety.pii.request"], | |
| "role": [], "role_note": "speaker role needs diarization (no speaker id on words)", | |
| "item": [], "state": {"content.mode": ["content.mode.call_in"]}, | |
| "emotion": [{"tag": "affect.emotion.neutral", "p": 0.93, "thr": 0.5, "on": true}, // top tag of each affect group: | |
| {"tag": "affect.arousal.low", ...}, {"tag": "affect.valence.neu", ...}, {"tag": "affect.intensity.1", ...}], | |
| "scores": {"intent": [{"tag": "...", "p": 0.0, "thr": 0.0, "on": false}], "act": [...], ...}} // top-k with probabilities | |
| ], | |
| "events": [{"tag": "speaker.change", "at": 15, "word": "Yeah,", "p": 0.94, "thr": 0.3, "tier": "T0", "read": "ev_rec"}] | |
| } | |
| ``` | |
| * The label lists (`intent`, `act`, `safety`, ...) hold the tags at or above their threshold. | |
| * `scores` keeps the top-k with probabilities. | |
| * `emotion` holds the most likely tag of each affect group (emotion, arousal, valence, intensity) with its probability. | |
| Each streaming update (from `push_words` or `endpoint`) contains: | |
| * `n_words` and `changed_from`; | |
| * `spans`: the re-decoded window (replace your spans that end after its start); | |
| * `events`: new events only; | |
| * `utterances`: the utterances that just ended; | |
| * `running`: the live intent, act, safety and state of the open utterance. | |
| ## Latency tiers | |
| | tier | lookahead | use | | |
| |---|---|---| | |
| | T0 | 0 words | live display and zero-lag redaction (default) | | |
| | T1 | up to 3 words | revisions a few words later | | |
| | T2 | to the utterance end | final utterance labels | | |
| All three tiers are read from the same pass, so they add no compute. Utterance-level labels (intent, act, safety, role, affect) | |
| are read at the endpoint. | |
| **Measured CPU latency** (1 thread, one word per call, on a heavily shared server CPU; expect lower on an idle core): | |
| | | p50 | p95 | | |
| |---|---|---| | |
| | model step per word | 16.62 ms | 31.66 ms | | |
| | full `push_words()` call per word (span decoding and read-outs included) | 24.3 ms | 46.4 ms | | |
| ## Evaluation | |
| The internal numbers are on the Whissle internal held-out test set (multi-domain: contact centre, assistant, meetings, news, | |
| talk, sports and courts; ASR word streams). Its labels are internal, so these numbers are not comparable to public | |
| leaderboards. Public benchmark results follow below. | |
| ### Whissle internal held-out test set, offline pass (609 sessions, 249,866 words) | |
| | metric | value | | |
| |---|---| | |
| | primary score (spans, AP, events, exclusive groups, PII leak) | 0.7694 | | |
| | span exact F1 (families, mean over tiers T0 to T2) | 0.7090 | | |
| | macro AP over tags (mean over tiers) | 0.7562 | | |
| | macro AP over tags, T0 / T1 / T2 | 0.7444 / 0.7632 / 0.7611 | | |
| | macro AUC over tags, T0 / T1 / T2 | 0.9764 / 0.9768 / 0.9769 | | |
| | event F1 (±2 words, T0, events with at least 20 positives) | 0.8961 | | |
| | exclusive-group macro F1 (T0) | 0.6356 | | |
| | PII words shown unmasked at first display (T0) | 1.23 % (non-PII words masked: 0.889 %) | | |
| | family | span exact F1, T0 | span exact F1, T2 | AP, T0 | | |
| |---|---|---|---| | |
| | PII (`safety.pii`) | 0.7314 | 0.7666 | 0.8750 | | |
| | Profanity (`safety.profanity`) | 0.8536 | 0.8586 | 0.9717 | | |
| | Numbers (`entity.number`) | 0.6865 | 0.7318 | 0.7618 | | |
| | Entities (`entity.mention`) | 0.5093 | 0.5099 | 0.5773 | | |
| | Speaker change (event) | F1 0.9688 | | 0.9953 | | |
| | Turn yield (event) | | | 0.7864 | | |
| | Speaker role (`speaker.role`) | | | 0.8210 | | |
| | Assistant intent (`discourse.intent`) | top-1 0.6395, top-3 0.8172 | | 0.6770 | | |
| | Solicitation (`safety.solicitation`) | | | 0.9475 | | |
| | Content mode (`content.mode`) | | | 0.9929 | | |
| Span F1 is for the span-head decode, which the package uses for the PII, profanity, number and entity families. | |
| ### Whissle internal held-out test set, streaming first display (35 sessions, 36,662 words) | |
| The sessions are replayed through the streaming engine as a live-ASR message sequence (interim words, revisions, finals, | |
| endpoints). Each label is scored as first shown to the user (T0), with the default thresholds. | |
| | read-out (streaming, T0 first display) | precision | recall | F1 | positives | | |
| |---|---|---|---|---| | |
| | spans (word level, all span families) | 0.758 | 0.702 | 0.729 | 3,516 | | |
| | intents (utterance) | 0.681 | 0.579 | 0.626 | 368 | | |
| | turn events (speaker change, turn yield, ...) | 0.770 | 0.377 | 0.506 | 1,668 | | |
| | speaker roles | 0.625 | 0.714 | 0.667 | 21 | | |
| | utterance safety flags | 0.818 | 1.000 | 0.900 | 9 | | |
| | discourse acts (utterance) | 0.346 | 0.450 | 0.391 | 40 | | |
| | span family (streaming, span exact F1) | T0 | T1 | T2 | | |
| |---|---|---|---| | |
| | Numbers (`entity.number`) | 0.5899 | 0.6827 | 0.6667 | | |
| | Entities (`entity.mention`) | 0.4151 | 0.4344 | 0.4689 | | |
| The speaker-role, act and safety rows have few positives (21, 40 and 9). | |
| ### Public safety benchmarks | |
| Whole-utterance mode, CPU, scored on the public test splits. Arc Tagger's utterance score is a logistic-regression | |
| combination of its safety-tag probabilities, fitted on a separate development split of the same benchmark (OpenAI | |
| Moderation: the other half of the 1,680 prompts; ToxicChat: a development subset disjoint from the test split). The other | |
| models were run locally on the same rows. 95 % bootstrap confidence intervals in brackets. | |
| | benchmark | metric | Arc Tagger 0.4.1 | Detoxify (original) | Detoxify (unbiased) | s-nlp RoBERTa toxicity | | |
| |---|---|---|---|---|---| | |
| | OpenAI Moderation (test half: 840 prompts, 261 flagged) | AUPRC | 0.726 [0.669, 0.779] | 0.783 | 0.730 | 0.790 | | |
| | ToxicChat 0124 test (5,083 prompts, 362 toxic) | AUPRC | 0.518 [0.470, 0.574] | 0.402 | 0.395 | 0.496 | | |
| | benchmark | metric | Arc Tagger 0.4.1 | | |
| |---|---|---| | |
| | SemEval-2021 Task 5 Toxic Spans (full test, 2,000 texts) | character F1 | 0.531 [0.513, 0.549] | | |
| SemEval toxic spans are read from the toxicity and profanity span scores, with one score threshold chosen on the SemEval | |
| trial split. | |
| ## ONNX / C runtime | |
| `onnx/` holds the same model exported for the T0 read-outs: `arctag.onnx` (fp32) and `arctag_int8.onnx` (int8), with | |
| `arctag_meta.json` (read-out columns, thresholds, domain profiles), `tokenizer.json` and `SHA256SUMS`. A dependency-light C | |
| runtime that loads these files is at [github.com/WhissleAI/arc-tagger-c](https://github.com/WhissleAI/arc-tagger-c). | |
| `arctag_int8.onnx` is a **weight-only int8** graph: the weight tables are stored as int8 with one scale per column, and | |
| activations are never quantised (compute stays in float). It uses standard ONNX operators only, streams exactly like the | |
| offline pass, and uses the same thresholds as the fp32 graph. The C runtime's native backend keeps the tables in int8 in | |
| memory; under ONNX Runtime they become float tables at load. If you run the graph with ONNX Runtime directly, set the graph | |
| optimization level to basic (`GraphOptimizationLevel.ORT_ENABLE_BASIC`): the default level fuses the int8 weights into | |
| int8 matrix products with quantised activations, and the scores move slightly. | |
| | | fp32 | int8 (weight-only) | | |
| |---|---|---| | |
| | file size | 34.0 MB | 9.4 MB | | |
| | C runtime, native backend: resident memory (RSS) / streaming p50 per word (1 thread) | 72 MB / 6.2 ms | 26 MB / 2.9 ms | | |
| | ONNX Runtime: resident memory (RSS) | 113 MB | 118 MB (tables become float at load) | | |
| | span F1 of int8 against fp32 output (Whissle internal held-out test set) | | 0.986 (toxicity spans 0.918) | | |
| | span F1 on the Whissle internal held-out test set (1,286 reference spans) | 0.544 | 0.543 | | |
| ## Kids / game domain (C runtime) | |
| `onnx/arctag_kids.json` holds the utterance thresholds of the `kids` domain (alias `game`), for short spoken turns of | |
| children in an app or game. The [C runtime](https://github.com/WhissleAI/arc-tagger-c) (0.5.0 and later; built-in copy of | |
| the same thresholds) and its Python reference apply them together with a guard layer: profanity, insult and threat also | |
| appear as utterance safety tags; `safety.pii.leak` / `safety.pii.request` need evidence in the words; self-harm, grooming, | |
| sexual-content and jailbreak backstops; and a `command_context` option so in-game commands ("kill the heaters") raise | |
| nothing. See `docs/KIDS.md` in that repository. | |
| Calibration on a held-out set of 400 short kid turns (synthetic; benign and command turns included), utterance level, int8: | |
| | tag | precision | recall | F1 | support | | |
| |---|---|---|---|---| | |
| | `safety.profanity` (either severity) | 1.00 | 1.00 | 1.00 | 46 | | |
| | `safety.profanity.strong` | 1.00 | 1.00 | 1.00 | 26 | | |
| | `safety.profanity.mild` | 1.00 | 1.00 | 1.00 | 20 | | |
| | `safety.toxicity.insult_harassment` | 1.00 | 0.73 | 0.85 | 30 | | |
| | `safety.toxicity.threat_violence` | 0.92 | 0.73 | 0.81 | 15 | | |
| | `safety.crisis.self_harm` | 1.00 | 0.51 | 0.68 | 35 | | |
| | `safety.child.grooming_cue` | 1.00 | 0.53 | 0.70 | 30 | | |
| | `safety.child.inappropriate` | 0.80 | 0.50 | 0.62 | 8 | | |
| | `safety.pii.leak` | 1.00 | 1.00 | 1.00 | 22 | | |
| | `safety.pii.request` | 1.00 | 0.79 | 0.88 | 14 | | |
| | `safety.jailbreak.attempt` | 1.00 | 0.70 | 0.82 | 10 | | |
| Benign turns (198, including 40 cockpit commands): any safety tag 0.5 %, self-harm / grooming 0.0 %, PII 0.0 %, jailbreak 0.0 %. Exact tag-set match over all 400 turns: 0.863. | |
| Self-harm, grooming and inappropriate content miss some indirect phrasings. Use them as review triggers for a caring | |
| adult or a moderator, not as the only safety net. | |
| ## Files | |
| | file | what | | |
| |---|---| | |
| | `model.safetensors` | weights (fp32, 45 MB) | | |
| | `config.json` | read-out columns, ontology id, default thresholds and the span families decoded by the span head | | |
| | `tok/bpe8k/` | tokenizer | | |
| | `ontology/` | tag ontology, text-tag table, the ontology extension (contact-centre roles, jailbreak, affect valence and intensity), tag-definition embeddings | | |
| | `thresholds/display.json` | display thresholds per tag, read-out and domain (tags without an entry use the default thresholds in `config.json`) | | |
| | `engine_settings.json` | the engine's display floors and domain mapping (for reference) | | |
| | `arctag/` | Python package: `ArcTagger`, `driver.py` (session to JSON), `asr.py` (live ASR adapters), `live.py` (audio input), `_core/` (streaming engine) | | |
| | `stream_deepgram.py`, `stream_speechmatics.py` | live ASR scripts | | |
| | `tests/` | adapter tests and provider message fixtures | | |
| | `example.py` | quickstart | | |
| | `LICENSE`, `NOTICE` | licence (CC BY-NC 4.0 + Whissle terms) and copyright notice | | |
| | `onnx/` | fp32 and int8 ONNX exports, their metadata, tokenizer and checksums (see ONNX / C runtime); `arctag_kids.json`: kids-domain thresholds | | |
| | `MANIFEST.sha256` | checksums | | |
| ## Limitations | |
| * **English only.** Non-English input gives meaningless output. | |
| * Text only, with no audio cues. The model cannot detect speaker identity, emotion in the voice or vocal events. | |
| **Speaker roles need speaker ids** (`spk`) on the words, for example from ASR diarization. | |
| * The internal test set above has no toxicity spans; the toxicity numbers on this card are the public benchmarks above. | |
| On OpenAI Moderation the tagger is behind the dedicated toxicity classifiers. Validate toxicity detection on your own | |
| data before relying on it. | |
| * Entity mention spans (`entity.mention`) are the weakest span family (exact F1 about 0.51 on the test set). | |
| * Spans shown at first display (T0, often on interim ASR words) are less accurate than the same spans a few words later (T1) | |
| or at the utterance end (T2). Use `tier="T1"` where a delay of up to 3 words is acceptable. | |
| * Utterance-level safety, jailbreak and scam flags can fire on short or out-of-domain utterances, and on utterances that | |
| follow a flagged one. For anything consequential, use `scores` (p) and your own policy. | |
| * The `safety.pii.leak` and `safety.scam.*` alerts are tuned for recall and can fire on benign contact-centre text, such | |
| as an agent's greeting or a caller giving their own details. Use the PII spans for redaction, and the `scores` (p) with | |
| your own threshold if false alerts matter. | |
| * Affect (emotion, arousal, valence, intensity) comes from the words only and is experimental; this card reports no affect | |
| evaluation. | |
| * The `auto` domain is weaker than setting the domain. Some sports events fire loosely outside sports. | |
| * Redaction masks the decoded spans. It is not a compliance guarantee: review it for your use case. | |
| * The default thresholds favour precision. Tune them for your traffic if you can. | |
| * The Python package runs fp32 on CPU; the ONNX exports cover the T0 read-outs only. | |
| ## Licence | |
| The weights and the code, including all inference code: CC BY-NC 4.0 plus the Whissle terms in [LICENSE](LICENSE). | |
| * Non-commercial use only. | |
| * Commercial use needs a separate written licence from Whissle Inc.: ksingla@whissle.ai. | |
| This covers every file in the repository, including `arctag/_core/` and `onnx/` (see [NOTICE](NOTICE)). | |