arc-tagger / README.md
ksingla025's picture
Kids / game domain thresholds (onnx/arctag_kids.json) and calibration table
5004492 verified
|
Raw History Blame Contribute Delete
21.5 kB
---
language:
- en
license: cc-by-nc-4.0
library_name: pytorch
pipeline_tag: token-classification
tags:
- streaming
- asr
- ner
- pii
- redaction
- intent
- dialogue-act
---
# Arc Tagger (10M), v0.4.1
Arc Tagger is a **text-only streaming tagger** for live speech transcripts. It reads the word stream of any ASR (Deepgram,
Speechmatics, Whisper, NeMo, your own ASR, or plain text). As each word arrives, it emits:
* entity and number spans;
* PII spans for redaction;
* profanity and toxicity;
* discourse acts, intents and safety flags, including prompt-injection / jailbreak attempts;
* speaker roles;
* events, such as speaker change, turn yield, sports, legal and meeting events;
* content state;
* affect: emotion, arousal, valence and intensity (experimental, text only).
The model has about 10 M parameters and runs on CPU. Interim ASR hypotheses can be revised: only the revised words are
re-processed.
## Usage terms
* **Inference use only.** This repository provides the model and the inference code, nothing else.
* **Non-commercial.** The weights and code are licensed under CC BY-NC 4.0 plus the Whissle terms (see [LICENSE](LICENSE)).
* **Commercial licensing:** contact WhissleAI (ksingla@whissle.ai).
## Tag families
284 tags across 10 read-outs (365 `tag@readout` columns). `ontology/` holds the full definitions, and `config.json["columns"]`
lists every column.
| family | tags (examples) | read-out |
|---|---|---|
| Entities | `entity.mention.{person,organization,team,place,event_name,legal_ref,term,...}` (10) | word spans |
| Numbers | `entity.number.{date,time_of_day,duration,count,money,percent,measure,score,clock,ordinal,id_pii,...}` (16), `entity.numeral` | word spans |
| PII / redaction | `safety.pii.*` (11): `{private_name,phone_number,email,street_address,card_account_number,gov_id,date_of_birth,url_handle,public_contact}` spans; `safety.pii.{request,leak}` per utterance | spans and utterance |
| Profanity / toxicity | `safety.profanity.{mild,strong}`, `safety.toxicity.*` (6) | spans and utterance |
| Safety | `safety.jailbreak.attempt` (experimental), `safety.scam.*` (6), `safety.crisis.self_harm`, `safety.threat.escalation`, `safety.child.*` (2), `safety.solicitation.dating_romantic` | utterance |
| Discourse act | `discourse.act.*` (7: statement, question, command_request, ...), `discourse.answer`, `discourse.backchannel`, `discourse.assistant.*` (2) | utterance |
| Intent | 26 cross-domain `intent.*` (request_information, schedule_arrange, complain_criticize, ...), 60 assistant `discourse.intent.*` (alarm_set, weather_query, ...) | utterance |
| Speaker role | `speaker.role.*` (19, including contact-centre agent and customer) | per speaker (needs speaker ids on the words) |
| Events | `speaker.change`, `discourse.turn_yield`, `discourse.interruption`, `event.sports.*` (18), `event.legal.*` (3), `event.meeting.{decision,action_item}` | per word |
| Content state | `content.mode` (8), `content.subject` (12), `content.game_state` (6), `content.legal_phase` (9), `content.meeting_phase` (6), `content.sports_focus` (7) | per word |
| Affect (experimental) | `affect.emotion.*` (5), `affect.arousal.{low,medium,high}`, `affect.valence.{neg,neu,pos,mixed}`, `affect.intensity.{1,2,3}` | utterance |
## Quickstart
```bash
pip install huggingface_hub
huggingface-cli download WhissleAI/arc-tagger --local-dir arc-tagger
cd arc-tagger
pip install -r requirements.txt # CPU torch is enough: pip install torch --index-url https://download.pytorch.org/whl/cpu
python example.py
```
### Offline
```python
import sys; sys.path.insert(0, "arc-tagger") # the arctag package ships in the repo
from arctag import ArcTagger
tg = ArcTagger.from_pretrained("arc-tagger") # a local dir, or the repo id "WhissleAI/arc-tagger" (the default)
out = tg.tag("Hi, this is Maria from Northwind Bank. Can I get your phone number?", domain="talk")
print(out["redacted_text"]) # Hi, this is [NAME] from Northwind Bank. ...
print(out["spans"]) # entity / number / PII / profanity spans with probabilities
print(out["utterances"]) # per-utterance intent, act, safety, role, state, affect
```
### Streaming
Push words as the ASR emits them. An interim hypothesis is replaced with `revise_from=k` (it replaces `words[k:]`).
```python
st = tg.stream(domain="assistant")
st.push_words(["set", "an", "alarm", "for"], final=False) # interim
up = st.push_words(["set", "an", "alarm", "for", "seven", "thirty."], revise_from=0, final=True)
up = st.endpoint() # end of utterance: utterance read-outs arrive in up["utterances"]
res = st.finalize() # the full result (same schema as tag())
```
## Live ASR scripts (low latency)
Two scripts connect a live ASR to the tagger. Each takes a wav file or the microphone, and prints the live tags and redacted text
as JSON lines. The API key is read **only** from an environment variable.
```bash
pip install websockets # plus sounddevice for --mic
export DEEPGRAM_API_KEY=...
python stream_deepgram.py --wav call.wav --domain talk
python stream_deepgram.py --mic --domain assistant --seconds 30
export SPEECHMATICS_API_KEY=...
python stream_speechmatics.py --wav call.wav --domain talk
python stream_speechmatics.py --mic --domain meeting --url wss://eu2.rt.speechmatics.com/v2
```
How provider messages map to the stream:
| provider message | tagger call |
|---|---|
| Deepgram `Results`, `is_final=false` | `push_words(words, final=False, revise_from=n_final)` |
| Deepgram `Results`, `is_final=true` | `push_words(words, final=True, revise_from=n_final)` |
| Deepgram `speech_final=true` or `UtteranceEnd` | `endpoint()` |
| Speechmatics `AddPartialTranscript` | `push_words(words, final=False, revise_from=n_final)` |
| Speechmatics `AddTranscript` | `push_words(words, final=True, revise_from=n_final)` |
| Speechmatics `EndOfUtterance` | `endpoint()` |
* `n_final` is the number of words the provider has already finalised, so interim words never rewrite final ones.
* An end of utterance that arrives while interim words are pending is applied right after those words become final.
* Speaker ids from diarization (Deepgram `diarize=true`, Speechmatics `diarization: "speaker"`) are passed on as `spk`.
They enable speaker roles.
* Useful options:
* `--no-interim`: print only final updates.
* `--result out.json`: write the full result at the end.
* `--tier T1`: delayed, more stable labels.
* Each line is `{"type": "update", "source": "interim" | "final" | "endpoint", "spans", "events", "utterances", "running", "redacted_text"}`.
The last line is `{"type": "done", ...}`.
**Status.**
* Deepgram: the adapter is tested offline against recorded-format message sequences.
* Speechmatics: tested against the documented message format; not yet tested live.
The fixtures and tests are in `tests/` (`python tests/test_adapters.py`). The adapters themselves (`arctag/asr.py`) have no
network code, so you can reuse them with any client.
## Input
`tag(x, domain="auto", tier="T0")`
`x` can be any of:
* a `str`;
* a list of words;
* a list of word dicts `{"w": str, "t0": s, "t1": s, "conf": float, "spk": str|int, "final": bool}`. Every field except `w` is optional.
Offline, utterances end after words ending in `. ? !`. In streaming, call `endpoint()` at the ASR's end of utterance.
`domain` is one of `auto | talk | meeting | court | assistant | sports | news`.
* It selects the per-domain behaviour and thresholds.
* `talk` covers contact centre, podcasts and chat.
* In `sports` and `court`, the specific league or court type is detected from the words.
* `auto` is weaker. **Set the domain when you know it.**
Streaming: `push_words(words, final=True, revise_from=None, endpoint=False)`, `endpoint()`, `finalize()`.
## Output (JSON)
```jsonc
{
"meta": {"model": "arc-tagger", "version": "0.4.1", "domain": "talk", "fine_domain": "talk", "tier": "T0",
"thresholds": "display + default"},
"words": ["Hi,", "this", "is", "Maria", ...],
"spans": [ // word indices [start, end)
{"tag": "safety.pii.private_name", "family": "pii", "start": 3, "end": 4, "text": "Maria", "p": 0.97, "thr": 0.42,
"alt": {"tag": "...", "p": 0.01}, // runner-up type
"ver": 0.99, // span verifier probability
"src": "head"}
],
"entities": [...], "pii": [...], "profanity_toxicity": [...], // spans split by family
"redacted_text": "Hi, this is [NAME] from ...", // PII, profanity and toxicity spans masked (public_contact is not)
"utterances": [
{"start": 0, "end": 8, "text": "...",
"intent": ["intent.request_information"], "act": ["discourse.act.question"], "safety": ["safety.pii.request"],
"role": [], "role_note": "speaker role needs diarization (no speaker id on words)",
"item": [], "state": {"content.mode": ["content.mode.call_in"]},
"emotion": [{"tag": "affect.emotion.neutral", "p": 0.93, "thr": 0.5, "on": true}, // top tag of each affect group:
{"tag": "affect.arousal.low", ...}, {"tag": "affect.valence.neu", ...}, {"tag": "affect.intensity.1", ...}],
"scores": {"intent": [{"tag": "...", "p": 0.0, "thr": 0.0, "on": false}], "act": [...], ...}} // top-k with probabilities
],
"events": [{"tag": "speaker.change", "at": 15, "word": "Yeah,", "p": 0.94, "thr": 0.3, "tier": "T0", "read": "ev_rec"}]
}
```
* The label lists (`intent`, `act`, `safety`, ...) hold the tags at or above their threshold.
* `scores` keeps the top-k with probabilities.
* `emotion` holds the most likely tag of each affect group (emotion, arousal, valence, intensity) with its probability.
Each streaming update (from `push_words` or `endpoint`) contains:
* `n_words` and `changed_from`;
* `spans`: the re-decoded window (replace your spans that end after its start);
* `events`: new events only;
* `utterances`: the utterances that just ended;
* `running`: the live intent, act, safety and state of the open utterance.
## Latency tiers
| tier | lookahead | use |
|---|---|---|
| T0 | 0 words | live display and zero-lag redaction (default) |
| T1 | up to 3 words | revisions a few words later |
| T2 | to the utterance end | final utterance labels |
All three tiers are read from the same pass, so they add no compute. Utterance-level labels (intent, act, safety, role, affect)
are read at the endpoint.
**Measured CPU latency** (1 thread, one word per call, on a heavily shared server CPU; expect lower on an idle core):
| | p50 | p95 |
|---|---|---|
| model step per word | 16.62 ms | 31.66 ms |
| full `push_words()` call per word (span decoding and read-outs included) | 24.3 ms | 46.4 ms |
## Evaluation
The internal numbers are on the Whissle internal held-out test set (multi-domain: contact centre, assistant, meetings, news,
talk, sports and courts; ASR word streams). Its labels are internal, so these numbers are not comparable to public
leaderboards. Public benchmark results follow below.
### Whissle internal held-out test set, offline pass (609 sessions, 249,866 words)
| metric | value |
|---|---|
| primary score (spans, AP, events, exclusive groups, PII leak) | 0.7694 |
| span exact F1 (families, mean over tiers T0 to T2) | 0.7090 |
| macro AP over tags (mean over tiers) | 0.7562 |
| macro AP over tags, T0 / T1 / T2 | 0.7444 / 0.7632 / 0.7611 |
| macro AUC over tags, T0 / T1 / T2 | 0.9764 / 0.9768 / 0.9769 |
| event F1 (±2 words, T0, events with at least 20 positives) | 0.8961 |
| exclusive-group macro F1 (T0) | 0.6356 |
| PII words shown unmasked at first display (T0) | 1.23 % (non-PII words masked: 0.889 %) |
| family | span exact F1, T0 | span exact F1, T2 | AP, T0 |
|---|---|---|---|
| PII (`safety.pii`) | 0.7314 | 0.7666 | 0.8750 |
| Profanity (`safety.profanity`) | 0.8536 | 0.8586 | 0.9717 |
| Numbers (`entity.number`) | 0.6865 | 0.7318 | 0.7618 |
| Entities (`entity.mention`) | 0.5093 | 0.5099 | 0.5773 |
| Speaker change (event) | F1 0.9688 | | 0.9953 |
| Turn yield (event) | | | 0.7864 |
| Speaker role (`speaker.role`) | | | 0.8210 |
| Assistant intent (`discourse.intent`) | top-1 0.6395, top-3 0.8172 | | 0.6770 |
| Solicitation (`safety.solicitation`) | | | 0.9475 |
| Content mode (`content.mode`) | | | 0.9929 |
Span F1 is for the span-head decode, which the package uses for the PII, profanity, number and entity families.
### Whissle internal held-out test set, streaming first display (35 sessions, 36,662 words)
The sessions are replayed through the streaming engine as a live-ASR message sequence (interim words, revisions, finals,
endpoints). Each label is scored as first shown to the user (T0), with the default thresholds.
| read-out (streaming, T0 first display) | precision | recall | F1 | positives |
|---|---|---|---|---|
| spans (word level, all span families) | 0.758 | 0.702 | 0.729 | 3,516 |
| intents (utterance) | 0.681 | 0.579 | 0.626 | 368 |
| turn events (speaker change, turn yield, ...) | 0.770 | 0.377 | 0.506 | 1,668 |
| speaker roles | 0.625 | 0.714 | 0.667 | 21 |
| utterance safety flags | 0.818 | 1.000 | 0.900 | 9 |
| discourse acts (utterance) | 0.346 | 0.450 | 0.391 | 40 |
| span family (streaming, span exact F1) | T0 | T1 | T2 |
|---|---|---|---|
| Numbers (`entity.number`) | 0.5899 | 0.6827 | 0.6667 |
| Entities (`entity.mention`) | 0.4151 | 0.4344 | 0.4689 |
The speaker-role, act and safety rows have few positives (21, 40 and 9).
### Public safety benchmarks
Whole-utterance mode, CPU, scored on the public test splits. Arc Tagger's utterance score is a logistic-regression
combination of its safety-tag probabilities, fitted on a separate development split of the same benchmark (OpenAI
Moderation: the other half of the 1,680 prompts; ToxicChat: a development subset disjoint from the test split). The other
models were run locally on the same rows. 95 % bootstrap confidence intervals in brackets.
| benchmark | metric | Arc Tagger 0.4.1 | Detoxify (original) | Detoxify (unbiased) | s-nlp RoBERTa toxicity |
|---|---|---|---|---|---|
| OpenAI Moderation (test half: 840 prompts, 261 flagged) | AUPRC | 0.726 [0.669, 0.779] | 0.783 | 0.730 | 0.790 |
| ToxicChat 0124 test (5,083 prompts, 362 toxic) | AUPRC | 0.518 [0.470, 0.574] | 0.402 | 0.395 | 0.496 |
| benchmark | metric | Arc Tagger 0.4.1 |
|---|---|---|
| SemEval-2021 Task 5 Toxic Spans (full test, 2,000 texts) | character F1 | 0.531 [0.513, 0.549] |
SemEval toxic spans are read from the toxicity and profanity span scores, with one score threshold chosen on the SemEval
trial split.
## ONNX / C runtime
`onnx/` holds the same model exported for the T0 read-outs: `arctag.onnx` (fp32) and `arctag_int8.onnx` (int8), with
`arctag_meta.json` (read-out columns, thresholds, domain profiles), `tokenizer.json` and `SHA256SUMS`. A dependency-light C
runtime that loads these files is at [github.com/WhissleAI/arc-tagger-c](https://github.com/WhissleAI/arc-tagger-c).
`arctag_int8.onnx` is a **weight-only int8** graph: the weight tables are stored as int8 with one scale per column, and
activations are never quantised (compute stays in float). It uses standard ONNX operators only, streams exactly like the
offline pass, and uses the same thresholds as the fp32 graph. The C runtime's native backend keeps the tables in int8 in
memory; under ONNX Runtime they become float tables at load. If you run the graph with ONNX Runtime directly, set the graph
optimization level to basic (`GraphOptimizationLevel.ORT_ENABLE_BASIC`): the default level fuses the int8 weights into
int8 matrix products with quantised activations, and the scores move slightly.
| | fp32 | int8 (weight-only) |
|---|---|---|
| file size | 34.0 MB | 9.4 MB |
| C runtime, native backend: resident memory (RSS) / streaming p50 per word (1 thread) | 72 MB / 6.2 ms | 26 MB / 2.9 ms |
| ONNX Runtime: resident memory (RSS) | 113 MB | 118 MB (tables become float at load) |
| span F1 of int8 against fp32 output (Whissle internal held-out test set) | | 0.986 (toxicity spans 0.918) |
| span F1 on the Whissle internal held-out test set (1,286 reference spans) | 0.544 | 0.543 |
## Kids / game domain (C runtime)
`onnx/arctag_kids.json` holds the utterance thresholds of the `kids` domain (alias `game`), for short spoken turns of
children in an app or game. The [C runtime](https://github.com/WhissleAI/arc-tagger-c) (0.5.0 and later; built-in copy of
the same thresholds) and its Python reference apply them together with a guard layer: profanity, insult and threat also
appear as utterance safety tags; `safety.pii.leak` / `safety.pii.request` need evidence in the words; self-harm, grooming,
sexual-content and jailbreak backstops; and a `command_context` option so in-game commands ("kill the heaters") raise
nothing. See `docs/KIDS.md` in that repository.
Calibration on a held-out set of 400 short kid turns (synthetic; benign and command turns included), utterance level, int8:
| tag | precision | recall | F1 | support |
|---|---|---|---|---|
| `safety.profanity` (either severity) | 1.00 | 1.00 | 1.00 | 46 |
| `safety.profanity.strong` | 1.00 | 1.00 | 1.00 | 26 |
| `safety.profanity.mild` | 1.00 | 1.00 | 1.00 | 20 |
| `safety.toxicity.insult_harassment` | 1.00 | 0.73 | 0.85 | 30 |
| `safety.toxicity.threat_violence` | 0.92 | 0.73 | 0.81 | 15 |
| `safety.crisis.self_harm` | 1.00 | 0.51 | 0.68 | 35 |
| `safety.child.grooming_cue` | 1.00 | 0.53 | 0.70 | 30 |
| `safety.child.inappropriate` | 0.80 | 0.50 | 0.62 | 8 |
| `safety.pii.leak` | 1.00 | 1.00 | 1.00 | 22 |
| `safety.pii.request` | 1.00 | 0.79 | 0.88 | 14 |
| `safety.jailbreak.attempt` | 1.00 | 0.70 | 0.82 | 10 |
Benign turns (198, including 40 cockpit commands): any safety tag 0.5 %, self-harm / grooming 0.0 %, PII 0.0 %, jailbreak 0.0 %. Exact tag-set match over all 400 turns: 0.863.
Self-harm, grooming and inappropriate content miss some indirect phrasings. Use them as review triggers for a caring
adult or a moderator, not as the only safety net.
## Files
| file | what |
|---|---|
| `model.safetensors` | weights (fp32, 45 MB) |
| `config.json` | read-out columns, ontology id, default thresholds and the span families decoded by the span head |
| `tok/bpe8k/` | tokenizer |
| `ontology/` | tag ontology, text-tag table, the ontology extension (contact-centre roles, jailbreak, affect valence and intensity), tag-definition embeddings |
| `thresholds/display.json` | display thresholds per tag, read-out and domain (tags without an entry use the default thresholds in `config.json`) |
| `engine_settings.json` | the engine's display floors and domain mapping (for reference) |
| `arctag/` | Python package: `ArcTagger`, `driver.py` (session to JSON), `asr.py` (live ASR adapters), `live.py` (audio input), `_core/` (streaming engine) |
| `stream_deepgram.py`, `stream_speechmatics.py` | live ASR scripts |
| `tests/` | adapter tests and provider message fixtures |
| `example.py` | quickstart |
| `LICENSE`, `NOTICE` | licence (CC BY-NC 4.0 + Whissle terms) and copyright notice |
| `onnx/` | fp32 and int8 ONNX exports, their metadata, tokenizer and checksums (see ONNX / C runtime); `arctag_kids.json`: kids-domain thresholds |
| `MANIFEST.sha256` | checksums |
## Limitations
* **English only.** Non-English input gives meaningless output.
* Text only, with no audio cues. The model cannot detect speaker identity, emotion in the voice or vocal events.
**Speaker roles need speaker ids** (`spk`) on the words, for example from ASR diarization.
* The internal test set above has no toxicity spans; the toxicity numbers on this card are the public benchmarks above.
On OpenAI Moderation the tagger is behind the dedicated toxicity classifiers. Validate toxicity detection on your own
data before relying on it.
* Entity mention spans (`entity.mention`) are the weakest span family (exact F1 about 0.51 on the test set).
* Spans shown at first display (T0, often on interim ASR words) are less accurate than the same spans a few words later (T1)
or at the utterance end (T2). Use `tier="T1"` where a delay of up to 3 words is acceptable.
* Utterance-level safety, jailbreak and scam flags can fire on short or out-of-domain utterances, and on utterances that
follow a flagged one. For anything consequential, use `scores` (p) and your own policy.
* The `safety.pii.leak` and `safety.scam.*` alerts are tuned for recall and can fire on benign contact-centre text, such
as an agent's greeting or a caller giving their own details. Use the PII spans for redaction, and the `scores` (p) with
your own threshold if false alerts matter.
* Affect (emotion, arousal, valence, intensity) comes from the words only and is experimental; this card reports no affect
evaluation.
* The `auto` domain is weaker than setting the domain. Some sports events fire loosely outside sports.
* Redaction masks the decoded spans. It is not a compliance guarantee: review it for your use case.
* The default thresholds favour precision. Tune them for your traffic if you can.
* The Python package runs fp32 on CPU; the ONNX exports cover the T0 read-outs only.
## Licence
The weights and the code, including all inference code: CC BY-NC 4.0 plus the Whissle terms in [LICENSE](LICENSE).
* Non-commercial use only.
* Commercial use needs a separate written licence from Whissle Inc.: ksingla@whissle.ai.
This covers every file in the repository, including `arctag/_core/` and `onnx/` (see [NOTICE](NOTICE)).