roleplay-jev-v2-int8

Dynamic per-channel INT8 ONNX build of the roleplay_jev_v2 student sentence-quality scorer (BGE-small-zh encoder + six-way head + per-token badness head).

• Task: score each sentence of Chinese roleplay/AIGC text on a 5-grade quality scale plus a skip class, and emit a badness probability for every token.
• Input: token ids from the bundled BGE-small tokenizer (max length 512).
• Size: 22.9 MiB (0.25x of the FP32 graph, 0.5x of the FP16 graph).
• Runtime: ONNX Runtime >= 1.20, CPU or GPU. Built for CPU/WASM-first deployment; the FP16 variant of the same graph is the better default for browser WebGPU.

The graph already contains the fitted calibration temperature and the token-head sigmoid, so its outputs are final probabilities — no post-processing besides renormalizing the five quality grades if you want them to sum to 1.

Files

roleplay-jev-v2-int8.onnx   INT8 graph
roleplay-jev-v2-int8.json   sidecar metadata (format, temperature, quantization)
infer/serve.py              standalone ONNX Runtime inference script (no torch)
infer/tokenizer/            BGE-small-zh tokenizer assets

Graph interface

input_ids       int64 [batch, sequence]
attention_mask  int64 [batch, sequence]
token_type_ids  int64 [batch, sequence]

sixway_probs    float32 [batch, 6]         softmax( logits / temperature )
token_badness   float32 [batch, sequence]  sigmoid( token logits )

Class index 0..4 are quality grades 1..5; index 5 is skip (out-of-scope / non-scorable text).

Inference

pip install onnxruntime tokenizers numpy
python3 infer/serve.py --format text --text '她重新把头转回来。转回来之后，她的眉弓往下压了一点。'

infer/serve.py defaults to the .onnx file next to it, so it runs from any working directory. Useful flags: --file, --batch-size, --provider, --threads, --token-threshold (default 0.3), --all-tokens, --format json|text, --output, --list-providers.

JSON output per sentence:

{
  "sentence": "...",
  "probs": {"1": 0.579, "2": 0.230, "3": 0.109, "4": 0.049, "5": 0.016},
  "skip_prob": 0.0172,
  "conditional_probs": {"1": 0.589, "2": 0.234, "3": 0.111, "4": 0.050, "5": 0.016},
  "expected_quality": 1.67,
  "needs_modification_probability": 0.918,
  "tokens": [{"text": "像", "start": 17, "end": 18, "bad_prob": 0.712, "flag": true}]
}

needs_modification_probability is P1 + P2 + P3. Token bad_prob is the badness probability of that input token; start/end are character offsets in the original sentence, so tokens can be mapped back onto raw text.

Score a document as a whole by weighting expected_quality with 1 - skip_prob; sentences with a high skip_prob carry little weight.

Quantization

mode                 dynamic INT8, per-channel weights
weight type          QInt8
activation           dynamic QUInt8 where applicable
operators quantized  MatMul, Gather, Transpose

Gather is included because the embedding table is a large share of a BGE-small graph. reduce_range is off.

Drift versus the FP32 graph, measured on a 694-sentence held-out test set:

metric	drift
six-way probability mean abs. diff	0.0049
expected-quality mean abs. diff	0.028
expected-quality Spearman (int8 vs fp32)	0.998
token badness Pearson (int8 vs fp32)	0.994
compression vs fp32	0.25x

Training data and scope

Trained on human six-way vote distributions (grades 1-5 plus skip) over Chinese roleplay text, cleaned to require at least three independent valid annotators per sentence, and split by work so no work appears in more than one split. Test-set metrics of this checkpoint:

metric	value
n	694
expected-quality MAE	0.735
Pearson	0.631
Spearman	0.623
ordinal EMD (normalized)	0.215
calibrated temperature	1.5194

The score is an aesthetic/quality judgement learned from human votes, not a safety or factuality classifier, and it is tuned for Chinese roleplay prose.