sg-sem-h4

sg-sem-h4 is a small cross-encoder classifier for Skill Graph files. For one step, it reads the step's gate (the condition a checker decides before the run continues) together with the inputs that checker receives. It answers one question: can a checker decide this gate from these inputs?

Label	Meaning
checkable	The inputs carry what the gate asks about
partly	Some of the gate can be decided from the inputs
not	The gate cannot be decided from the inputs (for example, it judges how the user will feel)

The model runs in the Skill Graph connector for Claude. A gate scored not at or above the threshold gets the warning semantic-gate-uncheckable, which replaces the word-count rule gate-vague. You can read how it is used at devscratchpad.tech/graph/gate-check.

Files

File	What it is
sg-sem-h4-1.int8.onnx	Encoder, feature layer and H4 head. ONNX opset 17, dynamic int8 quantisation, 33.9 MB
sg-sem-h4-1.json	Manifest: input/output names, labels, temperature, threshold, SHA-256 of the model, gold results
vocab.txt, tokenizer*.json, added_tokens.json, special_tokens_map.json	Uncased WordPiece tokenizer (30,522 words + 12 special tokens)

SHA-256 of sg-sem-h4-1.int8.onnx: acc472f003202690547e1abeb875aac3fcc4b9a998d11da0ea70ab36768c6578

Architecture

• Encoder: cross-encoder/ms-marco-MiniLM-L12-v2 (MiniLM-L12-H384, about 33M parameters), with 12 added special tokens: [H1]–[H4], [GATE], [TASK], [INPUT], [CONTRACT], [READS], [VERDICT], [SEN], [CANDIDATE]
• Graph features: 16 numbers per example, scaled to 0–1 (step type, gate and prompt word counts, number of inputs, contract format, and so on), projected 16 → 64 with GELU
• Head: concat([CLS], features) → 3 logits

Sequences are cut to 384 tokens.

Input

The text is a sentence pair:

text_a: [H4] [GATE] The result is excellent and the user will be happy.
text_b: [INPUT] itinerary · Worker · A day-by-day plan; every item has a cost with its source or an estimate mark, and the plan ends with the total.

ONNX input	Type	Shape
ids	int64	[batch, seq]
mask	int64	[batch, seq]
types	int64	[batch, seq] (token type ids)
feats	float32	[batch, 16]

The output is h4: 3 logits in the order checkable, partly, not.

To score a gate, apply softmax(logits / 2.5) (temperature 2.5) and flag it when p(not) ≥ 0.8937.

import json, numpy as np, onnxruntime as ort
from transformers import AutoTokenizer

manifest = json.load(open("sg-sem-h4-1.json"))
tok = AutoTokenizer.from_pretrained(".")
sess = ort.InferenceSession(manifest["model"])

enc = tok("[H4] [GATE] The result is excellent and the user will be happy.",
          "[INPUT] itinerary · Worker · A day-by-day plan; every item has a cost with its source or an estimate mark, and the plan ends with the total.",
          truncation=True, max_length=manifest["max_len"], return_tensors="np")
feats = np.zeros((1, 16), dtype=np.float32)  # the 16 graph features; zeros only for a quick try
logits = sess.run(["h4"], {"ids": enc["input_ids"], "mask": enc["attention_mask"],
                           "types": enc["token_type_ids"], "feats": feats})[0][0]
p = np.exp(logits / manifest["temperature"]); p /= p.sum()
print(dict(zip(manifest["labels"], p.round(3))), "flagged:", p[2] >= manifest["threshold"])

Training

• Run msmarco-6. Fine-tuned in PyTorch with Hugging Face transformers in fp16 for up to 10 epochs, with early stopping on dev macro-F1. The H4 loss is 3-class cross-entropy with class weights.
• The model was trained together with three other heads (H1–H3) on one shared encoder. Only H4 met its bar on the gold set, so this file holds only H4.
• Data: gates written on a scale from precise to empty for real inputs of the Skill Graph starter graphs. The text was generated by local open-weights models (Qwen2.5-7B-Instruct and Qwen2.5-14B-Instruct). Data is split by seed graph.
• The int8 model scores the same as fp32 on the gold set.

Results

The gold set is a hand-labelled set of gates that was kept out of training.

Metric	Value
Precision of not at threshold	1.00 (0 of 42 other gates flagged)
Recall of not at threshold	0.95 (20 of 21 caught)
Macro-F1, 3 classes	0.86

Limits

• The model is trained only on Skill Graph gates and step inputs, which are short English text. It is not a general text classifier.
• It reads only the gate and the inputs. It does not know what the checking model can do beyond what those inputs carry.
• The gold set is small (63 gates), so read the numbers above with that in mind.

Licence

The weights are released under Apache-2.0, which is the licence of the base checkpoint. Note that cross-encoder/ms-marco-MiniLM-L12-v2 was trained on MS MARCO, whose terms describe that data as intended for non-commercial research purposes. Check those terms before you use this model commercially. The underlying microsoft/MiniLM-L12-H384-uncased is MIT. Qwen2.5 models (used only to generate training text) are Apache-2.0.