FluidInference/jeff-coreml

Jeff / GLiFormer Large classification for Core ML

This is a fixed L128, batch-1 Core ML conversion of the trained classification path in knowledgator/gliformer-large-v1, as used by Jeff. It includes the 24-layer DeBERTa encoder, learned prompt-marker embeddings, and trained linear classification head in one FP16 ML Program. The original checkpoint contains 575,637,510 parameters; this Core ML package is about 880 MB on disk. The source checkpoint, tokenizer and decision-engine revisions are pinned in assets.lock.json.

The model scores one classification group with 1–8 labels and up to 128 tokenized input tokens. Choice, yes/no and ordinal labels use the same trained classification head; the probabilities are independent sigmoid scores, not a normalized softmax. Only the first len(labels) logits are returned. Other GLiFormer tasks such as NER, layout extraction, vision, audio and structuring are not exported here. This is not a Decision Index benchmark result.

The original model's classifier uses CLS pooling, a parent prompt anchor, linear parent+category fusion and dot scoring. Its word-level RNN is evaluated upstream but cannot affect these classification logits under this checkpoint's settings. jeff_decision.py checks those settings and carries the trained encoder and head; trace_compat.py makes fixed-shape DeBERTa attention convertible without changing trained weights. The trace-only finite mask substitutes -10000 for fp32-min before FP16 conversion; patched PyTorch logits match native logits on the checked cases.

Local inference

On macOS 15 or later with Python 3.12:

uv sync --frozen
uv run python - <<'PY'
from runtime import JeffCoreML

model = JeffCoreML(".", "JeffDecision-L128-FP16.mlpackage")
print(model.score(
    "The invoice was charged twice and the customer asks for a refund.",
    ["billing: invoice or payment issue", "support: technical product issue"],
    name="Choose the correct support queue",
))
PY

JeffCoreML uses the bundled tokenizer/config and GLiFormer processor, and loads no original PyTorch weights. Inputs exceeding 128 tokens or eight labels fail explicitly. Build from the pinned source checkpoint with uv run python export.py --convert --precision fp16; the source checkpoint must be in the local Hugging Face cache. uv run python verify.py --precision fp16 compares Core ML logits to native PyTorch.

Validation

Four real source-checkpoint fixtures cover billing, technical support, a three-label intent choice and yes/no classification. The mathematical decision wrapper matched native logits within 3.82e-6. Tracing-only patches matched native logits exactly on those fixtures. The exported FP16 Core ML model preserved all four chosen labels with maximum absolute logit error 0.1139. FP32 Core ML is a diagnostic control with four-of-four agreement and maximum logit error 3.44e-5; the FP32 package is not included because it is much larger. Full Decision Index quality, additional input lengths, other task heads and broad latency/ANE performance have not been evaluated. See native-parity.json and coreml-parity-fp16.json.

An optional JeffDecision-L128-W8.mlpackage compresses the FP16 package's trained embedding and linear constants using per-channel symmetric int8. Its 488,374,793-byte package (versus 922,812,014 bytes FP16) retained the chosen label on all four source-checkpoint fixtures with maximum absolute logit error 0.129469, below the predeclared 0.25 gate. Choose it by passing that package path to JeffCoreML. On a small battery-powered Apple M5 Pro probe, W8 had no demonstrated full-request speed advantage: All-compute p50 was 10.07 ms for W8 and 10.12 ms for FP16, with an outlier in W8's eight-call p95. Forcing CPU plus Neural Engine was about 19 ms for both. This is a size option with four-fixture validation, not a broad quality benchmark. Exact artifact hashes are in reports/w8-validation.json; see tools/decision-coreml-profile/RESULTS.md in the repository for timing scope and placement details.

When using huggingface_hub.snapshot_download, pass local_dir="./jeff-coreml" and point JeffCoreML there. Core ML compilation on the tested macOS release rejected a symlinked Hub-cache weight file; a materialized local directory avoids that path issue.

This conversion is derived from the Apache-2.0 GLiFormer weights and Transformers DeBERTa implementation. Jeff's decision adapter code is MIT licensed; the helper source here retains attribution. The model card makes no claim that this conversion is faster or more accurate than another model on a benchmark.