Unseen1980/mockgen-classifier-v1

MockGen Classifier v1

Artifact status

The classifier is trained locally, export verified, and published to Hugging Face. This is the accepted v1 classifier artifact. Experimental retraining runs are evaluated offline and only replace this artifact when they improve both service-holdout and domain-holdout benchmarks.

Publication status

Public repository: https://huggingface.co/Unseen1980/mockgen-classifier-v1/tree/v1

Published artifacts are the quantized ONNX classifier, this card, and non-secret metrics. The publication-safe classifier dataset is published separately at https://huggingface.co/datasets/Unseen1980/mockgen-classifier-data-v1/tree/v1.

The card includes pipeline_tag: text-classification and widget examples so the Hub can display the classifier's task and sample outputs. Live browser inference depends on Hugging Face provider support for this repository. For a dedicated Hugging Face Inference Endpoint, publish the included handler.py and requirements.txt; the handler wraps the ONNX model with the same field-text normalization used by MockGen.

Intended use

The classifier predicts MockGen semantic vocabulary hints for enterprise schema properties. The runtime uses those hints to route properties to retrieval banks, deterministic patterns, grounded templates, and validated fallbacks.

Local inference

The classifier can be tested without a Fiori app:

npx tsx -e '
import { readFileSync } from "node:fs";
import { createOnnxClassifierRunner } from "./packages/mockgen-core/src/classifier-runner";

(async () => {
  const report = JSON.parse(readFileSync("models/classifier/onnx/export-report.json", "utf8"));
  const runner = await createOnnxClassifierRunner({
    modelPath: "packages/mockgen-models/artifacts/classifier.onnx",
    hints: report.labels,
    inputNormalizer: report.inputNormalizer
  });

  const result = await runner.classify({
    entity: "JournalEntryItem",
    name: "SAKNR",
    type: "string",
    constraints: { nullable: true },
    semantic: {
      hint: "unknown",
      label: "G/L Account",
      related: ["CompanyCode", "FiscalYear", "AmountInCompanyCodeCurrency"]
    },
    values: [],
    examples: []
  });

  console.log(JSON.stringify({
    hint: result.hint,
    confidence: result.confidence,
    top: result.top.slice(0, 5)
  }, null, 2));
})();
'

Expected top prediction:

{
  "hint": "gl_account",
  "confidence": 0.6323356628417969
}

The raw ONNX input is a single string tensor named text. MockGen normally builds this string from property metadata, for example:

property_name: SAKNR [SEP] label: G/L Account [SEP] entity_type_name: JournalEntryItem [SEP] neighbor_properties: CompanyCode, FiscalYear, AmountInCompanyCodeCurrency [SEP] type: string

Training data

Training data came from reviewed schema-property rows built during the cross-family classifier pipeline. The public dataset repository contains the publication-safe subset: project-authored fixture rows, reviewed public API field metadata, and reviewed recovery fixtures with source/provenance metadata. The exportable checkpoint uses a word-sgd backend with unicode-codepoint-v1 normalization rather than a pretrained encoder, because this path produced a compact ONNX artifact with stable multilingual signal.

Evaluation

Local validation evidence:

• Training rows: 21,533; validation rows: 2,393.
• Full-precision validation top-1: 0.9820309235269536.
• Full-precision validation top-3: 0.9887170915169243.
• Quantized ONNX validation top-1: 0.9816130380275805.
• Quantized ONNX validation top-3: 0.9887170915169243.
• Absolute top-1 drop after quantization: 0.00041788549937316066, below the 0.01 gate.
• Quantized ONNX size: 5.101428985595703 MiB, below the 50 MiB gate.
• Quantized ONNX SHA-256: b8827b79fc4def92ab6dbac7990a708771a8e72693df63c55f3b4f88d420ded9.
• Classifier replacement gate: a new trained artifact must improve both the service-holdout macro-F1 baseline 0.8247 and the domain-holdout macro-F1 baseline 0.7073.
• Most recent experimental retraining status: not published because it did not improve both benchmarks.

Limitations

The v1 vocabulary has about 50 trained hints with reserved slots for expansion. Expected weaknesses include rare enterprise fields, ambiguous labels without neighboring-property context, non-Western naming conventions, and schema domains not represented in the benchmark corpus.

Reproducibility

Use models/classifier/training-config.yaml, models/classifier/metrics.json, models/classifier/onnx/export-report.json, and the paired dataset repository as the local reproducibility anchors. Public artifact checksums are verified from the model repository after publication.