| --- |
| license: apache-2.0 |
| library_name: onnx |
| pipeline_tag: text-classification |
| tags: |
| - perseval |
| - agent-evaluation |
| - task-completion |
| - onnx |
| - rust |
| --- |
| |
| # Perseval Task Completion — development runtime candidate |
|
|
| This repository contains the compact ONNX artifact used to exercise Perseval's |
| local Task Completion evaluator and model-management flow. |
|
|
| ## Status |
|
|
| This is a **development candidate**, not a release-certified evaluator. It is |
| published so the macOS application can install, verify, update, and execute a |
| real local model end to end. Do not interpret its scores as production-quality |
| ground truth. |
|
|
| The frozen private comparison holdout remains sealed. A later release will be |
| marked certified only after it passes the documented holdout gates. |
|
|
| ## Files |
|
|
| - `manifest.json` — versioned model, projector, feature, calibration, and file |
| integrity contract. |
| - `model.onnx` — compact binary Task Completion head. |
| - `tokenizer.json` — pinned tokenizer reference. |
| - `parity.json` — Python/ONNX/Rust parity fixtures. |
| - `catalog.json` — mutable release pointer whose artifact revision is immutable. |
|
|
| Perseval downloads files from the immutable revision named in `catalog.json`, |
| checks every size and SHA-256 digest, then validates the internal manifest |
| before making the installation active. |
|
|
| ## Runtime contract |
|
|
| - Projector: `traceeval.compact-task-completion-projector.v1` |
| - Structured feature schema: |
| `traceeval.task_completion_structured_evidence.v2` |
| - Inputs: 39 structured evidence features |
| - Outputs: binary completion logits |
| - Calibration threshold: `0.745` |
|
|
| The application keeps trace projection, privacy enforcement, revision checks, |
| hash verification, inference, evidence resolution, and threshold application |
| in Rust. |
|
|
| ## Intended use |
|
|
| Use this candidate to: |
|
|
| - test Perseval's local learned-assessment runtime; |
| - validate model installation and updates without manual folder entry; |
| - run development benchmarks and Python/ONNX/Rust parity checks. |
|
|
| Do not use it as an automated decision-maker or as a substitute for human |
| review in high-impact settings. |
|
|
| ## Training data and limitations |
|
|
| The development pipeline combines source-balanced agent traces, structured |
| proof-bearing evidence, and group-safe splits. Synthetic, redistributable |
| examples are published separately at |
| [`Etolith/perseval-agent-evaluator-synthetic`](https://huggingface.co/datasets/Etolith/perseval-agent-evaluator-synthetic). |
|
|
| Known limitations include domain shift, unsupported-success claims, sparse |
| external-side-effect evidence, and incomplete calibration across long, |
| multi-agent traces. The compact head consumes a projected representation, not |
| raw screenshots or arbitrary full-context traces. |
|
|
| ## Reproducibility |
|
|
| The manifest pins the model, tokenizer, parity fixtures, projector, feature |
| schema, training run, and calibration versions. The public application verifies |
| those contracts before loading the artifact. Private traces, credentials, |
| teacher prompts, and unreleased checkpoints are not included. |
|
|
