FluidInference/cua-s1-forms-coreml

CUA-S1-FORMS — Core ML

FP16 Core ML conversion of Cua's CUA-S1-FORMS, a 706,048-parameter specialist that selects among supplied form actions. The portable package is 1,511,163 bytes (1.51 MB). No text generation, KV cache, or external tokenizer is required.

The model uses small Transformer encoders over UTF-8 bytes and an attention readout. It is a classifier, not an autoregressive LLM. The inherited checkpoint configuration contains an unused hf_model value; this tinyx checkpoint does not load Qwen weights.

Files

File	Purpose
cua_s1_forms_fp16_options32.mlpackage/	Portable model; compile locally or add to Xcode
cua_s1_forms_fp16_options32.mlmodelc/	Compiled bundle for FluidAudio's model loader
preprocessing.py	Upstream-compatible byte encoding and input validation
conversion.json	Architecture, conversion versions, and portable package hashes
assets.lock.json	Pinned upstream source, model, and demo hashes
synthetic-test.lock.json	Full published synthetic test revision, count, size, and SHA-256
checksums.json	SHA-256 of each distributed file except this checksum file
reports/	Per-row parity, original Cua metrics, and compute-placement report

The model targets iOS 17/macOS 14 or newer. Runtime validation used an Apple silicon Mac. Use the portable package for local compilation on other supported systems; iPhone performance and compatibility of the precompiled bundle across older OS versions have not been measured.

Python usage

Install coremltools==9.0, numpy==1.26.4, and huggingface_hub on macOS. Download this repository, then run from its directory:

import coremltools as ct
from preprocessing import InputLimits, prepare_inputs

model = ct.models.MLModel(
    "cua_s1_forms_fp16_options32.mlpackage",
    compute_units=ct.ComputeUnit.CPU_AND_NE,
)
options = ["fill E-mail: person@example.com", "check", "click", "skip"]
inputs = prepare_inputs(
    'TASK fill the form from the document, then submit\n'
    'FORM Contact details\nELEMENT Edit "Email address" value=""',
    options,
    InputLimits(),
)
raw_probabilities = model.predict(inputs)["probabilities"][0, :len(options)]
print(options[int(raw_probabilities.argmax())], raw_probabilities)

Download the model artifacts:

hf download FluidInference/cua-s1-forms-coreml --revision main --local-dir ./cua-coreml
cd cua-coreml

Swift usage

The FluidAudio integration provides CuaS1FormsManager:

import FluidAudio
import Foundation

let manager = try await CuaS1FormsManager.load(
    from: URL(fileURLWithPath: "/models/cua_s1_forms_fp16_options32.mlpackage"))
let decision = try await manager.score(
    context: "TASK fill the form from the document, then submit\nFORM Contact details\nELEMENT Edit \"Email address\" value=\"\"",
    options: ["fill E-mail: person@example.com", "check", "click", "skip"])
print(decision.selectedOption, decision.probabilities)

decision.probabilities uses stable Swift softmax; decision.rawProbabilities preserves the package output shown in the Python example. See the full runtime validation.

try await CuaS1FormsManager.load() downloads and caches the compiled artifact automatically.

Tensor interface

Name	Type	Shape
context_ids	int32	[1, 224]
option_ids	int32	[1, 32, 96]
option_mask	int32	[1, 32]
logits	float32 output	[1, 32]
probabilities	float32 output	[1, 32]

Encode UTF-8 bytes plus one, pad with zero, and truncate by bytes at 224 for context and 96 per option. Supply a nonempty context and 2–32 nonempty options. Set option-mask entries to one for supplied options and zero for padding. Padded logits are -10000; padded probabilities are zero. Inputs above 32 options must be rejected or use a separately exported larger-capacity model. The example helper rejects overflow rather than dropping choices.

Conversion verification

On the complete pinned 196-row upstream demo, PyTorch and Core ML both selected 196/196 labeled options correctly, and matched each other's selected option on every row. The unmodified upstream Cua evaluator reports 36 fill, 4 check, 6 click, and 150 skip decisions, with zero wrong actions, wrong targets, or unsafe actions on these saved predictions. It counts skip as abstention, so its 23.47% coverage corresponds to 46 actionable decisions.

Check	Core ML ALL	Core ML CPU_AND_NE
Selected options matching PyTorch	196 / 196	196 / 196
Maximum absolute probability error	0.003099	0.002336
Warm model-call median	1.85 ms	0.90 ms
Warm model-call p95	2.49 ms	0.94 ms

Measured September 19, 2026 on Apple M5 Pro, 24 GB, macOS 27.0, using Python 3.11.11, PyTorch 2.7.0, and coremltools 9.0. Timing is exploratory and includes Python call overhead; model loading, encoding, document extraction, UI observation, and action execution are excluded. It is not an optimized PyTorch/MPS speed comparison.

The conversion gates require 100% selected-option agreement, no accuracy loss, maximum absolute probability error ≤ 0.005, finite outputs, normalized live probabilities, and zero probability for padding. The FP32 export adapter differs from the unmodified PyTorch reference by at most 0.00000113. Six reversed-option checks also pass on each Core ML configuration. The checked-in placement report counts 149 Neural Engine operations, 24 CPU operations, and zero GPU operations; operation counts are not a measurement of time spent on each processor.

These results verify conversion on three demo forms and three PDFs. They do not establish generalization, live GUI completion rates, or production safety. The original demo is not redistributed here; its exact revision and SHA-256 are in the asset lock. No training was performed. The separate full synthetic test below evaluates the unchanged artifacts and exposes numerical failures outside this demo.

Full published synthetic test

Evaluated the complete published synthetic test.jsonl: 24,370 decisions across 1,040 episode seeds, with no excluded rows or truncated inputs. The dataset revision is 8273f34778b99ac2e12d9f6e7d57dad99ae20845; its SHA-256 is d63a7e0db195d4d20154a40b2f8dd09ce3bb65487a158c638da5c609d4475e7c. The upstream model card reports 99.95% on an approximately 15,000-row synthetic test. Our result rounds to that accuracy, but the released file contains 24,370 rows; this does not reconstruct the card's unspecified smaller manifest.

Model / backend	Correct decisions	Top-1 accuracy	Median call	p95 call
Upstream PyTorch / CPU	24,359 / 24,370	99.9549%	1.787 ms	3.104 ms
Original FP16 Core ML / CPU + ANE	24,359 / 24,370	99.9549%	1.003 ms	1.133 ms
ANE-gather FP16 Core ML / CPU + ANE	24,359 / 24,370	99.9549%	1.052 ms	1.177 ms

Both Core ML exports select the same option as PyTorch on all 24,370 rows. All three share the same 11 errors: choosing fill when the label is skip. The original Cua evaluator counts these as wrong/unsafe actions; this offline benchmark executes no actions. All 9,802 fill, 816 check, and 1,040 click labels are correct; skip accuracy is 12,701/12,712. Higher ANE placement is approximately 4.8% slower by median here, so the original remains the default.

Strict numerical conversion parity fails for both exports. Each has 11 rows above the original 0.005 absolute probability-error limit, with a maximum error of 0.0204874. One further row (zero-based index 19270) has a live-probability sum of 0.99893665, which failed the original Swift manager's 0.001 normalization guard despite a correct argmax. The Swift probability fix now handles that output. These raw conversion reports retain the original scores, failed gates, tolerances, and model weights; the runtime fix does not establish raw numerical parity. The earlier 196-row demo passed its numerical gates.

Measured September 19, 2026 on Apple M5 Pro, 24 GB, macOS 27.0 (26A428), Python 3.11.11, PyTorch 2.7.0, coremltools 9.0. Batch size 1, three warmup rows per model, one timed pass over the whole split; both Core ML models remain loaded and alternate AB/BA order by row. PyTorch uses two CPU threads and one inter-op thread, with the Transformer fast path disabled. Timers cover PyTorch forward + softmax or synchronous Core ML prediction, excluding encoding, validation, loading, UI, and network. These compare deployment backends, not algorithms on equal hardware, and differ from the separate Swift timings at the end of this card.

Upstream describes this synthetic split as disjoint from training/validation by form signature; those signatures were not independently re-audited here. No training, validation inference, test-based tuning, or hosted Jev/API comparison was performed. This measures supplied-option classification, not unseen real-world GUI completion or document extraction.

Full report, complete compressed per-row trace, and test manifest retain the exact protocol, hashes, paired decisions, action metrics, and all failed numerical checks. Reproduce with benchmark-synthetic.py --require-parity in the Mobius toolkit. That command exits 1 for the recorded numerical failures; it does not normalize scores or weaken the original gates. Treat these artifacts as under review until the numerical failures are resolved and validated.

ANE profile

A separate September 19 profile uses the same portable-package hashes on the M5 Pro, with real demo rows 0, 68, and 130 (27, 21, and 19 options). Each policy runs two warmup passes and ten timed passes, totaling 30 timed predictions. All 120 timed predictions select the correct labels.

Policy	CPU ops	GPU ops	ANE ops	Warm p50	Warm p95
CPU_ONLY	173	0	0	1.527 ms	1.602 ms
CPU_AND_GPU	0	173	0	0.929 ms	2.380 ms
CPU_AND_NE	24	0	149	0.929 ms	0.973 ms
ALL	0	173	0	0.912 ms	1.229 ms

CPU_AND_NE assigns 86.1% of operations to ANE; ALL chooses the GPU on this Mac. CPU fallbacks cover integer/mask preparation and embedding gathers. These are public MLComputePlan preferred-device assignments, not measurements of utilization, energy, or time spent on each device. No Instruments runtime trace was captured.

ANE model loading took 566.8 ms, followed by a 1.65 ms first prediction, with system caches retained. These are not first-install cold-start numbers. Warm timing includes Python model-call overhead and excludes encoding, Swift/UI work, and animation. This three-row timing manifest differs from the full conversion parity run above; no weights or graph were changed.

See reports/ane-profile.json for all operation assignments, individual timings, hashes, and the protocol; reports/ane-fallback.json records rejection reasons. Reproduce with uv run --frozen python profile-coreml.py in the Mobius conversion directory.

Optional higher-ANE variant

The ane-gather/ directory contains an alternative portable package and compiled bundle with the same int32 inputs and float32 outputs and all trained weights. The variant uses shared float16 mask inputs and unsigned 16-bit embedding indices to eliminate negative-index correction and place the gathers on ANE. Valid byte IDs 0–256 remain exact. It is 1,509,491 bytes as a portable package.

On this M5 Pro, the scheduler plan is 162 ANE operations and 3 CPU input casts (98.2% ANE) for both CPU_AND_NE and ALL. The default model has 149 ANE and 24 CPU operations (86.1%) under CPU_AND_NE. Counts are not runtime or energy shares, and host byte encoding still runs outside the model.

The optional variant passes 196/196 decisions against upstream on ALL and CPU_AND_NE, with maximum probability error 0.002336 under the unchanged 0.005 tolerance. 28 Python regression tests pass, including all byte-ID boundaries, full option capacity, truncation, and reordered choices. The Swift manager independently passes all 196 reference decisions, compiled-cache loading, and concurrent/reordered requests; the native demo passes its three-form checks.

A matched same-process ABBA comparison uses three real inputs and 60 timed calls per model after warmup:

Artifact	CPU ops	ANE ops	Warm p50	Warm p95
Root/default	24	149	0.915 ms	0.968 ms
ane-gather/	3	162	0.970 ms	0.988 ms

Higher ANE placement is about 6% slower in this local comparison, so the root/default artifact remains unchanged. No energy or CPU-time saving is claimed. The original input names, dtypes, shapes, and byte encoding still apply. Load ane-gather/cua_s1_forms_fp16_options32.mlpackage with the existing Python or Swift APIs, or pass its local path to the Swift demo's --model argument.

Reports: parity, Swift validation, compute plans, fallbacks, and matched comparison. Reproduce with uv run --frozen python convert-coreml.py --optimization ane-gather --output-dir build/ane-gather in the Mobius conversion directory.

INT8 weight trial

A matched run over all 24,370 synthetic decisions on the same M5 Pro:

Export	Package size	Accuracy	Median	p95
Original FP16	1.51 MB	99.9549%	0.990 ms	1.102 ms
INT8 weights, FP16 compute	0.81 MB	99.9549%	0.990 ms	1.104 ms

46.2% smaller, with every selected option unchanged and effectively identical latency. Numerical parity still fails: 64 rows exceed the 0.005 probability-error limit (maximum 0.067738), versus 11 for FP16. No INT8 probability-sum violations were observed. The original remains the default.

The experimental int8-weights/cua_s1_forms_int8_options32.mlpackage uses INT8 weights with FP16 computation. Compile it locally or load it with the existing Swift load(from:) API. Inputs and outputs match the original model. This variant was measured on M5 Pro; iPhone behavior has not been measured.

Full report · Complete per-row trace · Quantization manifest · Reproduction

The matched benchmark uses CPU+ANE, batch 1, three warmups/model, and one complete pass with alternating Core ML order; encoding/loading/UI are excluded. All models get the same 24,359 decisions correct. Public compute plans assign 149 ANE and 32 CPU operations, with 19 unassigned constant-dequantization operations. This compression does not establish an INT8 activation path or a speed/energy gain. No calibration, training, or test-based tuning was performed. Numerical parity remains failed; the default FP16 artifacts are unchanged.

INT4 weight trial

A matched run over all 24,370 synthetic decisions on M5 Pro, CPU+ANE:

Export	Package size	Accuracy	Median	p95
FP16 control, iOS 18 target	1.51 MB	99.9549%	0.982 ms	1.081 ms
INT4 weights, FP16 compute	0.45 MB	99.9302%	0.982 ms	1.082 ms

70.1% smaller, with essentially unchanged latency. INT4 makes 17 errors versus 11 for FP16: 14 choices change, introducing 10 errors and correcting four. Numerical parity fails: 356 rows exceed the 0.005 probability-error limit (maximum 0.754359); no INT4 probability-sum violations were observed.

Packed INT4 requires iOS 18/macOS 15. Both exports use the same decomposed attention graph and FP16 computation. Batch-1 timing excludes encoding/loading/UI. INT8 preserves all choices at 0.81 MB; the original FP16 remains the default.

The experimental INT4 package is int4-weights/cua_s1_forms_int4_options32.mlpackage. Its matched FP16 control is int4-source-fp16/cua_s1_forms_fp16_options32.mlpackage. Nineteen weight tensors use per-channel symmetric INT4 with a 2,048-element threshold; no calibration or retraining. This is weight compression with FP16 computation. The graph is retargeted from the verified original to preserve decomposed attention.

Full report · Complete per-row trace · Quantization manifest · FP16 control manifest · Reproduction

The demo retains 196/196 choices but fails numerical parity. The full benchmark above exposes the accuracy loss. Public compute-plan placement is 149 ANE / 32 CPU operations, plus 19 unassigned constant dequantizations; no speed/energy claim.

Application responsibilities

The application must extract document entities, describe UI elements, build candidate actions, and validate and order the selected actions. Submission and other effects require application authorization. Scores are not calibrated confidence guarantees. Text outside the byte limits is truncated, and arbitrary new forms and languages require their own evaluation.

Source, reproduction, and license

The Mobius conversion toolkit contains the conversion code, lockfile, original reference implementation and evaluator, tests, and full reproduction instructions. Adaptations are limited to export-compatible masking, a floating-point clamp constant, finite padded logits, and disabling the fused PyTorch Transformer fast path during tracing. All trained layers and checkpoint tensors are retained; internal compute and weights are converted to FP16.

• Model: cua-ai/cua-s1-forms at f54adbf447f4ca6ec259f529ee3f2e3e09f8cc71.
• Demo: cua-ai/cua-s1-forms at 8273f34778b99ac2e12d9f6e7d57dad99ae20845.
• Code: trycua/cua at 83f142c4290a0f7d9ed545ae8532858c6e4f8145.

The pinned model and dataset cards declare MIT. See LICENSE, NOTICES.md, and the preserved upstream third-party notices.

Swift probability fix

The Swift manager computes a stable softmax from live logits using Double arithmetic and returns Float probabilities. The model's original softmax output remains available as rawProbabilities, including FP16 rounding errors.

All 73,110 real Swift API calls completed on the pinned synthetic split (Apple M5 Pro, 24 GB, macOS 27.0; Swift 6.2.3, release build, CPU+ANE):

Variant	Calls completed	Correct decisions	Changed choices after fix	Probability-sum failures after fix
FP16	24,370	24,359 (99.9549%)	0	0
INT8	24,370	24,359 (99.9549%)	0	0
INT4	24,370	24,353 (99.9302%)	0	0

The recorded FP16 sum failure is fixed. Stable probabilities agree with an independent float64 softmax within 0.000000030. Raw conversion-parity failures and INT4's accuracy loss remain. This validation changes no model artifacts and makes no new latency claim. Earlier complete Swift timings predate this fix.

Runtime report · Full runtime trace The report pins the dataset, packages, saved reference traces, Swift sources, and validation harness by SHA-256.

Reproduce with the Mobius Swift validation harness.

Live browser proof and expanded Swift benchmark

The native Swift browser demo loads both variants into independent WKWebViews. It reads actual DOM labels, roles and state, asks the model for a choice, applies compatible fill/check actions, dispatches events, and independently verifies the resulting DOM. Source values are user-entered or supplied by the original public examples. HTML contains controls, not source values or expected choices. The recording shows patient, job and insurance forms: 100/100 original decisions across both models, with event-count, stale-observation and explicit-click checks. Full actual contexts/candidates/actions are in browser-validation.json. This is bounded local browser automation, not arbitrary desktop control or PDF extraction.

The expanded release Swift comparison uses all 50 initial controls, both models resident, one warmup pass per model and ABBA with two full passes per block (200 timed calls/model). All 400 choices match upstream labels. On the M5 Pro / 24 GB / macOS 27.0 (26A428), original median/p95 is 0.912/0.933 ms; ANE gather is 0.961/0.984 ms, about 5.4% slower by median. This timer includes Swift encoding + Core ML + output decoding and excludes browser/rendering/animation. See swift-variant-comparison.json for raw samples, exact model hashes, per-form statistics, and load/first-call costs. The earlier compute-plan counts still apply to these unchanged artifacts; no utilization, energy saving or held-out accuracy claim is made.

Reproduce from FluidAudio commit 7f9eb92b0af8594c4e048a9e57f697340aacfa67 (the example is retained in history and is not part of the current library PR):

git worktree add --detach /tmp/cua-s1-browser-repro 7f9eb92b0af8594c4e048a9e57f697340aacfa67
cd /tmp/cua-s1-browser-repro
Examples/CuaS1FormsDemo/run.sh --browser
swift run --package-path Examples/CuaS1FormsDemo -c release CuaS1FormsDemo \
  --benchmark --report /absolute/path/to/variant-comparison.json \
  --hardware "Describe the measured Mac"

Both packages are fetched by pinned revisions and verified hashes, or supplied with --model /path/to/original.mlpackage --ane-model /path/to/ane-gather.mlpackage. The demo README also documents real-browser recording and its separate validation trace.