validation/coreml_runtime_status.md

Surya CoreML Runtime Status

Generated on 2026-06-19 in studio@100.102.185.54:~/datalab-quants-cairo.

Packages

• artifacts/coreml/surya-ocr-2-coreml-8bit/surya_vision_fp16.mlpackage
• artifacts/coreml/surya-ocr-2-coreml-8bit/surya_vision_int8.mlpackage
• artifacts/coreml/surya-ocr-2-coreml-8bit/surya_prefill_fp16_seq300_cache512.mlpackage
• artifacts/coreml/surya-ocr-2-coreml-8bit/surya_decode_step_fp16_cache512.mlpackage

Passing Runtime Gates

• Prefill parity before CoreML export:
  • prefill_parity.json
  • native/custom first token: 1039
  • logits max diff: 2.6702880859375e-05
  • logits mean diff: 8.332146421707876e-07
• Prefill CoreML smoke:
  • prefill_coreml_smoke.json
  • Torch/CoreML first token: 1039
  • logits max diff: 0.3057253360748291
  • logits mean diff: 0.03853870555758476
• Decode CoreML iterative smoke with advancing native cache:
  • decode_step_iterative_smoke_fixed_native_cache.json
  • tokens match 9/9
  • text: <p>Invoice
• Combined language runtime smoke:
  • combined_prefill_decode_smoke.json
  • CoreML prefill cache -> CoreML decode step
  • tokens match 9/9
  • text: <p>Invoice
• Vision-inclusive runtime smoke, FP16 vision:
  • vision_fp16_prefill_decode_smoke.json
  • CoreML vision -> CoreML prefill -> CoreML decode
  • tokens match 9/9
  • vision mean diff vs torch: 0.019221976399421692
• Vision-inclusive runtime smoke, INT8 vision:
  • vision_int8_prefill_decode_smoke.json
  • CoreML INT8 vision -> CoreML prefill -> CoreML decode
  • tokens match 9/9
  • vision mean diff vs torch: 0.021211756393313408

Current Host Responsibilities

• Tokenization.
• Initial text token embedding lookup.
• Image placeholder insertion.
• Rotary position embedding generation.
• Generated-token embedding lookup.
• Full-attention KV cache insertion.

Export Notes

• Prefill export uses skip_model_load=True and compute_units=CPU_ONLY during ct.convert to avoid CoreMLTools eagerly compiling the large MLProgram through ANE before saving.
• Runtime smokes still instantiate .mlpackage files with CPU_ONLY and run real predictions.
• The current prefill package is fixed to the canary prompt sequence length 300 and full-attention cache length 512.