The CPU plot doesn't say what CPU. Was the 28s at 8k a single sequence on Apple silicon, or a many core x86 part? It matters for the use case you're describing, because "scan a contract" in practice means scan forty thousand of them.
trust_remote_code=True is the other thing. There's no ORT or OpenVINO path yet, so anyone following this post is on PyTorch eager on CPU, and I'd bet a decent chunk of that 28s is eager mode rather than the architecture. Any plans for int8 or ONNX exports? That's what people serving these all day are going to need.
We run bare metal EPYC behind Firecracker microVMs at Tenki. I can run encoder_eval and your latency sweep at 16, 64 and 128 vCPU and post the numbers. Seems worth knowing either way whether "the hardware you already have" means a laptop or a server.
Separate note: the policy linting demo is the one I'd pay for. Right now there's nowhere good to put a guardrail that runs on CPU inside the same sandbox as the agent, so it either goes out to an API or doesn't happen.
