Headroom Eval Space

A persistent Level 4 Hill Climbing Loop that answers: "Does compression break agents?"

What It Is

A HuggingFace Space that continuously runs SWE-bench tasks through the Headroom proxy with kompress compression. It tracks behavioral regressions such as extra tool calls and redundant loops.

• Features: OpenTelemetry traces, public HuggingFace Dataset export, Plotly dashboard, and an OpenAPI schema.

• Academic Context: This project is part of an ICLR 2027 submission on the Voting Ensemble Paradox.

The Need

We are seeking a GPU grant for the Space to run on a T4 or A10 GPU instead of a CPU.

• Current Limitation: We are currently limited to simulated runs on CPU.

• Impact of GPU: With GPU support, we can run real agent trajectories through Claude Code/Codex via the proxy.

Open Science Commitment

Models: All 18 models are available on HuggingFace (PeetPedro/kompress-v3 through v17).

Logs: All experiment logs are fully public at pocoo.vaked.dev.

Codebases: Everything is open-source:

• github.com/peterlodri-sec/loopkit

• github.com/peterlodri-sec/longrun-eval-kompress

Grant Tier Request

Tier: Community GPU grant — T4 Small or equivalent.

Cost breakdown: Approximately $0.60/hr. Running 24/7 would be around $430/month, though even 8 hours/day would be sufficient to let us run real evaluations.

Quick Links

HuggingFace Space: huggingface.co/spaces/PeetPedro/headroom-eval

Project Page: kompress.vaked.dev

Repository: github.com/peterlodri-sec/loopkit