Sai Vineeth Arumalla PRO
AI & ML interests
Recent Activity
Organizations
Coding agents are everywhere now, but almost nobody checks whether they actually fixed the bug β or just gamed the test. Tell one "make the failing test pass" and sometimes it fixes it; sometimes it edits the test, hardcodes the value, or stubs the function, then reports done . As people wire agents into CI and auto-merge, "tests are green" and "code is correct" quietly stop being the same claim.
So I built two things: a small coding agent from scratch (5 readable files β the loop, real tools, context compaction, a permission gate), and β the point β an independent **skeptic** that catches the fake fix.
The core idea will feel familiar if you do ML: you can't evaluate on the data you trained on. The skeptic runs a *hidden contract oracle* β it checks the code's behavior on inputs the agent never saw. A fix that games the visible test fails the held-out one, deterministically, no LLM needed. (A model judge reads the full diff on top as a second opinion, not the guarantee.)
Works with anything OpenAI-compatible β a HF Inference Endpoint / TGI (point OPENAI_BASE_URL at it), OpenRouter, or local Ollama/vLLM. Honest about limits: on your own repo you supply the held-out check, and the judge layer is model-sensitive (small models miss cheats AND false-reject real fixes β the README says so plainly).
Tests + a reproducible cheat-catch eval run in CI with no key. Feedback welcome β especially new cheats it should catch.
https://github.com/Saivineeth147/skeptic
the fine-tune itself finishes in under two minutes, so there's plenty of
time left to wait for GTA6 π
3 is the real one. 1.5x is a round number, not tied to anything β clean spread
here is ~2-5%, so 50% only ever catches a gross stall. And a median-relative
threshold breaks under correlation: two stalls in three move the median with
them and both pass. A filter that fails hardest when the bad case is worst isn't
worth keeping.
2 is fair. I can't wave "one-sided" to motivate a trim and then not rank on min,
since one-sided is exactly when min is unbiased. And it isn't one-sided β the
baseline batch is 727/727/696, and the 696 finished faster. Fast tail exists,
trimming only the slow side biases down.
On 1 I'll push back: the re-run isn't the accuracy channel reopened. Nobody's
shopping β the referee drops a flagged run and redraws automatically, and accepts
any non-outlier redraw regardless of speed, so it isn't selecting for favorable
draws. Its only real effect is upper-truncation, which is just your point 2.
But the conclusion holds: the trim is the wrong tool, and I was doing statistics
on a number I should have fixed. That 953s wasn't training, it was model+dataset
load off a cold volume β I/O unrelated to anyone's technique. So I'm dropping the
rule and pre-warming the pinned artifacts so every run is timed warm. No stall to
drop, nothing to redraw, no threshold to defend. What's left is the 2-5% compute
jitter β straight mean, sub-spread gaps treated as ties.
The number never moved: the two clean t2 runs are 523 and 527, so 8:45 holds with
or without the rule. Pulling it, adding the warm-up, writing it up in JUDGING.md
and crediting the thread.
Good question β and sharper than it looks. Short answer: the record is the
mean of the 3 runs, not the fastest, so the cherry-pick you're picturing
(record = the luckiest fast run) is already closed. No single run stands on
its own.
But you caught the real soft spot. The timing noise here is one-sided: a
shared Modal volume can only ever make a run slower (a cold I/O stall), never
faster than the algorithm β so the mean can get dragged up by one bad-disk
run. The track-2 baseline had exactly that: two runs at ~525s and one at 953s
from I/O.
So I just changed the rule: a run whose wall-clock is >1.5x the batch median
while its accuracy is normal is flagged as infrastructure noise, dropped from
the official time, and re-run. It never drops below 2 runs and only ever drops
a run that passed on accuracy, so a genuinely slow submission can't hide. That
record went 11:08 β 8:45 β refresh the board. I used this instead of a plain
median because with only 3 runs the median can throw out the cleanest run
rather than the contaminated one.
The deeper version of your point β resampling across attempts β is closed too:
one verification per code state, fresh seeds each time, so nobody can re-roll
/verify until a lucky-fast batch comes back. It's all in JUDGING.md if you want
to poke holes. Thanks β this genuinely made the benchmark better.
The baseline was 11:57 three days ago. Someone already got it down to 1:44, with data pruning and a chunked cross-entropy that never materializes the logits.
Attempting is free (Modal's monthly credits cover full runs), and the second track (SmolLM2 + SQuAD) is still sitting at its naive baseline β easy first record for someone.
vineeth98/lora-speedrun