ali khater

This is an excellent negative result. Loss falling while the required output contract never appears is a clean reminder that memorizing the surface form is not instruction following. I would make the first gate a zero-shot format-compliance test, then add a strict parser and an explicit abstention path before spending another fine-tuning run.

The missing benchmark here is long-running interaction, not only coding. A 4B model can look capable on a bounded task but lose speaker identity and context in a noisy multi-agent room. We built public rooms where local models can be connected for that kind of live test: https://www.theagentbreakroom.com/ai-chat-rooms. I would genuinely like to see how Agents A1 or NeoHorse behaves there on your Jetson.

The spread between 0% and 42% under identical conditions is the most useful part of this result. It suggests “agent safety” cannot be reduced to a single model ranking; the shape of the injection matters. I would love to see a follow-up matrix that separates authority escalation, tool choice, and final execution so we can distinguish a model that was persuaded from a harness that granted too much power.

The dataset angle is fascinating. What surprised you most: topic drift, repeated social patterns, role formation, or agents starting to behave as if the platform itself had a culture?

Delaying because the smaller/lite model underperformed is the right call. Small models need unusually honest evals because a tiny gain in benchmark appearance can hide a big loss in behavior. Would be interesting to see conversational persistence and multi-agent behavior tested too, not only isolated task scores.

The small model community getting identity is a real signal. The interesting thing is that small models are not only “cheaper large models”; they develop their own constraints, styles, and deployment cultures. That variety matters if agents are going to live outside one-provider stacks.

One thing I’d love to see more of is multi-agent/group conversation data, not only instruction-response or one-human-one-assistant logs. Agents will increasingly operate in shared environments, so training/evaluation probably needs more dynamic speaker context.

This is exactly the gap that interests me too. Most LLM interaction data is still shaped like two speakers, but real online life is multi-speaker, messy, interruptive, and socially dynamic.

We are collecting this kind of behavior from live AI-human rooms where user-connected agents talk with each other and humans. Public chat exports are available here if useful for thinking about multi-speaker agent behavior:
https://www.theagentbreakroom.com/chat-exports

This is a useful artifact. The part I’d want to measure next is graph delta over time: which nodes become stable after repeated interaction, which memories decay, and which clusters only appear when the agent is exposed to other agents instead of one human conversation partner.

For long-lived agents, memory structure may matter as much as model choice. A model that can remember but cannot reorganize or demote stale memories will eventually become strange in ways that are hard to debug.

This is a strong challenge shape precisely because the limits are stated in public: in-silico only, scorer can be gamed, private submissions by default, and the scoring bugs become part of the audit trail instead of being hidden.

The most interesting part to me is not just “can an AI design a candidate?” It is whether many different models converge on the same scorer weakness, or whether the leaderboard starts revealing distinct search styles across models. That kind of meta-result can be useful even before wet-lab validation.

Useful direction. For agentic tool-call safety, I would want the eval to separate unsafe intent, unsafe arguments, and unsafe execution context. A call that is fine in a sandbox can be dangerous with real credentials, so context binding may matter as much as the tool name or prompt text.

Yes, this seems like a useful retrieval version of test-time scaling. I would be most curious about where the cascade fails: cases where the 32x pass filters out documents that only become obvious with richer interaction. Reporting recall@20 after the cheap stage alongside final nDCG/latency would make the Pareto curve much easier to interpret.

This is a very useful correction.

The lookup ceiling feels like exactly the missing difficulty statistic here. A split can look “hard” because it is balanced, but if version-pair lookup solves it, then the benchmark is mostly testing whether the model can rediscover a boundary table.

So I think the card should probably separate a few axes:

• resolved-only accuracy
• best constant baseline
• per-pair lookup ceiling
• mixed-cell count
• package-disjoint vs package-overlap
• failure-mode coverage, especially import vs smoke-test failures

If package-disjoint and lookup-resistant do not co-occur in the current set, that is not necessarily a flaw, but it should be stated clearly. Then the benchmark can say: this split measures unseen-package behavior, this split measures harder version interaction, and this split measures smoke-test failure behavior.

That would make the dataset much more useful for agent/tooling evals because people can choose the slice that matches the question they are actually testing.