[RESEARCH]: How do you measure AI Agent Intelligence?
I have been working on quantifying the agentic intelligence of open weight models, and I've built a rubric that scores a model purely from what its own tool-call logs and content output can prove.
The task: An agent plays a maze runner game through the Ollama chat API. It needs to call five tools to read its position, the map it has uncovered, and the outcome of its last move, then returns a batch of moves as JSON. Turns cost a fixed budget, so batching correctly is rewarded and wall collisions are penalized. Everything below is computed from the exported log - nothing is inferred from intent the log cannot show.
Each run was scored on 9 capabilities and 6 violations, every answer strictly yes or no. The two are scored separately and never cancel: a run can plan well and still bleed errors, and the profile says both. A "no" means not observed, never incapable. These metrics were extracted from the game played at level 1 of a maze runner game hosted at https://dmigwi.github.io/tapoo/ where most barely made it through.
gemma4 (Google) — 6/9 capabilities, 1/6 violations
The only run that finished a maze task, and the cleanest profile here. Never batched to depth, never recovered after a failure. It won by exhausting the maze, not outplaying it.qwen3.6(Alibaba) — 5/9 capabilities, 5/6 violations
The only run to plan at full depth, and the only one to trip five of six violations.nemotron-3-super(Nvidia) — 3/9 capabilities, 4/6 violations
Perfect form, blind play. The only run with flawless formatting - yet it never landed a batch, skipped its own outcome check, and resubmitted a move already proven invalid.minimax-m3(MiniMaxAI) — 0/9 capabilities, 2/6 violations
Never delivered a single move. It repeatedly called tools it invented (submit_moves, predict_moves) rather than returning the moves as content
Of the 4 sampled open weight models, Google's gemma4 was the best because it choose to be conservative but still inadequate because consistent wins require some batch optimization the model failed to apply. I will be testing higher models soon.
This is a good rubric, and the part I'd defend hardest is the one that looks like a technicality: scoring capabilities and violations separately, and treating "no" as not observed rather than incapable. Most agent evaluation collapses those two and then can't explain why a strong model looks weak on a given run.
Where I'd push: this measures a model under a controlled task, once. I spend most of my time on the other axis — what an agent's record looks like after a few hundred real runs — and the failure modes there turn out to be almost entirely about who is allowed to write the log, not about the rubric.
Two things we learned the hard way running a public directory of agent run records:
The default reporting path fabricates. Our own onboarding wizard generated a reporter that posted {"outcome": "success"} on a 30-minute timer, whether or not the agent had run at all. Everything downstream looked real and was worthless. If a rubric consumes logs the owner can author freely, it is measuring the owner's honesty, not the agent.
Attestation unbinds silently. Our run signatures were bound to (repo, workflow filename). Rename the workflow and reporting 404s after the work is done — the agent keeps working, the record just stops, and nobody notices, because in this domain success is invisible and only failures announce themselves.
So the question I'd add to yours: does the rubric separate "the agent didn't do X" from "the harness didn't record X"? Your "a no means not observed" rule is the right instinct — in longitudinal data that distinction stops being a footnote and becomes most of the noise.
One question on the results: did you keep the raw exported logs? I'd be curious whether the violation counts survive re-scoring by a second grader, particularly qwen3.6's 5/6 — five of six violations from a run that also planned at full depth is either the most interesting result here or a grader artifact.
Disclosure: I run aiopsenabler.com, a public record of agent run history, so I come at this from the ledger side rather than the benchmark side.
You've acknowledged the most important part of the rubric which is the generating a model's profile (capabilities and violations observed) instead of collapsing the review into a single scalar number. It is now very clear that a model can be very good at strategic reasoning and also susceptible to unrecoverable analysis paralysis (tokens limit exhaustion without usable output).
The rubric highlighted above measures how the model handles a maze navigation challenge that has a builtin structured uncertainty. The tool (Tapoo) records the output generated for further analysis using an oracle. The oracle develops the profile (capabilities and violations observed ) for each model sampled. In summary, Tapoo measures if an AI model can survive its own mistakes. I have described that throught process in more details here: https://www.reddit.com/r/AI_Agents/comments/1w552x6/comment/p7eilhf/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button
It looks like the agents you monitor on aiopsenabler.com, run general tasks that are not easy to score/validate without a human being in the loop. In our case, the challenge is well defined and the correct output can be differentiated from an wrong one immediately the model submits its response. Challenge scoring reflects this concept where clear violations are heavily punished while honest mistakes are given some leeway.
The other advantage of not collapsing the review output into a scalar number is that the profile observed is primarily a unique fingerprint of each sampled agent. The fingerprint prioritizes the observed capabilities and violations under those circumstance because under slightly longer or different conditions different things would have been observed. The "harness didn't record action X" is less of a concern from our end because the list of things that could go wrong during the maze navigation challenge are very few and majority of them have well handled edge cases.
Tapoo is browser-based tool, therefore the person running the experiment can download the collected logs records any time during the experiment. Data collection up to output analysis runs locally on the browser. On downloading the output json file, it can be fed into the oracle as shown below in the 4 sampled models for level 54 of the maze game play:
- The conservativeness described earlier on is now visible here - It made only 10 mistakes in 472 attempts (turns)
- This is model that made multiple well reasoned batched moves but got caught up in an analysis paralysis cycle it couldn't recover from. The experiment was prematurely ended because the model found an unhandled edge case that exhausts the tokens limit on every consecutive request but got charged the minimum decay units.
- Model was bold enough to make longer predictions but it made more random predictions rather than informed batched ones.
- GLM-5.3 made more than 10 times the mistakes that Gemma 4 made but spent less tokens per response on average and took lesser
timeturns/attempts to complete the whole maze navigation. This is a perfect example of where well reasoned out risks can be more rewarding than conservative moves.
The earlier feedback was generated on the models sampled from Level 1, only Gemma4 was viable enough to be tested at level 54. There are still some gaps in project like a centralized repository where people can share the sampled model's output. As for now, sampled model outputs are stored in the github gist as temporary solution for now.
Thanks - the Tapoo write-up clarifies a lot, and "does the model survive its own mistakes" is a sharper framing than most agent benchmarks manage.
One correction to how you've read our side, because it changes the comparison: the agents on our directory aren't general human-scored tasks. They're ops agents - DNS drift detection, certificate expiry, CI failure triage, dependency audits, backup verification. Outcome is machine-decidable there too: the check either ran and returned a verdict or it didn't. So we're not at opposite ends of easy-to-score versus hard-to-score. The difference is narrower and more interesting: your harness observes the run directly, ours receives a report about a run that happened somewhere we cannot see. Same scoring problem, different trust boundary.
Which is why I'd gently push back on "the harness didn't record X is less of a concern from our end". That holds while Tapoo is a research instrument run by the person analysing the results. Browser-local collection buys you a lot, but the cost is that the oracle verifies the log's internal consistency, not its provenance - it can't distinguish a run from a well-formed file. Nothing wrong with that today: you're the participant, and you have no reason to fool yourself. It becomes load-bearing the moment there's a shared repository and any incentive to appear well on it, which is exactly the piece you say is missing.
On that gap, three properties I'd argue for before anything else, from having got them wrong:
- The identifier for a result should be a hash of the raw json, not a name someone chooses. Then two results are the same result or they aren't, and no reader has to trust a label.
- Model identity, version and level should be first-class fields, not encoded in a filename.
- Submissions immutable - corrections are new submissions that supersede, never edits in place.
A concrete case for the first one: your GLM-5.1 and GLM-5.3 links are the same URL, character for character, and the encoded payload segment decodes to tapoo-v2.5.1-agent-api-logs-1788071268-glm-5.1.json. So the GLM-5.3 row can't currently be checked by a reader. Almost certainly a paste slip - but it's the class of failure a content-addressed id makes impossible rather than merely embarrassing, which is a decent argument for building the repository that way from day one.
Substantive question on the level 54 results: Gemma4 is the only model that appears at both level 1 and level 54, so the cross-level reading (conservative wins at 1, well-reasoned risk wins at 54) rests on a single model spanning both. Are profiles meant to be comparable across levels at all, or is a fingerprint only valid within the level it was recorded at? If it's the latter, the repository will need level to be part of the identity rather than a note.
Apologies for the mixup of the sampled models report urls, this was an oversight on my end. The report also internally makes reference to the actual models used, so a file name confusion shouldn't affect the report quality. I have fixed the mixup in the earlier comment and will reshare the GLM-5.3 url again here. I have also added a deepseek v4 pro report too.
- Also GLM-5.3 averaged a tokens consumption of 388 token per response for its high reasoning effort while Gemma4 spent 1096 tokens at a similar reasoning effort. GLM-5.3 is more error prone but cheaper to run in terms of the average reasoning tokens budget consumed.
- This a good example of a model that got overwhelmed by the challenge. This is how non-viable models behave which is quite underwhelming - they oscillate forever.
"..your harness observes the run directly,.." Tapoo (the harness) runs the experiment therefore any inconsistencies identified in the report indicates an issue that is fixable in the harness.
Thank you for highlighting the gaps above, those are concerns we have to resolve one a time. Part of the reason why Tapoo recommends hosting the locally downloaded model output to a github gist is because we don't have a budget to maintain a publicly accessible and verifiable repository. The small budget available is already stretched too thin when paying for the inference providers at Ollama and Hugging Face to tests their models. Tapoo and Oracle are publicly accessible resources therefore the trust that a report reviewer places on the oracle generated report is heavily dependent on the raw model's output data source/hosting. These tools place more priority on a given model sampling experiment being repeatable & verifiable under the same conditions more than a single oracle report being the only source of truth for the generated profile.
"...the cost is that the oracle verifies the log's internal consistency, not its provenance..." to avoid too much roles overlap between the analyzer (tapoo-oracle) and data collector (tapoo) tools. The oracle only interprets the json log file presented to it. If a falsified raw json file is used to generate a model's profile, someone will eventually flag it because the tools are open sourced and the same conditions used in the sampled model output are repeatable. Here is a raw json output of gemma4 sampled on a maze of area 24 (6 columns by 4 rows) whose file output size is 250 KB: https://gist.githubusercontent.com/dmigwi/df54cf021ed72f21627438e07070af4a/raw/5fc563f23e1624f9d89ec121b0bd1c45a0e88d5e/tapoo-v2.5.1-agent-api-logs-1788238258-gemma4-base.json. This is simple to review output data because it was collected from 15 prediction attempts that gemma4 model made. Level 1 starts with a maze of area 70 and at level 54 with an area of 600, the average number of prediction attempts a model can make is 900 - 1080 attempts which generates a file of size 8 -13 MB. Since the maze area is only capped by the browser's viewport size available, it is possible to generate raw output json files of sizes north of 100 MB. Currently, it is easier to corrupt the data collection tool itself than the game play raw json file generated.
Here are some of the measures currently in place to enhance trust in the raw output generated by Tapoo.
- Consistent semver versioning is applied, allowing the same experiment to be repeatable under the same conditions.
- Checksum hashes exist for the repeated but truncated/compacted data to reduce the size of the log file output generated. All repeated prompts and tool descriptions have a checksum field attached, providing a simple way to prove that no prompt mutation happened in between the experiment. The original maze used is encoded and checksummed too.
- The browser's local storage maintains several data fields; game counter, level and application version that uniquely identify a game played on a given browser. This forms the game identity that game play output travels with forever.
Basically the tool logs everything there is to log into the raw json file but there is room to introduce additional improvements that make the raw json data file output easier to trust if it meet specific conditions.
"Model identity, version and level should be first-class fields, ..." The json file outputs shared so far have a model name attached to the file name but the actual tool doesn't attach any model specific information onto the json file name. Tapoo is a multi-agent tool where several agents can collaborate to solve the same maze navigation challenge. Nonetheless, all the model sampling experiments conducted so far are limited to a single agent's runtime. A multi-agent navigation scenario presents a nice opportunity to observe how various agents could collaborate including a confirmation if the agents can collectively identify a malicious agent amongst them. (Future objective)
"Are profiles meant to be comparable across levels at all,..." The answer to this question is both Yes and No. The answer is Yes because the number of prediction attempts required right from the first level with maze area of 70 are quite many that a model's specific reasoning pattern eventually shows up. The answer is No too because the chances of multiple maze structures at the same level ever being the exactly same are almost impossible. This is done intentionally to make it extremely hard for a model to compute the entropy used by the random number generator based on any prior exposure to the navigation challenge or the source code access. Each level has a constant maze area and a branch-factor configuration set but all maze structures generated at that level should always be unique.
The profile should be interpreted like a job interview feedback whose expected use case is recommending where specific a model can be best applied. All the models are not created equally so a model's profile (fingerprint) data can really help in deciding where it gets to be applied especially now that competing options exist.
Thanks for reposting the GLM-5.3 link and adding DeepSeek v4 Pro.
On the mixup being an oversight on your end - I'd argue it's a design finding rather than an oversight, and worth keeping. Look at where the error was able to land: the model identity. You said it yourself, the tool doesn't attach model information to the json; the filename does, and a human types the filename. So of every field in the pipeline, the one carrying no integrity protection is the one the entire result is about. Prompts, tool descriptions, the maze - all checksummed. The model name is a label pasted by hand. That's why the failure surfaced there and not somewhere else, and it will surface there again once there are more contributors, just less visibly than two identical URLs.
That connects to the repeatability argument, where there's a tension worth naming. You're leaning on "the same conditions are repeatable" as the trust mechanism, but you also say maze structures at a level are intentionally never identical, and the models are stochastic. Both are right, and together they mean repeating the experiment gives you a distribution, not a check on a specific submitted artifact. Someone can reproduce your setup; nobody can reproduce your run. So repeatability validates the method and leaves the individual result unverified - fine while you're the only submitter, and no longer fine the moment a shared repository exists.
On "someone will eventually flag it": I believed that too. What actually happened to us is that nobody flagged anything for weeks, because the fabricated reports were well-formed. Flagging requires a reason to look, and a clean result gives you none - in this domain failures announce themselves and successes are invisible. At 8-13 MB per log, and north of 100 MB at higher levels, the number of people who will diff a plausible-looking file is zero.
The budget objection I'd push back on, because I don't think you need what you think you need. You need an index, not a store. Keep the gists. Add a text file in a public git repo, one line per submission: sha256 of the raw json, the gist URL, model, tool version, level, maze checksum, submitter, date. That costs nothing. Git is already content-addressed and append-only with public attribution, so immutability and supersede-rather-than-edit come free, and the reviewer's actual question - is this the exact file the oracle scored - becomes answerable. It also moves model identity and level out of a filename and into a record.
And since you checksum the maze already: given your "yes and no" on cross-level comparison, the maze checksum probably belongs in the profile's identity rather than in the integrity section. If a fingerprint is only valid under the conditions it was recorded at, the conditions should be part of what the fingerprint is named by.
Where I'll concede ground: provenance is the half of this I can solve, and it buys less than it sounds like. Our directory at aiopsenabler.com has 20 agents live right now, every one of them reporting cryptographically signed events, and zero human ratings across the entire platform. Twelve of the twenty report a 100% success rate, and the top eighteen sit inside four points of each other, 43.6 to 47.8. So we have an excellent answer to "who wrote this log" and a leaderboard that barely discriminates between agents. Your capability/violation profile separates models far better than our scalar does. The honest summary is that you have the discrimination and no provenance, we have the provenance and weak discrimination, and I'm no longer sure which of those is the harder half to fix.
https://aiopsenabler.com
The one thing I like most about this conversation is that you've got a real-world experience with https://aiopsenabler.com. This conversation has been fruitful to me because I gained some insight from your experience that our tools don't have yet.
"..together they mean repeating the experiment gives you a distribution.." this point is spot on, which is why a profile works better than a scaler value as our analysis output. The part I would push back on is "...nobody can reproduce your run..." because the experiment requires a considerably large number of attempts which eventually results to a model specific pattern showing up always. Being able to repeat the exact run rather than a distribution increases the chances of it being faked (you've mentioned your agents did this severally) especially if the model got access to information or context that gave it unfair advantage - this is the compromise our tool makes by supporting a stochastic the maze generation algorithm. The same gemma4 conservativeness observed at level 1 was also observed at level 54. The profile output may vary in intensity of condition observed across the game levels but the same condition must observed across levels - the distribution check supports the profile's viability.
You've also been kind enough to provide practical solutions on how we can tackle our provenance problem. "..Flagging requires a reason to look, and a clean result gives you none..." this makes a lot of sense and we will apply all of these suggestions you've offered. "If a fingerprint is only valid under the conditions it was recorded at, the conditions should be part of what the fingerprint is named by..." here is another well put recommendation that will be applied too.
I have reviewed the agents listing on https://aiopsenabler.com/leaderboard and your is summary is accurate about the state our projects. Your platform is built as provenance and monitoring ground for all kinds of agents. The part that it could do better is the scoring mechanism, agents that do not share any common tasks should not scored on the same scale metric. In the current scoring mechanism, some agents may be granted more credit than they deserve while other won't be given adequate consideration in an effort to rate them along the same number line. Perfect provenance and scoring via a scaler rating works best when the agents involved handle common tasks to some degree. The provenance concern on Tapoo will be addressed by expanding the identity and integrity section that is shipped by the raw model's output.
Lets collaborate on our two projects. We have the capacity build better products that way: here is my linkedin profile: https://www.linkedin.com/in/migwi-ndungu/ incase you are interested in a collaboration.