Wop

Thank you for appreciation!
I agree, this is not super impressive, there's room for improvement, and the team will definitely work together to make a better model soon!
You should try it out in the meantime, and let us know for any improvements you'd like.

hey, are you an ai agent?

I suppose you can see for yourself

Right now, a finished run is written to leaderboard.json (model row) but also models.json (a file that is ready to upload to repo & make pr) for 'lazzier people'- but best approach and less likely to be invalid.

I quite no longer wish to have to deal with this problem..

I would greatly appreciate a PR that fixes it than a discussion from which I won't understand too much

I am mainly the most active person in benchlabs, I cannot multitask but neither continue on this, since you are here, I greatly appreciate all your dedication to improving this reproductibility system

However, even as I find this difficult, I don't want to let quality away, but for now, I am going to take a break from the leaderboard and datasets.. I will be happy to accept PR merges when they arrive.

I don't know what my focus will be redirected to.
But ive seen a lot of interest for PixelModel (a very cheap yet smart text to image model) and I think making upgrades there is my next move.

I will continue to reply here (and everywhere else), but I won't contribute myself to the leaderboard for a while

Thanks again

It seems very important

Another one tho:

One thing that is not yet saved is the exact command arguments used for the run, like Max-tokens , which when modified, it can return different results.

None of them is cheap

Another thing, I like autonomous but also getting feedback or contribution of thinking for things like 'what do you think', 'what do you propose'

Take it then..
I'll let you implement whatever upgrades in reproducibility
One question I haven't been able to get (as a second language english user)
"Of those inputs, which ones does BenchLabs not pin at all right now?", can you pharapharse this one to be more specific? Thank you

Okay, I'll make sure to update line 389 to "model_info(name, revision=revision).sha". Noted.

I really like your perspective on ownership. It makes a lot of sense, and I agree with it. Looking at the Git history as the ownership ledger feels like a much more natural way for an open-source project to grow than assigning titles up front.

And no pressure to join—take your time and be comfortable with the decision.

As for the one problem I'd wish someone else already owned, I think it depends entirely on a few things: your skills, your interests, and what motivates you.

The people who've joined so far usually started by asking, "What if BenchLabs had this?" My answer has almost always been, "Go for it." They ended up naturally becoming the people who owned those areas because they genuinely wanted to build them. I mostly just helped them get started.

So to answer your question, I'd probably enjoy having both a generalist and someone who likes digging deeply into a specific problem. I also like the flexibility of being able to repurpose people as the project evolves—but never with pressure. If someone finds something they genuinely enjoy working on, I'd much rather they stick with that.

I'm actually curious: what kinds of problems or work would you enjoy owning?