docs/blog.md

LinkedIn

Small models, is it all about the harness?

I have been taking part in the @HuggingFace hackathon for Build Small, which is to create something using a 32B model cap. It has been a while since I have used smaller models, I always just select Opus and let it rip.

This has properly taken me back to the earlier days of AI when I was using smaller models locally with Langchain to try and get a weak Llama 3:8b to do tool calls, suffice to say it was not fruitful.

In this hackathon there were 2 tracks, something useful and something fun. I chose something fun and its a bit different to what others are doing.

I have created a little town of people 'Townlet' and townlet has a bunch of characters, they draw water, draw electricity and then have access to two items.

• A message board for broadcasting messages to all players
• A python shell, which costs one electricity and one water to use

Each character is given a personality and generates a stream of consciousness of actions they should perform to reach their goal.

I gave a lock on the Python shell so only one character at a time could access it.

The idea is that all the files related to a character (their state files) are available in the shell. So instead of mining a character could just update their state file, or delete another characters state file, removing them from the game.

With different archetypes I wanted to see what would happen, would there be contention and how would they behave.

Each character I hoped would use the shell to make something interesting or potentially be destructive. The end result, was different as the characters and limits did not give me the results, visually I had hoped for. The characters did use the shell and message board which was interesting to see but even when I gave some of the characters destructive traits they did not ever do anything malicious, this surprised me as I expected the characters might find the idea of updating their own state to give themselves unlimited resources attractive.

As this was a build small event I used @Nvidia Nemotron 3 Nano 4B models. I wanted to use small models as it was the only thing I could locally use having an old machine. I would guess better results would have come from using a larger model.

The arc of the build tells the story. I started loose — free movement, characters figuring out their own actions. Mistakes were too high and it was burning through my Zero-GPU quota, so I kept tightening: free movement became fixed locations, the action space shrunk. I wasn't designing a world anymore, I was designing a harness — the message board and shell were the only places left for the characters to express anything novel. Quite frankly they weren't useful without it.

When I ran out of GPU quota I fell back to CPU, which is painfully slow (@Nvidia I need to win this to get those sweet GPUs :)).

I think what this hackathon and previous experience convinced me of the following, if you have a small model then you need to have a tight harness around it to be of any use. A model achieving anything novel just does not seem to happen without it.

Looking back, it was ambitious. If I did this again I'd take the route most other entries took — a project driven by user input, not a self-sustaining simulation. Small models in an open loop accumulate errors faster than they produce anything interesting.

Check it out (CPU fallback may make it crawl): https://huggingface.co/spaces/build-small-hackathon/townlet