Measuring what a 4B parameter model running on my own GPU can do in an open-ended environment rather than on a benchmark: a fully self-hosted 2004-era RuneScape private server, bot accounts, and an initial standing goal to make money and train every skill.
Episode variance is enormous: a single run that appears to prove a feature works is noise. A factorial experiment harness and a flag ledger reveal which features actually help a tiny model play - and the pattern in what won and what lost.
Because the whole stack is self-hosted, the game engine itself is a variable. Freezing the world during inference, sweeping the tick rate, and scoring models on wall time instead of turns - several of these outperformed anything done to the prompt.
The bot fights goblins bare-handed with a bronze sword in its inventory, and walks into fences it cannot see. Two experiment rounds on equipment and spatial perception - both produced results that complicate the story.