200 RP Benchmark
Really impressive work on the model! Would you mind walking through how you benchmarked the model against the 200 roleplay prompts — the judge, rubric, and whether the eval is single-turn or multi-turn?
Sure!
So, a long, long time ago, each of the 200 benchmark prompts was extracted from a multi-turn dataset at various intervals, ranging from first turn until, say, the twentieth turn. All these samples end with a user turn, which is fed to the model at 0.0 temperature, aka greedy deterministic sampling. This dataset is certified cliché free using the same code that is used in the detection method outlined below as to ensure the results won't be polluted, and the data itself is something the model hasn't seen before.
Responses are then verified against a known list of cliches, something I carefully built across the years based on former feedback and frequency analyses, a fair bit involving Regex to catch those sneaky cases like "calculating xx" and "predatory xx". From there it's a simple case of collecting a count per sample and eventually, a xx clichés/100 words metric! In the case of V2 we went from 1.141 to 0.551.