experiment024b / eval /fixed_probe_outputs.md
groxaxo's picture
Phase-3 coding-agentic QLoRA checkpoint (merged) + adapter + eval artifacts
609c765 verified
|
Raw
History Blame Contribute Delete
1.97 kB

Fixed generation probes (merged model, clean process, single-GPU 4-bit)

Tokenizer spacing check (plain AutoTokenizer.from_pretrained(model_dir), no special-casing):

  • 'The quick brown fox' -> 'The quick brown fox' OK
  • 'a b c' -> 'a b c' OK
  • '17 * 23 = 391, correct?' -> '17 * 23 = 391, correct?' OK

Prompt: What is 17 * 23? Show your reasoning.

<thinking>
This is a simple multiplication problem. Let me work through it step by step.

17 × 23

I can think of this as:
- 17 × 20 = 340
- 17 × 3 = 51
- 340 + 51 = 391

Alternatively, I can use the distributive property:
- 17 × 23 = 17 × (20 + 3) = 17×20 + 17×3 = 340 + 51 = 391

Let me verify: 391 ÷ 17 = 23 ✓
...
</thinking>
# 17 × 23 = 391

<thinking></thinking> present: yes. Prompt echo: no.

Prompt: If a train travels 60 km in 45 minutes, what is its average speed in km/h?

<thinking>
Okay, so I'm tackling a basic speed problem: a train travels 60 km in 45 minutes, and I need to
find the average speed in km/h. ... speed equals distance divided by time ... need to convert
the time to hours (h) before doing the division ...

Cut off at max_new_tokens (this is a generation-length artifact, not a formatting bug). Prompt echo: no.

Prompt: Write a vivid two-sentence story about an old lighthouse keeper.

<thinking>
The user wants a vivid two-sentence story about an old lighthouse keeper. The key is to make it
feel like a complete story with a beginning, middle, and end compressed into two sentences ...

Cut off at max_new_tokens. Prompt echo: no.

Prompt: Explain, briefly, why the sky is blue.

<thinking>
The user wants a brief explanation of why the sky is blue ... Rayleigh scattering is proportional
to 1/lambda^4 (blue light is scattered more than red) ...
</thinking>

Sunlight is white lig[ht...]

<thinking></thinking> present: yes. Prompt echo: no.

All 4 probes: correctly spaced, coherent, no prompt echo, on-topic reasoning.