coreprinciple Claude Opus 4.8 commited on
Commit
36d6af8
·
1 Parent(s): 5ef2ba5

Call-1 weights: drop the parroted example, default to no-think (A/B verdict)

Browse files

Live A/B on "quiet green wander" was decisive:
- No-think: 9.6s, clean JSON — but byte-identical to the worked example
in the prompt (the 1B copied the example's numbers verbatim).
- Think: 26s (3x slower/quota), and corrupted the JSON (zeroed the
matching categories). Reasoning did not help this scoring task.

Two parrot traps now removed: the old `0.0-1.0` schema AND the worked
JSON example. New prompt gives NO copyable number-set — it states 0..1 in
words and teaches the mapping with short word->type cues, so the model
scores the actual vibe. Wrong/empty output still falls through to the
embed tier (_is_degenerate). Default flag flipped to no-think; thinking
stays toggleable via DISCOVERROUTE_VIBE_THINKING=1.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

src/discoverroute/config.py CHANGED
@@ -151,13 +151,13 @@ LLM_MODEL = "openbmb/MiniCPM5-1B"
151
 
152
  # A/B toggle for Call 1 (vibe→weights): run the model's REASONING pass
153
  # (enable_thinking, MiniCPM5-1B is hybrid-reasoning) or the fast no-think path.
154
- # Thinking may score a fuzzy vibe better but needs a far bigger token budget and
155
- # more of the ZeroGPU slice. Flip the Space variable DISCOVERROUTE_VIBE_THINKING
156
- # (1=think [default], 0=no-think); the chosen mode is recorded on every trace row
157
- # so the two can be compared head-to-head. (Either way, degenerate/empty output
158
- # falls through to the bge-small embed tier the route is never left tasteless.)
159
  VIBE_THINKING = os.environ.get(
160
- "DISCOVERROUTE_VIBE_THINKING", "1").lower() in ("1", "true", "on")
161
 
162
  # --- Trace logging (Open Trace) ----------------------------------------------
163
  # Every inference call logs a row locally to logs/traces.jsonl; when a write
 
151
 
152
  # A/B toggle for Call 1 (vibe→weights): run the model's REASONING pass
153
  # (enable_thinking, MiniCPM5-1B is hybrid-reasoning) or the fast no-think path.
154
+ # DEFAULT no-think: the live A/B was decisive thinking ran ~26s (3× no-think's
155
+ # ~9s, the ZeroGPU slice) and *corrupted* the JSON (zeroed the matching
156
+ # categories), while no-think returned clean output. Reasoning did not help this
157
+ # short scoring task. Flip DISCOVERROUTE_VIBE_THINKING=1 to re-run the comparison;
158
+ # the chosen mode is recorded on every trace row.
159
  VIBE_THINKING = os.environ.get(
160
+ "DISCOVERROUTE_VIBE_THINKING", "0").lower() in ("1", "true", "on")
161
 
162
  # --- Trace logging (Open Trace) ----------------------------------------------
163
  # Every inference call logs a row locally to logs/traces.jsonl; when a write
src/discoverroute/interpret/llm_vibe.py CHANGED
@@ -15,28 +15,33 @@ import time
15
 
16
  from discoverroute.interpret import mapping
17
 
18
- # NOTE on prompt design: the previous schema literally showed each key as
19
- # `"cafe": 0.0-1.0`, and the 1B model parroted the `0.0` back as the value —
20
- # returning an all-zero weighting that the router can't act on (a "quiet green
21
- # wander" came out tasteless). We now state the 0..1 meaning in words, forbid the
22
- # all-zero / all-equal degenerate answers explicitly, and give ONE worked example
23
- # (a vibe unrelated to any preset) so the model copies the *shape*, not a value.
 
 
 
 
24
  SYSTEM_PROMPT = (
25
  "You convert a walk/ride 'vibe' into place-type preference weights for a "
26
  "routing engine.\n"
27
- "For each place type, score how strongly the vibe calls for it: 0 means "
28
- "irrelevant, 1 means central to the vibe. Most vibes strongly want only two "
29
- "to four types give those 0.7-1.0 and keep the rest low. Never set every "
30
- "value to 0, and never give every type the same value.\n"
31
- "detour_budget_multiplier is how far off the direct line the vibe justifies: "
32
- "0.5 = stay direct, 2.0 = big detours welcome.\n"
33
- "Reply with ONLY a JSON object (no prose, no markdown) using EXACTLY these "
34
- "keys: cafe, park, bookshop, museum, bakery, restaurant, bar, viewpoint, "
35
- "market, quiet, green, historic, busy, detour_budget_multiplier.\n"
36
- "Example for the vibe \"sunny riverside picnic\":\n"
37
- '{"cafe":0.4,"park":0.9,"bookshop":0.1,"museum":0.1,"bakery":0.6,'
38
- '"restaurant":0.2,"bar":0.1,"viewpoint":0.7,"market":0.5,"quiet":0.6,'
39
- '"green":0.9,"historic":0.2,"busy":0.1,"detour_budget_multiplier":1.2}'
 
40
  )
41
 
42
  REQUIRED_KEYS = (
 
15
 
16
  from discoverroute.interpret import mapping
17
 
18
+ # NOTE on prompt design (two parrot traps, both seen live in the traces):
19
+ # 1) the original schema showed each key as `"cafe": 0.0-1.0`; the 1B parroted
20
+ # the `0.0` as the value → all-zero weighting tasteless route.
21
+ # 2) replacing it with ONE worked JSON example backfired worse: the model copied
22
+ # the example's numbers VERBATIM regardless of the vibe (a "quiet green wander"
23
+ # came back byte-identical to the "riverside picnic" example).
24
+ # Lesson: give the 1B no concrete number-set to copy. State the 0..1 meaning in
25
+ # words, teach the *mapping* with short word→type cues (not a full object), and
26
+ # forbid the degenerate answers. Wrong/empty output is still caught downstream
27
+ # (_is_degenerate → embed tier), so the route is never left tasteless.
28
  SYSTEM_PROMPT = (
29
  "You convert a walk/ride 'vibe' into place-type preference weights for a "
30
  "routing engine.\n"
31
+ "For EACH place type listed below, output a number from 0 to 1 for how "
32
+ "strongly THIS vibe calls for it: 0 = irrelevant, ~0.5 = a little, 0.8-1.0 = "
33
+ "central to the vibe. A typical vibe strongly wants only two to four types "
34
+ "score those high and keep the rest low.\n"
35
+ "Read the actual words of the vibe and score from them, for example: "
36
+ "'green' or 'park' or 'nature' park and green high; 'quiet' or 'calm' → "
37
+ "quiet high; 'lively' or 'buzzing' busy and bar high; 'coffee' or 'café' "
38
+ "cafe high; 'books' → bookshop high; 'history' or 'old' historic high. "
39
+ "Never make every value 0, and never make every value the same.\n"
40
+ "detour_budget_multiplier: 0.5 = keep it direct, up to 2.0 = big detours "
41
+ "welcome.\n"
42
+ "Output ONLY a JSON object (no prose, no markdown) with EXACTLY these keys, "
43
+ "in this order: cafe, park, bookshop, museum, bakery, restaurant, bar, "
44
+ "viewpoint, market, quiet, green, historic, busy, detour_budget_multiplier."
45
  )
46
 
47
  REQUIRED_KEYS = (