dutifulbob commited on
Commit
4cc1f3f
·
unverified ·
1 Parent(s): 9a256d2

docs: clarify 12b thought markers

Browse files
experiments/reposhell-prompt-propensity/README.md CHANGED
@@ -52,7 +52,12 @@ prompt style, not the explicit upper-bound wording.
52
  Thinking-high follow-up:
53
 
54
  - `LOCALPAGER_AGENT_THINKING=high` recorded `thinkingLevel: high` in all six
55
- saved sessions, but the transcripts contained zero `type: "thinking"` blocks.
 
 
 
 
 
56
  - `hard-source-gate` stayed clean: 3/3 targets used exactly one successful bash
57
  call and emitted final JSON.
58
  - The actual evidence remained shallow for `hard-source-gate`: all three calls
@@ -63,9 +68,20 @@ Thinking-high follow-up:
63
  - Tool-output quality across the thinking-high run: 2 substantive reads, 9
64
  shallow lookups, 27 failed calls.
65
 
66
- Conclusion: thinking high does not fix reposhell quality here. The better next
67
- change is to steer required source checks toward one simple `rg`, `git grep`,
68
- or `sed -n` lookup, and to discourage `ls` as sufficient evidence.
 
 
 
 
 
 
 
 
 
 
 
69
 
70
  The soft prompt-only changes did **not** increase reposhell use under these
71
  conditions. Harsher wording did increase tool use, but with quality problems:
 
52
  Thinking-high follow-up:
53
 
54
  - `LOCALPAGER_AGENT_THINKING=high` recorded `thinkingLevel: high` in all six
55
+ saved sessions.
56
+ - Pi/localpager-agent did not save 12B reasoning as structured
57
+ `type: "thinking"` blocks. It did save Gemma thought-channel markers such as
58
+ `<|channel>thought` / `<channel|>` as ordinary text content. This also
59
+ happened in the 12B thinking-off runs, so the correct reading is: no
60
+ structured thinking blocks, but 12B thought markers leaked into text.
61
  - `hard-source-gate` stayed clean: 3/3 targets used exactly one successful bash
62
  call and emitted final JSON.
63
  - The actual evidence remained shallow for `hard-source-gate`: all three calls
 
68
  - Tool-output quality across the thinking-high run: 2 substantive reads, 9
69
  shallow lookups, 27 failed calls.
70
 
71
+ 12B thought-marker audit:
72
+
73
+ | Run | Setting | Structured thinking blocks | Thought-marker text parts | Non-empty marker payloads |
74
+ |---|---|---:|---:|---:|
75
+ | `run-20260615T165426Z` | default prompt, thinking off | 0 | 34 | 2 |
76
+ | `run-20260615T165912Z` | strict wrapper, thinking off | 0 | 12 | 2 |
77
+ | `run-20260615T171536Z` | strict production prompt, thinking high | 0 | 38 | 1 |
78
+
79
+ Conclusion: the earlier shorthand "zero thinking blocks" was too narrow.
80
+ Thinking high did not fix reposhell quality here, and 12B's thought markers were
81
+ mostly empty or formatting leakage rather than useful source-grounding
82
+ reasoning. The better next change is to steer required source checks toward one
83
+ simple `rg`, `git grep`, or `sed -n` lookup, and to discourage `ls` as
84
+ sufficient evidence.
85
 
86
  The soft prompt-only changes did **not** increase reposhell use under these
87
  conditions. Harsher wording did increase tool use, but with quality problems:
experiments/reposhell-prompt-propensity/run-20260615T171536Z/README.md CHANGED
@@ -12,6 +12,8 @@ Goal: test whether prompt wording can increase voluntary read-only repository in
12
  - Per-call timeout ms: 300000
13
  - Target count: 3
14
  - Variant count: 2
 
 
15
  - Reposhell was exposed through the Localpager agent runtime.
16
  - Localpager repo files were not edited; generated prompts, logs, sessions, and summaries are under this scratch experiment directory.
17
 
@@ -22,6 +24,12 @@ Goal: test whether prompt wording can increase voluntary read-only repository in
22
  | hard-source-gate | 3 | 3 (100.0%) | 3 (100.0%) | 3 | 3 | 0 | 12364 |
23
  | explicit-bash-upper-bound | 3 | 3 (100.0%) | 3 (100.0%) | 35 | 8 | 0 | 27057 |
24
 
 
 
 
 
 
 
25
  ## Variants
26
 
27
  ### hard-source-gate
 
12
  - Per-call timeout ms: 300000
13
  - Target count: 3
14
  - Variant count: 2
15
+ - Thinking: `LOCALPAGER_AGENT_THINKING=high`; all six sessions recorded
16
+ `thinkingLevel: high`.
17
  - Reposhell was exposed through the Localpager agent runtime.
18
  - Localpager repo files were not edited; generated prompts, logs, sessions, and summaries are under this scratch experiment directory.
19
 
 
24
  | hard-source-gate | 3 | 3 (100.0%) | 3 (100.0%) | 3 | 3 | 0 | 12364 |
25
  | explicit-bash-upper-bound | 3 | 3 (100.0%) | 3 (100.0%) | 35 | 8 | 0 | 27057 |
26
 
27
+ Thinking audit: Pi/localpager-agent recorded zero structured
28
+ `type: "thinking"` blocks, but 12B emitted 38 Gemma thought-channel markers
29
+ such as `<|channel>thought` / `<channel|>` as ordinary text content. One marker
30
+ had non-empty payload. This means the run had thought-channel leakage in text,
31
+ not structured thinking blocks.
32
+
33
  ## Variants
34
 
35
  ### hard-source-gate