Spaces:
Configuration error
Configuration error
docs: clarify 12b thought markers
Browse files
experiments/reposhell-prompt-propensity/README.md
CHANGED
|
@@ -52,7 +52,12 @@ prompt style, not the explicit upper-bound wording.
|
|
| 52 |
Thinking-high follow-up:
|
| 53 |
|
| 54 |
- `LOCALPAGER_AGENT_THINKING=high` recorded `thinkingLevel: high` in all six
|
| 55 |
-
saved sessions
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 56 |
- `hard-source-gate` stayed clean: 3/3 targets used exactly one successful bash
|
| 57 |
call and emitted final JSON.
|
| 58 |
- The actual evidence remained shallow for `hard-source-gate`: all three calls
|
|
@@ -63,9 +68,20 @@ Thinking-high follow-up:
|
|
| 63 |
- Tool-output quality across the thinking-high run: 2 substantive reads, 9
|
| 64 |
shallow lookups, 27 failed calls.
|
| 65 |
|
| 66 |
-
|
| 67 |
-
|
| 68 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 69 |
|
| 70 |
The soft prompt-only changes did **not** increase reposhell use under these
|
| 71 |
conditions. Harsher wording did increase tool use, but with quality problems:
|
|
|
|
| 52 |
Thinking-high follow-up:
|
| 53 |
|
| 54 |
- `LOCALPAGER_AGENT_THINKING=high` recorded `thinkingLevel: high` in all six
|
| 55 |
+
saved sessions.
|
| 56 |
+
- Pi/localpager-agent did not save 12B reasoning as structured
|
| 57 |
+
`type: "thinking"` blocks. It did save Gemma thought-channel markers such as
|
| 58 |
+
`<|channel>thought` / `<channel|>` as ordinary text content. This also
|
| 59 |
+
happened in the 12B thinking-off runs, so the correct reading is: no
|
| 60 |
+
structured thinking blocks, but 12B thought markers leaked into text.
|
| 61 |
- `hard-source-gate` stayed clean: 3/3 targets used exactly one successful bash
|
| 62 |
call and emitted final JSON.
|
| 63 |
- The actual evidence remained shallow for `hard-source-gate`: all three calls
|
|
|
|
| 68 |
- Tool-output quality across the thinking-high run: 2 substantive reads, 9
|
| 69 |
shallow lookups, 27 failed calls.
|
| 70 |
|
| 71 |
+
12B thought-marker audit:
|
| 72 |
+
|
| 73 |
+
| Run | Setting | Structured thinking blocks | Thought-marker text parts | Non-empty marker payloads |
|
| 74 |
+
|---|---|---:|---:|---:|
|
| 75 |
+
| `run-20260615T165426Z` | default prompt, thinking off | 0 | 34 | 2 |
|
| 76 |
+
| `run-20260615T165912Z` | strict wrapper, thinking off | 0 | 12 | 2 |
|
| 77 |
+
| `run-20260615T171536Z` | strict production prompt, thinking high | 0 | 38 | 1 |
|
| 78 |
+
|
| 79 |
+
Conclusion: the earlier shorthand "zero thinking blocks" was too narrow.
|
| 80 |
+
Thinking high did not fix reposhell quality here, and 12B's thought markers were
|
| 81 |
+
mostly empty or formatting leakage rather than useful source-grounding
|
| 82 |
+
reasoning. The better next change is to steer required source checks toward one
|
| 83 |
+
simple `rg`, `git grep`, or `sed -n` lookup, and to discourage `ls` as
|
| 84 |
+
sufficient evidence.
|
| 85 |
|
| 86 |
The soft prompt-only changes did **not** increase reposhell use under these
|
| 87 |
conditions. Harsher wording did increase tool use, but with quality problems:
|
experiments/reposhell-prompt-propensity/run-20260615T171536Z/README.md
CHANGED
|
@@ -12,6 +12,8 @@ Goal: test whether prompt wording can increase voluntary read-only repository in
|
|
| 12 |
- Per-call timeout ms: 300000
|
| 13 |
- Target count: 3
|
| 14 |
- Variant count: 2
|
|
|
|
|
|
|
| 15 |
- Reposhell was exposed through the Localpager agent runtime.
|
| 16 |
- Localpager repo files were not edited; generated prompts, logs, sessions, and summaries are under this scratch experiment directory.
|
| 17 |
|
|
@@ -22,6 +24,12 @@ Goal: test whether prompt wording can increase voluntary read-only repository in
|
|
| 22 |
| hard-source-gate | 3 | 3 (100.0%) | 3 (100.0%) | 3 | 3 | 0 | 12364 |
|
| 23 |
| explicit-bash-upper-bound | 3 | 3 (100.0%) | 3 (100.0%) | 35 | 8 | 0 | 27057 |
|
| 24 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 25 |
## Variants
|
| 26 |
|
| 27 |
### hard-source-gate
|
|
|
|
| 12 |
- Per-call timeout ms: 300000
|
| 13 |
- Target count: 3
|
| 14 |
- Variant count: 2
|
| 15 |
+
- Thinking: `LOCALPAGER_AGENT_THINKING=high`; all six sessions recorded
|
| 16 |
+
`thinkingLevel: high`.
|
| 17 |
- Reposhell was exposed through the Localpager agent runtime.
|
| 18 |
- Localpager repo files were not edited; generated prompts, logs, sessions, and summaries are under this scratch experiment directory.
|
| 19 |
|
|
|
|
| 24 |
| hard-source-gate | 3 | 3 (100.0%) | 3 (100.0%) | 3 | 3 | 0 | 12364 |
|
| 25 |
| explicit-bash-upper-bound | 3 | 3 (100.0%) | 3 (100.0%) | 35 | 8 | 0 | 27057 |
|
| 26 |
|
| 27 |
+
Thinking audit: Pi/localpager-agent recorded zero structured
|
| 28 |
+
`type: "thinking"` blocks, but 12B emitted 38 Gemma thought-channel markers
|
| 29 |
+
such as `<|channel>thought` / `<channel|>` as ordinary text content. One marker
|
| 30 |
+
had non-empty payload. This means the run had thought-channel leakage in text,
|
| 31 |
+
not structured thinking blocks.
|
| 32 |
+
|
| 33 |
## Variants
|
| 34 |
|
| 35 |
### hard-source-gate
|