RhinoWithAcape commited on
Commit
1234fac
Β·
verified Β·
1 Parent(s): 933b987

Card: correct Honest Limits after stability reruns (self-ref was temp-noise, not a failure); v0.1 stays recommended over v0.2

Browse files
Files changed (1) hide show
  1. README.md +16 -12
README.md CHANGED
@@ -109,27 +109,31 @@ over your system message or tool calls.
109
  ## Honest limits β€” where it actually breaks
110
 
111
  We pushed it with a **hard trap-suite** (10 tasks built specifically to make a strong 4B fail β€” 8-hop
112
- dependent chains, an RPN evaluator, byte-exact files, self-referential puzzles). smeagle scored **6/10**,
113
- and *what it missed is the useful part* β€” double-check it on these:
 
114
 
115
  - **Byte-exact / multi-line formatting.** Told to write four exact lines with trailing spaces and no final
116
- newline, it collapsed them onto one line. It'll *normalize* formatting β€” if you need bytes exactly, verify.
117
- - **Self-referential / fixed-point reasoning.** "Write a sentence that says how many words it has" β†’ it wrote
118
- "…has 1 word" (the true answer is 5). Puzzles whose answer depends on the answer trip it up.
119
  - **Operand-order-sensitive code.** In a generated RPN evaluator it got the subtract/divide order wrong
120
  (`a op b` vs `b op a`) β€” the classic bug. Review order-sensitive logic it writes.
121
- - **Long single-file running state.** Overwriting one file with a running value across many steps, it can
122
- lose the thread. Give it distinct filenames or fewer hops when precision matters.
 
 
 
123
 
124
- What it *does* do well (6/10 of the same brutal suite): long dependent tool-chains, multi-step error
125
- recovery (writing broken code, reading the error, fixing it), building larger working modules with correct
126
- logic, recognizing a tool's limits, and hitting strict output templates.
127
 
128
  - It's **4B** β€” a frontier model beats it on hard long-horizon work; smeagle's job is to be the *capable
129
  little one that fits*, not to out-muscle models 100Γ— its size. And it's a **specialist** (agentic/terminal/
130
  SWE), not a generalist oracle.
131
- - v0.1. A v0.2 exists (trades a little SWE loss for agentic gains); we benchmark in the open and are still
132
- deciding which serves you best. If you have a view, tell us.
 
 
133
 
134
  ## License & rules β€” free for almost everyone
135
 
 
109
  ## Honest limits β€” where it actually breaks
110
 
111
  We pushed it with a **hard trap-suite** (10 tasks built specifically to make a strong 4B fail β€” 8-hop
112
+ dependent chains, an RPN evaluator, byte-exact files, self-referential puzzles) and re-ran the flaky ones
113
+ several times so we'd report *reliable* behaviour, not a lucky single draw. The genuine limits β€” double-check
114
+ it on these:
115
 
116
  - **Byte-exact / multi-line formatting.** Told to write four exact lines with trailing spaces and no final
117
+ newline, it collapsed them onto one line. It *normalizes* formatting β€” if you need bytes exactly, verify.
 
 
118
  - **Operand-order-sensitive code.** In a generated RPN evaluator it got the subtract/divide order wrong
119
  (`a op b` vs `b op a`) β€” the classic bug. Review order-sensitive logic it writes.
120
+ - **Long single-file running state.** Overwriting one file with a running value across many steps, it loses
121
+ the thread (0/3 in our reruns). Give it distinct filenames or fewer hops when precision matters.
122
+ - **Temperature-sensitive on one-shot precision.** Some exact/self-referential tasks it gets right *most*
123
+ of the time but not every time (e.g. "write a sentence stating its own word count" β€” ~2 in 3 at default
124
+ temp 0.6). If you need it reliable in one shot, lower the temperature or let it retry.
125
 
126
+ What it *does* do well: the full agentic suite (16/16 β€” single-tool β†’ multi-tool chains β†’ file ops β†’
127
+ code-gen+verify β†’ error recovery β†’ constrained output), plus long dependent tool-chains, building larger
128
+ working modules with correct logic, and recognizing a tool's limits.
129
 
130
  - It's **4B** β€” a frontier model beats it on hard long-horizon work; smeagle's job is to be the *capable
131
  little one that fits*, not to out-muscle models 100Γ— its size. And it's a **specialist** (agentic/terminal/
132
  SWE), not a generalist oracle.
133
+ - **Why v0.1 and not v0.2?** We built a v0.2 and benchmarked both, in the open, on all 26 tasks with reruns.
134
+ v0.2's apparent edge turned out to be temperature noise once repeated β€” no *reliable* agentic gain β€” and it
135
+ regressed slightly on held-out SWE loss. So v0.1 stays the recommended release. That's the honest call, and
136
+ we'll only ship a v0.2 when it clearly earns it.
137
 
138
  ## License & rules β€” free for almost everyone
139