hamishfromatech commited on
Commit
546e275
·
verified ·
1 Parent(s): c8b133f

Add AI Owner's Club CTA

Browse files
Files changed (1) hide show
  1. README.md +138 -128
README.md CHANGED
@@ -1,128 +1,138 @@
1
- ---
2
- library_name: transformers
3
- base_model: Qwen/Qwen3.5-4B-Base
4
- tags:
5
- - code
6
- - agent
7
- - tool-use
8
- - exploration
9
- - swe-bench
10
- - qwen3.5
11
- language:
12
- - en
13
- license: apache-2.0
14
- pipeline_tag: text-generation
15
- ---
16
-
17
- # Amaara Code Explore
18
-
19
- A 4B code **exploration** agent. Give it a bug report or a question about a repository and it
20
- works the repo with shell and file reads until it locates the relevant code — then stops. It
21
- does not modify files. That is deliberate, and measured.
22
-
23
- Built by [The A-Tech Corporation](https://theatechcorporation.com). Base model
24
- `Qwen/Qwen3.5-4B-Base`, two-stage LoRA SFT.
25
-
26
- ## What it is measured to do
27
-
28
- Evaluated on **10 SWE-bench Lite (dev) instances at 262,144 context**, 60-step budget, one
29
- run each. Metric: did the agent touch a file named in the instance's gold patch?
30
-
31
- | metric | result |
32
- |---|---|
33
- | **gold-file recall** (located the file) | **9 / 10 (90%)** |
34
- | **median tool calls to first reach it** | **2** |
35
- | mean tool calls to first reach it | 2.9 |
36
- | **episodes that MODIFIED a gold file** | **7 / 10** |
37
- | source-file writes (sed -i, redirects) | 23 |
38
- | episodes that submitted a patch | 2 / 10 |
39
-
40
- **It edits through the shell.** The evaluation scaffold exposes only a `bash` tool, so every
41
- modification is a `sed -i`, a redirect, or a heredoc — not a structured edit call. For example,
42
- on `pvlib__pvlib-python-1606` it ran
43
- `sed -i 's/np\.Inf/np.inf/g' ./pvlib/singlediode.py`, which is one of the gold-patch files.
44
- An earlier version of this card reported "0 edit calls" and concluded the model never modifies
45
- code. That was an instrument error: it counted calls to an `edit` tool the harness never
46
- provides. The model does edit files.
47
-
48
- Per instance:
49
-
50
- | instance | tool calls | first hit at |
51
- |---|---|---|
52
- | marshmallow-code__marshmallow-1343 | 60 | 2 |
53
- | marshmallow-code__marshmallow-1359 | 59 | 2 |
54
- | pvlib__pvlib-python-1072 | 60 | 2 |
55
- | pvlib__pvlib-python-1606 | 49 | 2 |
56
- | pvlib__pvlib-python-1707 | 60 | 2 |
57
- | sqlfluff__sqlfluff-1517 | 58 | 2 |
58
- | sqlfluff__sqlfluff-1625 | 59 | 3 |
59
- | sqlfluff__sqlfluff-1733 | 60 | never |
60
- | sqlfluff__sqlfluff-1763 | 59 | 8 |
61
- | sqlfluff__sqlfluff-2419 | 27 | 3 |
62
-
63
- ## What it is NOT
64
-
65
- **Its editing is coarse.** It modifies files with shell redirection and `sed -i` rather than
66
- surgical replacements, and 3 of 10 episodes never touched a gold file. It also writes a lot of
67
- scratch files (106 in these runs) — reproduction scripts and ad-hoc tests — so a working
68
- directory will accumulate junk. Treat it as a capable explorer that will attempt fixes
69
- roughly 70% of the time, not as a reliable patch generator.
70
-
71
- **Seven of ten episodes exhausted the 60-step budget** without submitting. It explores long
72
- and terminates late. Budget for that in any harness.
73
-
74
- **It was trained at 8,192 tokens** but served here at 262,144. The longer window is real
75
- (llama.cpp reports `n_ctx_slot = 262144` and an independent 67,243-token needle test passed),
76
- but quality beyond the trained length is unverified — the window prevents truncation, it does
77
- not guarantee comprehension.
78
-
79
- **Ten instances is a small sample.** 90% recall on n=10 carries wide error bars. Treat it as a
80
- strong indication, not a headline benchmark result.
81
-
82
- ## Reproduce the measurement
83
-
84
- The evaluation used [mini-swe-agent](https://github.com/SWE-agent/mini-swe-agent) against a
85
- local `llama-server`:
86
-
87
- ```bash
88
- llama-server -m amaara-code-explore-Q8_0.gguf \
89
- -ngl 99 -c 262144 --parallel 1 \
90
- -ub 512 -b 1024 --jinja --host 0.0.0.0 \
91
- --cache-type-k q8_0 --cache-type-v q8_0
92
- ```
93
-
94
- `--parallel 1` matters: `-c` is a shared pool, so with the default 4 slots a 262k setting
95
- silently gives each request 65k.
96
-
97
- ## Training
98
-
99
- | stage | steps | context | notes |
100
- |---|---|---|---|
101
- | 1 | 3,000 | 4,096 | harness + agent traces |
102
- | 2 | 2,000 | 8,192 | continued from stage 1; 28.2% of loss tokens from generalised Paperclip agent traces |
103
-
104
- Stage 2 was aimed at a specific defect: the stage-1 model repeated identical tool calls in
105
- 65% of episodes. After stage 2 the duplicate rate fell to 26%, and after raising the serving
106
- context from 32k to 98k one instance fell from 41.8% to 7.0% — part of the original figure was
107
- truncation, not looping.
108
-
109
- Training data was generalised before use: client names, paths, personal names and task codes
110
- were substituted with stable pseudonyms, and 25 banned patterns were verified absent from the
111
- final corpus.
112
-
113
- ## Quants
114
-
115
- | file | size | notes |
116
- |---|---|---|
117
- | `amaara-code-explore-F16.gguf` | 8.1 GB | reference |
118
- | `amaara-code-explore-Q8_0.gguf` | 4.3 GB | the configuration measured above |
119
- | `amaara-code-explore-Q6_K.gguf` | 3.4 GB | |
120
- | `amaara-code-explore-Q5_K_M.gguf` | 3.0 GB | |
121
- | `amaara-code-explore-Q4_K_M.gguf` | 2.6 GB | smallest |
122
-
123
- Quantisation damage has **not** been measured for this model. The numbers in this card were
124
- taken on Q8_0; if you use Q4_K_M, re-measure before relying on them.
125
-
126
- ## Licence
127
-
128
- Apache-2.0, inherited from the Qwen3.5-4B base.
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: transformers
3
+ base_model: Qwen/Qwen3.5-4B-Base
4
+ tags:
5
+ - code
6
+ - agent
7
+ - tool-use
8
+ - exploration
9
+ - swe-bench
10
+ - qwen3.5
11
+ language:
12
+ - en
13
+ license: apache-2.0
14
+ pipeline_tag: text-generation
15
+ ---
16
+
17
+ # Amaara Code Explore
18
+
19
+ A 4B code **exploration** agent. Give it a bug report or a question about a repository and it
20
+ works the repo with shell and file reads until it locates the relevant code — then stops. It
21
+ does not modify files. That is deliberate, and measured.
22
+
23
+ Built by [The A-Tech Corporation](https://theatechcorporation.com). Base model
24
+ `Qwen/Qwen3.5-4B-Base`, two-stage LoRA SFT.
25
+
26
+ ## What it is measured to do
27
+
28
+ Evaluated on **10 SWE-bench Lite (dev) instances at 262,144 context**, 60-step budget, one
29
+ run each. Metric: did the agent touch a file named in the instance's gold patch?
30
+
31
+ | metric | result |
32
+ |---|---|
33
+ | **gold-file recall** (located the file) | **9 / 10 (90%)** |
34
+ | **median tool calls to first reach it** | **2** |
35
+ | mean tool calls to first reach it | 2.9 |
36
+ | **episodes that MODIFIED a gold file** | **7 / 10** |
37
+ | source-file writes (sed -i, redirects) | 23 |
38
+ | episodes that submitted a patch | 2 / 10 |
39
+
40
+ **It edits through the shell.** The evaluation scaffold exposes only a `bash` tool, so every
41
+ modification is a `sed -i`, a redirect, or a heredoc — not a structured edit call. For example,
42
+ on `pvlib__pvlib-python-1606` it ran
43
+ `sed -i 's/np\.Inf/np.inf/g' ./pvlib/singlediode.py`, which is one of the gold-patch files.
44
+ An earlier version of this card reported "0 edit calls" and concluded the model never modifies
45
+ code. That was an instrument error: it counted calls to an `edit` tool the harness never
46
+ provides. The model does edit files.
47
+
48
+ Per instance:
49
+
50
+ | instance | tool calls | first hit at |
51
+ |---|---|---|
52
+ | marshmallow-code__marshmallow-1343 | 60 | 2 |
53
+ | marshmallow-code__marshmallow-1359 | 59 | 2 |
54
+ | pvlib__pvlib-python-1072 | 60 | 2 |
55
+ | pvlib__pvlib-python-1606 | 49 | 2 |
56
+ | pvlib__pvlib-python-1707 | 60 | 2 |
57
+ | sqlfluff__sqlfluff-1517 | 58 | 2 |
58
+ | sqlfluff__sqlfluff-1625 | 59 | 3 |
59
+ | sqlfluff__sqlfluff-1733 | 60 | never |
60
+ | sqlfluff__sqlfluff-1763 | 59 | 8 |
61
+ | sqlfluff__sqlfluff-2419 | 27 | 3 |
62
+
63
+ ## What it is NOT
64
+
65
+ **Its editing is coarse.** It modifies files with shell redirection and `sed -i` rather than
66
+ surgical replacements, and 3 of 10 episodes never touched a gold file. It also writes a lot of
67
+ scratch files (106 in these runs) — reproduction scripts and ad-hoc tests — so a working
68
+ directory will accumulate junk. Treat it as a capable explorer that will attempt fixes
69
+ roughly 70% of the time, not as a reliable patch generator.
70
+
71
+ **Seven of ten episodes exhausted the 60-step budget** without submitting. It explores long
72
+ and terminates late. Budget for that in any harness.
73
+
74
+ **It was trained at 8,192 tokens** but served here at 262,144. The longer window is real
75
+ (llama.cpp reports `n_ctx_slot = 262144` and an independent 67,243-token needle test passed),
76
+ but quality beyond the trained length is unverified — the window prevents truncation, it does
77
+ not guarantee comprehension.
78
+
79
+ **Ten instances is a small sample.** 90% recall on n=10 carries wide error bars. Treat it as a
80
+ strong indication, not a headline benchmark result.
81
+
82
+ ## Reproduce the measurement
83
+
84
+ The evaluation used [mini-swe-agent](https://github.com/SWE-agent/mini-swe-agent) against a
85
+ local `llama-server`:
86
+
87
+ ```bash
88
+ llama-server -m amaara-code-explore-Q8_0.gguf \
89
+ -ngl 99 -c 262144 --parallel 1 \
90
+ -ub 512 -b 1024 --jinja --host 0.0.0.0 \
91
+ --cache-type-k q8_0 --cache-type-v q8_0
92
+ ```
93
+
94
+ `--parallel 1` matters: `-c` is a shared pool, so with the default 4 slots a 262k setting
95
+ silently gives each request 65k.
96
+
97
+ ## Training
98
+
99
+ | stage | steps | context | notes |
100
+ |---|---|---|---|
101
+ | 1 | 3,000 | 4,096 | harness + agent traces |
102
+ | 2 | 2,000 | 8,192 | continued from stage 1; 28.2% of loss tokens from generalised Paperclip agent traces |
103
+
104
+ Stage 2 was aimed at a specific defect: the stage-1 model repeated identical tool calls in
105
+ 65% of episodes. After stage 2 the duplicate rate fell to 26%, and after raising the serving
106
+ context from 32k to 98k one instance fell from 41.8% to 7.0% — part of the original figure was
107
+ truncation, not looping.
108
+
109
+ Training data was generalised before use: client names, paths, personal names and task codes
110
+ were substituted with stable pseudonyms, and 25 banned patterns were verified absent from the
111
+ final corpus.
112
+
113
+ ## Quants
114
+
115
+ | file | size | notes |
116
+ |---|---|---|
117
+ | `amaara-code-explore-F16.gguf` | 8.1 GB | reference |
118
+ | `amaara-code-explore-Q8_0.gguf` | 4.3 GB | the configuration measured above |
119
+ | `amaara-code-explore-Q6_K.gguf` | 3.4 GB | |
120
+ | `amaara-code-explore-Q5_K_M.gguf` | 3.0 GB | |
121
+ | `amaara-code-explore-Q4_K_M.gguf` | 2.6 GB | smallest |
122
+
123
+ Quantisation damage has **not** been measured for this model. The numbers in this card were
124
+ taken on Q8_0; if you use Q4_K_M, re-measure before relying on them.
125
+
126
+ ## Licence
127
+
128
+ Apache-2.0, inherited from the Qwen3.5-4B base.
129
+
130
+ ---
131
+
132
+ ## Join the AI Owner's Club
133
+
134
+ We build these models in the open with a community of people who want to **own** their AI instead
135
+ of renting it. If you're training your own models, running them on your own hardware, and refusing
136
+ to accept that you have to pay per token forever, you'll fit right in.
137
+
138
+ **[Join the AI Owner's Club on Skool](https://www.skool.com/open-source-ai-builders-club/about)**