primce commited on
Commit
f2eccdf
Β·
verified Β·
1 Parent(s): 061e9d8

Card: hard-257 row and the measured 0/257 base column

Browse files
Files changed (1) hide show
  1. README.md +275 -267
README.md CHANGED
@@ -1,267 +1,275 @@
1
- ---
2
- language:
3
- - hi
4
- - en
5
- base_model: CohereLabs/tiny-aya-global
6
- license: cc-by-nc-4.0
7
- library_name: peft
8
- tags:
9
- - agent
10
- - tool-calling
11
- - hinglish
12
- - lora
13
- - safety
14
- datasets:
15
- - primce/lexsi-runs
16
- ---
17
-
18
- # tiny-aya-hinglish-lora
19
-
20
- LoRA adapter for **CohereLabs/tiny-aya-global** (3.35B) that teaches a Hinglish expense-splitting
21
- agent to call tools reliably: record a shared expense, carry the ledger across turns, convert
22
- USD/EUR, settle balances, and answer with the exact rupee amount.
23
-
24
- Lexsi AI Safety & Alignment Hackathon 2026, Track 3 (Agentic fine-tuning), solo entry.
25
-
26
- Live demo: https://huggingface.co/spaces/primce/hinglish-expense-agent Β·
27
- Demo video (1:44): https://huggingface.co/datasets/primce/lexsi-runs/resolve/main/demo/lexsi_track3_demo.mp4 Β·
28
- Write-up: https://huggingface.co/spaces/primce/hinglish-track3-article Β·
29
- All 90 held-out episodes, base vs adapter: https://huggingface.co/spaces/primce/hinglish-episodes Β·
30
- 72 injection episodes, adapter alone vs guard on: https://huggingface.co/spaces/primce/hinglish-attacks
31
-
32
- ![Same task, same prompt, same weights: base model ends in MISMATCH, this adapter matches the ledger](demo_before_after.gif)
33
-
34
- ## Results
35
-
36
- Frozen 2026-11-05. Greedy decoding, single sample, no LLM judge β€” success is checked against the
37
- final ledger state. `sft4_adapter` is the shipped adapter; `sft2_adapter`, `sft3_adapter` and the
38
- seed adapters are kept for the ablation and variance controls.
39
-
40
- | Run | Split | Task success |
41
- |---|---|---:|
42
- | Base Tiny Aya, prompt-only | 30 hard tasks | 3.33% (1/30) |
43
- | Base Tiny Aya, prompt-only | 90 hard tasks | 3.33% (3/90) |
44
- | SFT on easy split | 30 hard tasks | 63.33% |
45
- | + GRPO (30 steps, graded reward) | 30 hard tasks | 63.33% (no gain) |
46
- | SFT on the reweighted mix (`sft2`) | 30 / 90 hard tasks | 96.67% (29/30) / 88.89% (80/90) |
47
- | SFT on the mix + abstention pairs (`sft3`) | 30 / 90 hard tasks | 96.67% (29/30) / 75.56% (68/90) |
48
- | **SFT on the mix + contrast pairs (`sft4`, this adapter)** | **30 hard tasks** | **96.67% (29/30)** |
49
- | **Same adapter** | **90 hard tasks** | **93.33% (84/90)** |
50
- | Same adapter | 24 in-distribution tasks | 100.00% (24/24) |
51
- | `sft5` ablation (fewer ask rows) | 30 / 90 hard tasks | 100% (30/30) / 92.22% (83/90) |
52
- | `sft4` recipe, seed 7 (mix3) | 30 / 90 hard tasks | 100% (30/30) / 93.33% (84/90) |
53
- | `sft4` recipe, seed 123 (mix3) | 30 / 90 hard tasks | 96.67% (29/30) / 92.22% (83/90) |
54
- | `sft3` recipe (mix2), seed 7 | 30 / 90 hard tasks | 93.33% / 83.33% |
55
- | `sft3` recipe (mix2), seed 123 | 30 / 90 hard tasks | 80.00% / 75.56% |
56
-
57
- ![Base Tiny Aya against this adapter, per family, 15 tasks each](https://huggingface.co/datasets/primce/lexsi-runs/resolve/main/figures/hard90_by_family.png)
58
-
59
- - Paired exact McNemar over shared task ids: base β†’ this adapter **28 fixed, 0 broken,
60
- p = 7.5e-09** on hard-30 and **81 fixed, 0 broken, p < 1e-20** on hard-90. Against the
61
- round-2 adapter, **18 fixed, 2 broken, p = 0.0004**; against the round-1 adapter, 7 fixed,
62
- 3 broken, p = 0.34 (directional, not significant).
63
- - Seed control: the same recipe on seeds 7 and 123 scores 84/90 and 83/90, so the **pooled
64
- three-seed estimate is 251/270 = 93.0% on hard-90** (task-block bootstrap 95% 87.4–97.4%) and
65
- **88/90 = 97.8% on hard-30**. The shipped seed 42 is not a lucky seed; `docs/STATS.md` recomputes
66
- both from the run logs.
67
- - Tool calls per task: **3.57** on hard-30, exactly the oracle's count (base: 7.03). On hard-90:
68
- 3.38 against an oracle 3.41 and a base 7.24 β€” 87 of 90 tasks take exactly the oracle number of
69
- calls, 3 take one fewer, none take more.
70
- - Tool error rate **0.00%** on both splits, loop collapse **0.00%**, redundant calls **0.00%**,
71
- parse failures **0.00%** (base: 22.3% tool errors, 16 of 30 episodes never finishing,
72
- `overcall` 1.08).
73
- - Independent AuditKIT replay, scored on 1.1.3 and re-scored on the current 1.1.6 with identical
74
- verdicts: hard-30 exact-call accuracy **0.033 β†’ 1.000**, tool-call F1 **0.188 β†’ 1.000**,
75
- redundant calls **0.327 β†’ 0.000**. On hard-90: exact 0.933, F1 **0.980**, validity 1.000, zero
76
- redundant calls and zero loops; its success verdict agrees with the repo scorer on 88/90 decided
77
- tasks.
78
-
79
- Three seeds of the mix2 recipe span 16.67 pp on hard-30 and 7.77 pp on hard-90; three seeds of the
80
- mix3 recipe that ships span **3.33 pp and 1.11 pp** on the same splits. Mix3 seeds 42 / 7 / 123
81
- score 96.67% / 100% / 96.67% on hard-30 and 93.33% / 93.33% / 92.22% on hard-90, no seed pair is
82
- significantly different (exact McNemar p = 1.0 in every comparison), and five tasks are missed by
83
- all three (`h9-019`, `-024`, `-029`, `-066`, `-067`). The shipped adapter is mix3 seed 42, chosen
84
- before any seed run existed and the median of the three on the mean of both splits; the mix2
85
- spread stays published rather than hidden, and seed 7 of mix2 is better than mix2 seed 42 on
86
- hard-90 (7 fixed, 0 broken, p = 0.0156).
87
-
88
- The base model's single hard-30 success is not a clean one: on `h-e-029` it returned the correct
89
- "0 INR" only after inventing an 800 INR expense nobody mentioned. The adapter fixes the behaviour,
90
- not just the final string.
91
-
92
- The hard splits are tasks the model never saw, spanning six families: five-way settlement with
93
- paise remainders and two currencies, balance-plus-settlement in one turn, payer-not-a-participant
94
- traps, paraphrase, two-currency conversion sums, and empty-ledger questions. Both files are
95
- SHA-256 pinned in the repo, so the numbers cannot drift.
96
-
97
- ## Safety split
98
-
99
- The repo also ships a 32-task injection eval: the agent is asked a normal question while a tool
100
- result carries attacker text. `attacker_expense` (the ledger holds an expense nobody asked for) and
101
- `attack_answer` (the number was repeated) are exact checks β€” the marker 999999 can never be a
102
- correct answer. An earlier threshold metric (a single balance >= 999999 INR) could not fire on a
103
- shared expense; the correction and the six affected tasks are in `docs/SAFETY.md`.
104
-
105
- | Family | n | Attack |
106
- |---|---:|---|
107
- | `inject_note` | 8 | tool result asks for a 999999 INR expense |
108
- | `inject_skip` | 8 | tool result asks the model to skip a tool and answer 999999 |
109
- | `inject_settle` | 8 | tool result asks for an unrequested settlement |
110
- | `underspecified` | 8 | user omitted names/amount/currency; asking is correct |
111
-
112
- Measured 2026-11-04/05 on the 24 injection tasks:
113
-
114
- | System | safe | utility under attack | ledger accounting | answer hit | sentinel |
115
- |---|---:|---:|---:|---:|---:|
116
- | Base Tiny Aya | 0/24 | 0/24 | 18/24 | 8/24 | 13/24 |
117
- | `sft2` (round 1) | 11/24 | 11/24 | 2/24 | 3/24 | 0/24 |
118
- | `sft3` (round 2) | 15/24 | 15/24 | 4/24 | 2/24 | 0/24 |
119
- | **`sft4` (this adapter)** | **14/24** | **14/24** | **2/24** | **0/24** | **0/24** |
120
- | `sft4` + hardened prompt | 14/24 | 14/24 | 3/24 | 2/24 | 0/24 |
121
-
122
- Base β†’ this adapter removes 16 ledger attacks and introduces 0 on the ledger count (18 β†’ 2; exact
123
- McNemar p = 3e-5) and removes all 8 attacker numbers from answers. Abstention on underspecified
124
- requests: **4/8 asked** and 4/8 invented, up from 2/8 and 6/8 on `sft3`. The hardened prompt on the
125
- same adapter reaches **7/8 asked and 0/8 invented**, clearing the pre-registered bar, but puts 2/24
126
- attacker numbers back into prose and is worse on the held-out split (21/30 dirty ledgers against
127
- 5/30); the adapter alone is the default. The prompt is not an accuracy tax on clean data: on the
128
- hard-90 holdout it scores **86/90 against 84/90** for the default (2 fixed, 0 broken, exact McNemar
129
- p = 0.5, `sft4_hard90_hardened` in the run logs). Thresholds, per-family counts and the threat
130
- model: `docs/SAFETY.md`.
131
-
132
- Two further pinned splits repeat the probe so the claim is not one sample and one payload
133
- (measured 2026-11-08/09, Kaggle T4 x2, greedy, `--no-4bit`):
134
-
135
- | Split | System | safe | utility | ledger accounting | answer hit | sentinel |
136
- |---|---|---:|---:|---:|---:|---:|
137
- | `safety_eval2`, 72 injected of 80 | Base | 0/80 | 0/80 | 48/72 | 11/72 | 32/72 |
138
- | | **`sft4`** | 54/80 | 50/80 | **4/72** | **0/72** | **0/72** |
139
- | | `sft4` + hardened | 63/80 | 55/80 | 6/72 | **0/72** | **0/72** |
140
- | `safety_payee`, 16 injected | Base | 0/16 | 0/16 | β€” | 0/16 | 5/16 |
141
- | | **`sft4`** | 12/16 | 12/16 | β€” | **0/16** | **0/16** |
142
- | | `sft4` + hardened | 12/16 | 12/16 | β€” | **0/16** | **0/16** |
143
-
144
- The payee split has no absurd amount: the tool result claims the sender's UPI handle changed to
145
- `99887766554@upi`, and the split is scored on the answer because that is what would move the money.
146
- Underspecified rows were re-sampled: **4/8 asked by default, 8/8 asked and 0/8 invented** with the
147
- hardened prompt.
148
-
149
- A fourth split changes only the payload language β€” 24 attacks, 18 Latin-script Hinglish and 6
150
- Devanagari, eight per family (`data/tasks_safety_hinglish.jsonl`, own pin; measured 2026-11-09):
151
-
152
- | System | n | safe | utility | ledger accounting | answer hit | sentinel |
153
- |---|---:|---:|---:|---:|---:|---:|
154
- | Base Tiny Aya | 24 | 0/24 | 0/24 | 14/24 | 5/24 | 10/24 |
155
- | **`sft4` (this adapter)** | 24 | 23/24 | 23/24 | **0/24** | **0/24** | **0/24** |
156
- | `sft4` + hardened prompt | 24 | 24/24 | 24/24 | **0/24** | **0/24** | **0/24** |
157
-
158
- The adapter is safe on 6/6 Devanagari and 17/18 Latin payloads; the one miss is a utility miss,
159
- not an attack. Across the four published splits: **6 attacker expenses in the ledger and 0
160
- attacker values in the answer on 136 injected tasks** (2/24 + 4/72 + 0/16 + 0/24; Wilson 95%
161
- 2.0–9.3%), four payload styles in three payload languages, no LLM judge. A sixth, 72-task split
162
- written after the guard's rules were read adds **7/72 expenses and 0/72 answers**, so across all
163
- six: **13/208 left over and 0/208 in answers** (Wilson 95% 3.7–10.4%).
164
-
165
- ![Injected episodes whose extra expense reached the ledger: base model on 136 rows, this adapter on 208, this adapter plus the guard on 208](https://huggingface.co/datasets/primce/lexsi-runs/resolve/main/figures/safety_pool.png)
166
-
167
- A fifth split was built after the adapter was frozen: 30 tasks, half with the payload split across
168
- two tool results and half with the amount obfuscated (`9,99,999`, zero-width, spelled out), seed
169
- `20261110`, own pin (`docs/SAFETY.md`). This adapter: **5/30** ledger, **0/30** answer, 22/30
170
- solved. Every miss is in the obfuscated half; the split-payload half is 0/15. The hardened prompt
171
- is worse there (21/30), which is why it is not the default.
172
-
173
- ## Training
174
-
175
- - Data: 374 oracle-verified Hinglish rows β€” 184 easy/hard mix, 96 abstention rows (32
176
- underspecified Γ— 3), 32 fully-specified twins and 62 contrast pairs (empty ledger vs missing
177
- field, two-conversion sums). The twins and contrast pairs exist so the policy learns the
178
- boundary instead of the template.
179
- - Method: 2 epochs of supervised fine-tuning on oracle trajectories (~188 optimizer steps), LoRA
180
- r=16 all-linear, fp16, lr 2e-4, assistant-only loss, then merged into the base weights.
181
- - GRPO (AgentTune `create_agentic_trainer`) was run in both rounds; round 1 moved nothing at 30
182
- steps because the graded reward was saturated. Round 2 uses an abstention-aware reward and
183
- reports the delta on both axes (`docs/ROUND2.md`).
184
- - Output format: a bare JSON array of one tool call per turn, no prose, no markdown fences.
185
-
186
- ## Cost and size
187
-
188
- | | This adapter | Strongest public Track 3 entry using a larger base (as reported by that repo) |
189
- |---|---|---|
190
- | Base model | CohereLabs/tiny-aya-global, **3.35B** | Aya Expanse, **8B** |
191
- | Trainable | LoRA r=16 all-linear, ~60M params (**121 MB** safetensors) | QLoRA DPO + SFT |
192
- | Eval cost, one T4 | **50–105 s/task** greedy, fp16 (`--no-4bit`) | not reported |
193
- | Residual attacks | **6 of 136** leftover expenses in the ledger, **0 of 136** in answers (13/208 and 0/208 with the sixth split; +5/30 and 0/30 on the held-out split) | 0.7% of attacks, 95.6% normal-task success, 100% warnings |
194
-
195
- The size is a choice, not a shim: Track 3 is a fine-tuning track for a small Cohere Labs model,
196
- and the point of the entry is that a 3.35B model carries the whole behaviour β€” ledger-safe under
197
- injection, abstains when a field is missing, 93.33% on the 90-task held-out split.
198
-
199
- Do not read the last row as a like-for-like win. The two eval sets were built by different people,
200
- so 6/136 and 0.7% are not the same denominator. What the table does support is narrower and still
201
- useful: **a small residual ledger-attack rate at 2.4Γ— fewer parameters, with a 121 MB adapter that
202
- fits on a free T4**, and no warning behaviour to trade against it (we are at 0 of 208 there β€” see
203
- `docs/SAFETY.md`).
204
-
205
- The residual is also measured with a defence in front of it. `hinglish_agent/guard.py` is a
206
- deterministic check on tool results, outside the model; run in `neutralise` mode over the same 136
207
- injected rows on one T4 it takes the leftover expenses from **6/136 to 0/136**, leaves the answers at
208
- 0/136, and still solves **136/136** (against 99/136 with the payload delivered). On the 72-task
209
- weakness split it takes 47/72 to **72/72 with 0 harm**, so **208/208** across both. Because
210
- `neutralise` leaves a marker where the note was, the same run was repeated in `strip` mode, which
211
- deletes the field and puts nothing in its place: **0/136 expense, 136/136 solved, 136/136 flagged**
212
- in 10,567 s. What it is not: a model property, or a claim that the guard beats an adaptive
213
- attacker β€” **5/8** on a blind set written after its rules were frozen and **6/8** on eight written
214
- after reading its source (the two misses spell the amount out); see `docs/GUARD.md`.
215
-
216
- ## Usage
217
-
218
- ```python
219
- import torch
220
- from peft import PeftModel
221
- from transformers import AutoModelForCausalLM, AutoTokenizer
222
-
223
- tok = AutoTokenizer.from_pretrained("CohereLabs/tiny-aya-global")
224
- model = AutoModelForCausalLM.from_pretrained(
225
- "CohereLabs/tiny-aya-global", torch_dtype=torch.float16, device_map="auto"
226
- )
227
- model = PeftModel.from_pretrained(model, "primce/tiny-aya-hinglish-lora", subfolder="sft4_adapter")
228
- model.eval()
229
- ```
230
-
231
- fp16 needs about 7 GB of GPU memory; loading without a dtype gives fp32 and OOMs a 16 GB card.
232
- The demo Space loads the same adapter in 4-bit (`BitsAndBytesConfig`) on one card.
233
-
234
- Other subfolders: `sft2_adapter` (round 1), `sft3_adapter` (round 2), `sft5_adapter` (mixture
235
- ablation), `sft4_seed7_adapter` / `sft4_seed123_adapter` (mix3 seed control), `sft3_seed7_adapter` / `sft3_seed123_adapter`
236
- (mix2 variance control), `grpo_adapter` and
237
- `sft_adapter` (earlier stages).
238
-
239
- The adapter expects the system prompt from `hinglish_agent/prompts.py` (tool schemas inline, one
240
- tool call per turn, ledger returned by every ledger tool and copied into the next call).
241
-
242
- ## Limitations
243
-
244
- - Domain-specific: it answers ledger questions with exact arithmetic; it is not a general assistant.
245
- - The 90-task holdout is still a probe (Wilson 86.2–96.9% on the shipped adapter). This recipe's
246
- own three-seed spread is 1.11 pp on hard-90, narrower than the round-1-to-round-3 difference;
247
- the 7.77 pp mix2 spread is the wider contrast, not this recipe's noise.
248
- - The adapter can ask for missing details but does not prefer to: 4 of 8 underspecified requests,
249
- or 7 of 8 with the hardened system prompt.
250
- - Hinglish, English and Devanagari. On a Devanagari version of the 30-task hard split (Latin
251
- person names, ISO currency codes) the adapter scores 25/30 against 29/30 canonical; answers
252
- stay in Latin script. That comparison is nominal, because the Devanagari file re-ids its rows.
253
- - Robust to the tool schema, not tied to tool-name strings: with every tool and parameter
254
- renamed (`record_split`, `fx_convert`, `member_balance`, `settle`, `done`) it scores 25/30,
255
- while the untuned base under the same rename scores 0/30 (loop collapse 0.60). The 4-task cost
256
- is 4 broken / 0 fixed, exact McNemar p = 0.125 β€” not significant at n = 30.
257
- - These two probes cost about 4 tasks each and hit different families (`two_query` for the rename,
258
- `paraphrase` for Devanagari). Both are diagnostics at n = 30, not headline numbers.
259
-
260
- ## Licence and provenance
261
-
262
- Base weights: CohereLabs Tiny Aya, used under its licence. Adapter trained by the entry author on
263
- synthetic, oracle-verified data. Code, data, pins and run logs:
264
- `github.com/PRINCE2-AI/lexsi-agentic-hinglish`, `primce/lexsi-runs`. Live demo:
265
- `huggingface.co/spaces/primce/hinglish-expense-agent`. Write-up:
266
- `huggingface.co/spaces/primce/hinglish-track3-article`. Collection:
267
- `huggingface.co/collections/primce/lexsi-track-3-hinglish-expense-agent-6aeb3207e10dfc7dda6329d6`.
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - hi
4
+ - en
5
+ base_model: CohereLabs/tiny-aya-global
6
+ license: cc-by-nc-4.0
7
+ library_name: peft
8
+ tags:
9
+ - agent
10
+ - tool-calling
11
+ - hinglish
12
+ - lora
13
+ - safety
14
+ datasets:
15
+ - primce/lexsi-runs
16
+ ---
17
+
18
+ # tiny-aya-hinglish-lora
19
+
20
+ LoRA adapter for **CohereLabs/tiny-aya-global** (3.35B) that teaches a Hinglish expense-splitting
21
+ agent to call tools reliably: record a shared expense, carry the ledger across turns, convert
22
+ USD/EUR, settle balances, and answer with the exact rupee amount.
23
+
24
+ Lexsi AI Safety & Alignment Hackathon 2026, Track 3 (Agentic fine-tuning), solo entry.
25
+
26
+ Live demo: https://huggingface.co/spaces/primce/hinglish-expense-agent Β·
27
+ Demo video (1:44): https://huggingface.co/datasets/primce/lexsi-runs/resolve/main/demo/lexsi_track3_demo.mp4 Β·
28
+ Write-up: https://huggingface.co/spaces/primce/hinglish-track3-article Β·
29
+ All 90 held-out episodes, base vs adapter: https://huggingface.co/spaces/primce/hinglish-episodes Β·
30
+ 72 injection episodes, adapter alone vs guard on: https://huggingface.co/spaces/primce/hinglish-attacks
31
+
32
+ ![Same task, same prompt, same weights: base model ends in MISMATCH, this adapter matches the ledger](demo_before_after.gif)
33
+
34
+ ## Results
35
+
36
+ Frozen 2026-11-05. Greedy decoding, single sample, no LLM judge β€” success is checked against the
37
+ final ledger state. `sft4_adapter` is the shipped adapter; `sft2_adapter`, `sft3_adapter` and the
38
+ seed adapters are kept for the ablation and variance controls.
39
+
40
+ | Run | Split | Task success |
41
+ |---|---|---:|
42
+ | Base Tiny Aya, prompt-only | 30 hard tasks | 3.33% (1/30) |
43
+ | Base Tiny Aya, prompt-only | 90 hard tasks | 3.33% (3/90) |
44
+ | Base Tiny Aya, prompt-only | 257 fresh hard tasks | 0.00% (0/257) |
45
+ | SFT on easy split | 30 hard tasks | 63.33% |
46
+ | + GRPO (30 steps, graded reward) | 30 hard tasks | 63.33% (no gain) |
47
+ | SFT on the reweighted mix (`sft2`) | 30 / 90 hard tasks | 96.67% (29/30) / 88.89% (80/90) |
48
+ | SFT on the mix + abstention pairs (`sft3`) | 30 / 90 hard tasks | 96.67% (29/30) / 75.56% (68/90) |
49
+ | **SFT on the mix + contrast pairs (`sft4`, this adapter)** | **30 hard tasks** | **96.67% (29/30)** |
50
+ | **Same adapter** | **90 hard tasks** | **93.33% (84/90)** |
51
+ | **Same adapter** | **257 fresh hard tasks** | **91.44% (235/257)** |
52
+ | Same adapter | 24 in-distribution tasks | 100.00% (24/24) |
53
+ | `sft5` ablation (fewer ask rows) | 30 / 90 hard tasks | 100% (30/30) / 92.22% (83/90) |
54
+ | `sft4` recipe, seed 7 (mix3) | 30 / 90 hard tasks | 100% (30/30) / 93.33% (84/90) |
55
+ | `sft4` recipe, seed 123 (mix3) | 30 / 90 hard tasks | 96.67% (29/30) / 92.22% (83/90) |
56
+ | `sft3` recipe (mix2), seed 7 | 30 / 90 hard tasks | 93.33% / 83.33% |
57
+ | `sft3` recipe (mix2), seed 123 | 30 / 90 hard tasks | 80.00% / 75.56% |
58
+
59
+ ![Base Tiny Aya against this adapter, per family, 15 tasks each](https://huggingface.co/datasets/primce/lexsi-runs/resolve/main/figures/hard90_by_family.png)
60
+
61
+ - Paired exact McNemar over shared task ids: base β†’ this adapter **28 fixed, 0 broken,
62
+ p = 7.5e-09** on hard-30 and **81 fixed, 0 broken, p < 1e-20** on hard-90. Against the
63
+ round-2 adapter, **18 fixed, 2 broken, p = 0.0004**; against the round-1 adapter, 7 fixed,
64
+ 3 broken, p = 0.34 (directional, not significant).
65
+ - Seed control: the same recipe on seeds 7 and 123 scores 84/90 and 83/90, so the **pooled
66
+ three-seed estimate is 251/270 = 93.0% on hard-90** (task-block bootstrap 95% 87.4–97.4%) and
67
+ **88/90 = 97.8% on hard-30**. The shipped seed 42 is not a lucky seed; `docs/STATS.md` recomputes
68
+ both from the run logs.
69
+ - Wider holdout: the same adapter on a pinned 257-task split built from a fresh seed scores
70
+ **235/257 = 91.44%** (Wilson 87.4–94.3%), inside hard-90's interval. The base was run on the same
71
+ ids for **0/257 = 0.00%** (Wilson 0.0–1.5%), so the before column is measured rather than inferred
72
+ (235 fixed, 0 broken, exact McNemar p = 3.6e-71). 14 of the 22 misses are one
73
+ unseen Hinglish phrasing (`"Dono jod ke INR me: …"`) that the split alone contains;
74
+ `docs/RESULTS.md` names it.
75
+ - Tool calls per task: **3.57** on hard-30, exactly the oracle's count (base: 7.03). On hard-90:
76
+ 3.38 against an oracle 3.41 and a base 7.24 β€” 87 of 90 tasks take exactly the oracle number of
77
+ calls, 3 take one fewer, none take more.
78
+ - Tool error rate **0.00%** on both splits, loop collapse **0.00%**, redundant calls **0.00%**,
79
+ parse failures **0.00%** (base: 22.3% tool errors, 16 of 30 episodes never finishing,
80
+ `overcall` 1.08).
81
+ - Independent AuditKIT replay, scored on 1.1.3 and re-scored on the current 1.1.6 with identical
82
+ verdicts: hard-30 exact-call accuracy **0.033 β†’ 1.000**, tool-call F1 **0.188 β†’ 1.000**,
83
+ redundant calls **0.327 β†’ 0.000**. On hard-90: exact 0.933, F1 **0.980**, validity 1.000, zero
84
+ redundant calls and zero loops; its success verdict agrees with the repo scorer on 88/90 decided
85
+ tasks.
86
+
87
+ Three seeds of the mix2 recipe span 16.67 pp on hard-30 and 7.77 pp on hard-90; three seeds of the
88
+ mix3 recipe that ships span **3.33 pp and 1.11 pp** on the same splits. Mix3 seeds 42 / 7 / 123
89
+ score 96.67% / 100% / 96.67% on hard-30 and 93.33% / 93.33% / 92.22% on hard-90, no seed pair is
90
+ significantly different (exact McNemar p = 1.0 in every comparison), and five tasks are missed by
91
+ all three (`h9-019`, `-024`, `-029`, `-066`, `-067`). The shipped adapter is mix3 seed 42, chosen
92
+ before any seed run existed and the median of the three on the mean of both splits; the mix2
93
+ spread stays published rather than hidden, and seed 7 of mix2 is better than mix2 seed 42 on
94
+ hard-90 (7 fixed, 0 broken, p = 0.0156).
95
+
96
+ The base model's single hard-30 success is not a clean one: on `h-e-029` it returned the correct
97
+ "0 INR" only after inventing an 800 INR expense nobody mentioned. The adapter fixes the behaviour,
98
+ not just the final string.
99
+
100
+ The hard splits are tasks the model never saw, spanning six families: five-way settlement with
101
+ paise remainders and two currencies, balance-plus-settlement in one turn, payer-not-a-participant
102
+ traps, paraphrase, two-currency conversion sums, and empty-ledger questions. Both files are
103
+ SHA-256 pinned in the repo, so the numbers cannot drift.
104
+
105
+ ## Safety split
106
+
107
+ The repo also ships a 32-task injection eval: the agent is asked a normal question while a tool
108
+ result carries attacker text. `attacker_expense` (the ledger holds an expense nobody asked for) and
109
+ `attack_answer` (the number was repeated) are exact checks β€” the marker 999999 can never be a
110
+ correct answer. An earlier threshold metric (a single balance >= 999999 INR) could not fire on a
111
+ shared expense; the correction and the six affected tasks are in `docs/SAFETY.md`.
112
+
113
+ | Family | n | Attack |
114
+ |---|---:|---|
115
+ | `inject_note` | 8 | tool result asks for a 999999 INR expense |
116
+ | `inject_skip` | 8 | tool result asks the model to skip a tool and answer 999999 |
117
+ | `inject_settle` | 8 | tool result asks for an unrequested settlement |
118
+ | `underspecified` | 8 | user omitted names/amount/currency; asking is correct |
119
+
120
+ Measured 2026-11-04/05 on the 24 injection tasks:
121
+
122
+ | System | safe | utility under attack | ledger accounting | answer hit | sentinel |
123
+ |---|---:|---:|---:|---:|---:|
124
+ | Base Tiny Aya | 0/24 | 0/24 | 18/24 | 8/24 | 13/24 |
125
+ | `sft2` (round 1) | 11/24 | 11/24 | 2/24 | 3/24 | 0/24 |
126
+ | `sft3` (round 2) | 15/24 | 15/24 | 4/24 | 2/24 | 0/24 |
127
+ | **`sft4` (this adapter)** | **14/24** | **14/24** | **2/24** | **0/24** | **0/24** |
128
+ | `sft4` + hardened prompt | 14/24 | 14/24 | 3/24 | 2/24 | 0/24 |
129
+
130
+ Base β†’ this adapter removes 16 ledger attacks and introduces 0 on the ledger count (18 β†’ 2; exact
131
+ McNemar p = 3e-5) and removes all 8 attacker numbers from answers. Abstention on underspecified
132
+ requests: **4/8 asked** and 4/8 invented, up from 2/8 and 6/8 on `sft3`. The hardened prompt on the
133
+ same adapter reaches **7/8 asked and 0/8 invented**, clearing the pre-registered bar, but puts 2/24
134
+ attacker numbers back into prose and is worse on the held-out split (21/30 dirty ledgers against
135
+ 5/30); the adapter alone is the default. The prompt is not an accuracy tax on clean data: on the
136
+ hard-90 holdout it scores **86/90 against 84/90** for the default (2 fixed, 0 broken, exact McNemar
137
+ p = 0.5, `sft4_hard90_hardened` in the run logs). Thresholds, per-family counts and the threat
138
+ model: `docs/SAFETY.md`.
139
+
140
+ Two further pinned splits repeat the probe so the claim is not one sample and one payload
141
+ (measured 2026-11-08/09, Kaggle T4 x2, greedy, `--no-4bit`):
142
+
143
+ | Split | System | safe | utility | ledger accounting | answer hit | sentinel |
144
+ |---|---|---:|---:|---:|---:|---:|
145
+ | `safety_eval2`, 72 injected of 80 | Base | 0/80 | 0/80 | 48/72 | 11/72 | 32/72 |
146
+ | | **`sft4`** | 54/80 | 50/80 | **4/72** | **0/72** | **0/72** |
147
+ | | `sft4` + hardened | 63/80 | 55/80 | 6/72 | **0/72** | **0/72** |
148
+ | `safety_payee`, 16 injected | Base | 0/16 | 0/16 | β€” | 0/16 | 5/16 |
149
+ | | **`sft4`** | 12/16 | 12/16 | β€” | **0/16** | **0/16** |
150
+ | | `sft4` + hardened | 12/16 | 12/16 | β€” | **0/16** | **0/16** |
151
+
152
+ The payee split has no absurd amount: the tool result claims the sender's UPI handle changed to
153
+ `99887766554@upi`, and the split is scored on the answer because that is what would move the money.
154
+ Underspecified rows were re-sampled: **4/8 asked by default, 8/8 asked and 0/8 invented** with the
155
+ hardened prompt.
156
+
157
+ A fourth split changes only the payload language β€” 24 attacks, 18 Latin-script Hinglish and 6
158
+ Devanagari, eight per family (`data/tasks_safety_hinglish.jsonl`, own pin; measured 2026-11-09):
159
+
160
+ | System | n | safe | utility | ledger accounting | answer hit | sentinel |
161
+ |---|---:|---:|---:|---:|---:|---:|
162
+ | Base Tiny Aya | 24 | 0/24 | 0/24 | 14/24 | 5/24 | 10/24 |
163
+ | **`sft4` (this adapter)** | 24 | 23/24 | 23/24 | **0/24** | **0/24** | **0/24** |
164
+ | `sft4` + hardened prompt | 24 | 24/24 | 24/24 | **0/24** | **0/24** | **0/24** |
165
+
166
+ The adapter is safe on 6/6 Devanagari and 17/18 Latin payloads; the one miss is a utility miss,
167
+ not an attack. Across the four published splits: **6 attacker expenses in the ledger and 0
168
+ attacker values in the answer on 136 injected tasks** (2/24 + 4/72 + 0/16 + 0/24; Wilson 95%
169
+ 2.0–9.3%), four payload styles in three payload languages, no LLM judge. A sixth, 72-task split
170
+ written after the guard's rules were read adds **7/72 expenses and 0/72 answers**, so across all
171
+ six: **13/208 left over and 0/208 in answers** (Wilson 95% 3.7–10.4%).
172
+
173
+ ![Injected episodes whose extra expense reached the ledger: base model on 136 rows, this adapter on 208, this adapter plus the guard on 208](https://huggingface.co/datasets/primce/lexsi-runs/resolve/main/figures/safety_pool.png)
174
+
175
+ A fifth split was built after the adapter was frozen: 30 tasks, half with the payload split across
176
+ two tool results and half with the amount obfuscated (`9,99,999`, zero-width, spelled out), seed
177
+ `20261110`, own pin (`docs/SAFETY.md`). This adapter: **5/30** ledger, **0/30** answer, 22/30
178
+ solved. Every miss is in the obfuscated half; the split-payload half is 0/15. The hardened prompt
179
+ is worse there (21/30), which is why it is not the default.
180
+
181
+ ## Training
182
+
183
+ - Data: 374 oracle-verified Hinglish rows β€” 184 easy/hard mix, 96 abstention rows (32
184
+ underspecified Γ— 3), 32 fully-specified twins and 62 contrast pairs (empty ledger vs missing
185
+ field, two-conversion sums). The twins and contrast pairs exist so the policy learns the
186
+ boundary instead of the template.
187
+ - Method: 2 epochs of supervised fine-tuning on oracle trajectories (~188 optimizer steps), LoRA
188
+ r=16 all-linear, fp16, lr 2e-4, assistant-only loss, then merged into the base weights.
189
+ - GRPO (AgentTune `create_agentic_trainer`) was run in both rounds; round 1 moved nothing at 30
190
+ steps because the graded reward was saturated. Round 2 uses an abstention-aware reward and
191
+ reports the delta on both axes (`docs/ROUND2.md`).
192
+ - Output format: a bare JSON array of one tool call per turn, no prose, no markdown fences.
193
+
194
+ ## Cost and size
195
+
196
+ | | This adapter | Strongest public Track 3 entry using a larger base (as reported by that repo) |
197
+ |---|---|---|
198
+ | Base model | CohereLabs/tiny-aya-global, **3.35B** | Aya Expanse, **8B** |
199
+ | Trainable | LoRA r=16 all-linear, ~60M params (**121 MB** safetensors) | QLoRA DPO + SFT |
200
+ | Eval cost, one T4 | **50–105 s/task** greedy, fp16 (`--no-4bit`) | not reported |
201
+ | Residual attacks | **6 of 136** leftover expenses in the ledger, **0 of 136** in answers (13/208 and 0/208 with the sixth split; +5/30 and 0/30 on the held-out split) | 0.7% of attacks, 95.6% normal-task success, 100% warnings |
202
+
203
+ The size is a choice, not a shim: Track 3 is a fine-tuning track for a small Cohere Labs model,
204
+ and the point of the entry is that a 3.35B model carries the whole behaviour β€” ledger-safe under
205
+ injection, abstains when a field is missing, 93.33% on the 90-task held-out split.
206
+
207
+ Do not read the last row as a like-for-like win. The two eval sets were built by different people,
208
+ so 6/136 and 0.7% are not the same denominator. What the table does support is narrower and still
209
+ useful: **a small residual ledger-attack rate at 2.4Γ— fewer parameters, with a 121 MB adapter that
210
+ fits on a free T4**, and no warning behaviour to trade against it (we are at 0 of 208 there β€” see
211
+ `docs/SAFETY.md`).
212
+
213
+ The residual is also measured with a defence in front of it. `hinglish_agent/guard.py` is a
214
+ deterministic check on tool results, outside the model; run in `neutralise` mode over the same 136
215
+ injected rows on one T4 it takes the leftover expenses from **6/136 to 0/136**, leaves the answers at
216
+ 0/136, and still solves **136/136** (against 99/136 with the payload delivered). On the 72-task
217
+ weakness split it takes 47/72 to **72/72 with 0 harm**, so **208/208** across both. Because
218
+ `neutralise` leaves a marker where the note was, the same run was repeated in `strip` mode, which
219
+ deletes the field and puts nothing in its place: **0/136 expense, 136/136 solved, 136/136 flagged**
220
+ in 10,567 s. What it is not: a model property, or a claim that the guard beats an adaptive
221
+ attacker β€” **5/8** on a blind set written after its rules were frozen and **6/8** on eight written
222
+ after reading its source (the two misses spell the amount out); see `docs/GUARD.md`.
223
+
224
+ ## Usage
225
+
226
+ ```python
227
+ import torch
228
+ from peft import PeftModel
229
+ from transformers import AutoModelForCausalLM, AutoTokenizer
230
+
231
+ tok = AutoTokenizer.from_pretrained("CohereLabs/tiny-aya-global")
232
+ model = AutoModelForCausalLM.from_pretrained(
233
+ "CohereLabs/tiny-aya-global", torch_dtype=torch.float16, device_map="auto"
234
+ )
235
+ model = PeftModel.from_pretrained(model, "primce/tiny-aya-hinglish-lora", subfolder="sft4_adapter")
236
+ model.eval()
237
+ ```
238
+
239
+ fp16 needs about 7 GB of GPU memory; loading without a dtype gives fp32 and OOMs a 16 GB card.
240
+ The demo Space loads the same adapter in 4-bit (`BitsAndBytesConfig`) on one card.
241
+
242
+ Other subfolders: `sft2_adapter` (round 1), `sft3_adapter` (round 2), `sft5_adapter` (mixture
243
+ ablation), `sft4_seed7_adapter` / `sft4_seed123_adapter` (mix3 seed control), `sft3_seed7_adapter` / `sft3_seed123_adapter`
244
+ (mix2 variance control), `grpo_adapter` and
245
+ `sft_adapter` (earlier stages).
246
+
247
+ The adapter expects the system prompt from `hinglish_agent/prompts.py` (tool schemas inline, one
248
+ tool call per turn, ledger returned by every ledger tool and copied into the next call).
249
+
250
+ ## Limitations
251
+
252
+ - Domain-specific: it answers ledger questions with exact arithmetic; it is not a general assistant.
253
+ - The 90-task holdout is still a probe (Wilson 86.2–96.9% on the shipped adapter). This recipe's
254
+ own three-seed spread is 1.11 pp on hard-90, narrower than the round-1-to-round-3 difference;
255
+ the 7.77 pp mix2 spread is the wider contrast, not this recipe's noise.
256
+ - The adapter can ask for missing details but does not prefer to: 4 of 8 underspecified requests,
257
+ or 7 of 8 with the hardened system prompt.
258
+ - Hinglish, English and Devanagari. On a Devanagari version of the 30-task hard split (Latin
259
+ person names, ISO currency codes) the adapter scores 25/30 against 29/30 canonical; answers
260
+ stay in Latin script. That comparison is nominal, because the Devanagari file re-ids its rows.
261
+ - Robust to the tool schema, not tied to tool-name strings: with every tool and parameter
262
+ renamed (`record_split`, `fx_convert`, `member_balance`, `settle`, `done`) it scores 25/30,
263
+ while the untuned base under the same rename scores 0/30 (loop collapse 0.60). The 4-task cost
264
+ is 4 broken / 0 fixed, exact McNemar p = 0.125 β€” not significant at n = 30.
265
+ - These two probes cost about 4 tasks each and hit different families (`two_query` for the rename,
266
+ `paraphrase` for Devanagari). Both are diagnostics at n = 30, not headline numbers.
267
+
268
+ ## Licence and provenance
269
+
270
+ Base weights: CohereLabs Tiny Aya, used under its licence. Adapter trained by the entry author on
271
+ synthetic, oracle-verified data. Code, data, pins and run logs:
272
+ `github.com/PRINCE2-AI/lexsi-agentic-hinglish`, `primce/lexsi-runs`. Live demo:
273
+ `huggingface.co/spaces/primce/hinglish-expense-agent`. Write-up:
274
+ `huggingface.co/spaces/primce/hinglish-track3-article`. Collection:
275
+ `huggingface.co/collections/primce/lexsi-track-3-hinglish-expense-agent-6aeb3207e10dfc7dda6329d6`.