shadymar19 commited on
Commit
ce34133
·
verified ·
1 Parent(s): babb7a8

add prompt-format robustness

Browse files
Files changed (1) hide show
  1. README.md +13 -0
README.md CHANGED
@@ -36,6 +36,19 @@ That check ships with it: `sadiq_fiqh.verify()` matches every «quote» in the o
36
 
37
  Two things are worth reading carefully. **Refusal discrimination is exact**: on a hundred questions given deliberately irrelevant context it refused every time, and on 282 answerable ones it refused once. Groundedness, on the other hand, is high but not perfect — roughly one quote in nine is reconstructed from what the model learned rather than copied from what it was given. That is the reason the verifier is part of the release rather than an afterthought: run it, and the residual becomes a rejected quote instead of a false citation.
38
 
 
 
 
 
 
 
 
 
 
 
 
 
 
39
  ## Training
40
 
41
  3,949 question–passage–answer examples in which the answer quotes and cites only, plus 500 refusal examples. LoRA (r=32) on all attention and MLP projections, two epochs, merged into the base weights.
 
36
 
37
  Two things are worth reading carefully. **Refusal discrimination is exact**: on a hundred questions given deliberately irrelevant context it refused every time, and on 282 answerable ones it refused once. Groundedness, on the other hand, is high but not perfect — roughly one quote in nine is reconstructed from what the model learned rather than copied from what it was given. That is the reason the verifier is part of the release rather than an afterthought: run it, and the residual becomes a rejected quote instead of a false citation.
38
 
39
+ ### Robustness to prompt format
40
+
41
+ Different RAG tools inject context differently, so the same 60 questions were re-run in four injection styles:
42
+
43
+ | Injection style | Answers with quotes | Quote verbatim rate | Citation matches a passage |
44
+ |---|---|---|---|
45
+ | Native (the shape below) | 100% | 88.8% | 67.8% |
46
+ | AnythingLLM-style context block | 98.3% | 85.5% | **79.7%** |
47
+ | XML `<doc source= page=>` tags | 100% | 87.7% | 66.7% |
48
+ | Bare passages, **no book or page given** | 100% | 83.0% | **0%** |
49
+
50
+ Quoting holds up across all four — the model does not need our exact template to find and copy the right text. Citation is a different matter: **if you do not pass the book and page with each passage, the model will produce a citation anyway, and it will be fabricated.** Always include source metadata in the passage header, or strip the citations from the output.
51
+
52
  ## Training
53
 
54
  3,949 question–passage–answer examples in which the answer quotes and cites only, plus 500 refusal examples. LoRA (r=32) on all attention and MLP projections, two epochs, merged into the base weights.