devaanand commited on
Commit
bf6c4d6
1 Parent(s): 6e3ac9f

docs: AI item-drafting scenarios in /guide and MANUAL_TESTING.md

Browse files

Click-through test steps for the Draft-with-AI tab (happy path, review
labeling, library guardrail, vague input, provenance in metadata, daily
cap, signed-out/no-key behavior) + the 20/day row in the limits table.

Files changed (2) hide show
  1. MANUAL_TESTING.md +30 -0
  2. backend/app/guide.html +45 -1
MANUAL_TESTING.md CHANGED
@@ -88,6 +88,36 @@ with a "Sign in (top right)" message. Sign in and retry: accepted.
88
  Nothing is saved until you review and press Save. Item files are never
89
  retained on the server.
90
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
91
  ## 4. Language, models, and warnings (Step 3 card + results)
92
 
93
  Run each of these and open the results page; the amber warnings panel should
 
88
  Nothing is saved until you review and press Save. Item files are never
89
  retained on the server.
90
 
91
+ ### Custom construct, drafted with AI (new)
92
+
93
+ Needs a generation key on the instance (`GROQ_API_KEY` or `ANTHROPIC_API_KEY`);
94
+ without one the "Draft with AI" tab is hidden entirely. Signed-in only, 20/day
95
+ (tune locally with `CCR_USER_MAX_GENERATIONS_PER_DAY=2` to test the cap fast).
96
+
97
+ 1. Happy path: sign in > "+ Custom construct" > "Draft with AI" tab. Enter a
98
+ name + a 1-2 sentence description, press "Draft items". Expect ~10
99
+ first-person, positively-worded items in the textarea (no (R) items - by
100
+ design), a "N of 20 used today" counter, the amber "AI-generated 路 not
101
+ validated - drafted by <model>" notice, and (sometimes) model notes.
102
+ 2. Review-edit-save: edit one item, delete one, Save. Picker shows the
103
+ construct under "My custom constructs" with the "AI-generated 路 not
104
+ validated" tag; the selected-construct block shows the caution paragraph.
105
+ The tag persists despite the edits (seed was AI; item hash records edits).
106
+ 3. Library guardrail: name it "Satisfaction with Life" - expect the warning
107
+ pointing at the existing library scale before you generate.
108
+ 4. Vague input: nonsense name + vague description still returns items, with
109
+ model notes explaining the problem.
110
+ 5. Run + provenance: run a corpus against the saved AI construct. Results page
111
+ shows the caution line; metadata JSON has `source_type: "llm_generated"`,
112
+ `generation` (model, prompt_version, generated_at), and `items_source_note`.
113
+ 6. Cap: with `CCR_USER_MAX_GENERATIONS_PER_DAY=2`, the third draft returns a
114
+ friendly 429 ("resets at midnight UTC"); failed drafts do NOT burn quota.
115
+ 7. Signed out: the tab shows a sign-in nudge; `POST
116
+ /api/constructs/generate-items` returns 401.
117
+ 8. No key set: unset both keys - the tab disappears; the endpoint returns 503.
118
+ 9. One-shot live check without the UI:
119
+ `cd backend && GROQ_API_KEY=... .venv/bin/python ../scripts/smoke_test_generation.py`
120
+
121
  ## 4. Language, models, and warnings (Step 3 card + results)
122
 
123
  Run each of these and open the results page; the amber warnings panel should
backend/app/guide.html CHANGED
@@ -181,6 +181,7 @@ what you hit first, not the file size.</p>
181
  <tr><td>Rows per file</td><td>200</td><td>50,000</td><td>50,000</td></tr>
182
  <tr><td>Runs per day</td><td>3, then sign in</td><td>unlimited</td><td>unlimited</td></tr>
183
  <tr><td>Saved runs kept</td><td>none (file deleted after each run)</td><td>15</td><td>unlimited</td></tr>
 
184
  </table>
185
  </div>
186
  <p>Every model below is available on all three, including signed out.</p>
@@ -291,7 +292,7 @@ are deleted after analysis. Upload
291
 
292
  <h3>Custom construct, from file</h3>
293
  <ol>
294
- <li>"+ Custom construct" &gt; "Upload items from CSV/XLSX".</li>
295
  <li>Try <a href="/samples/construct_items_demo.csv" download>construct_items_demo.csv</a>
296
  (<code>item,reverse</code> columns - 1/true/yes/R = reverse),
297
  <a href="/samples/construct_items_marker_demo.csv" download>construct_items_marker_demo.csv</a>
@@ -302,6 +303,49 @@ are deleted after analysis. Upload
302
  until you review and press Save. Item files are never retained on the server.</li>
303
  </ol>
304
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
305
  <h2 id="warnings">4. Language, models, and warnings (Step 3 card + results)</h2>
306
  <p>Run each of these and open the results page; the amber warnings panel should show exactly:</p>
307
  <div class="tablewrap">
 
181
  <tr><td>Rows per file</td><td>200</td><td>50,000</td><td>50,000</td></tr>
182
  <tr><td>Runs per day</td><td>3, then sign in</td><td>unlimited</td><td>unlimited</td></tr>
183
  <tr><td>Saved runs kept</td><td>none (file deleted after each run)</td><td>15</td><td>unlimited</td></tr>
184
+ <tr><td>AI item drafts per day</td><td>0 (sign in to use)</td><td>20</td><td>20</td></tr>
185
  </table>
186
  </div>
187
  <p>Every model below is available on all three, including signed out.</p>
 
292
 
293
  <h3>Custom construct, from file</h3>
294
  <ol>
295
+ <li>"+ Custom construct" &gt; the "Upload CSV/XLSX" tab.</li>
296
  <li>Try <a href="/samples/construct_items_demo.csv" download>construct_items_demo.csv</a>
297
  (<code>item,reverse</code> columns - 1/true/yes/R = reverse),
298
  <a href="/samples/construct_items_marker_demo.csv" download>construct_items_marker_demo.csv</a>
 
303
  until you review and press Save. Item files are never retained on the server.</li>
304
  </ol>
305
 
306
+ <h3 id="ai-draft">Custom construct, drafted with AI (new)</h3>
307
+ <p>For constructs with no validated questionnaire: the platform can draft candidate
308
+ items from the construct's name and a short explanation. The draft is a starting
309
+ point, <b>not</b> a validated scale - you review, edit, and delete before saving, and
310
+ everything the construct touches is labeled "AI-generated 路 not validated".
311
+ Signed-in users only, 20 drafts/day.</p>
312
+ <ol>
313
+ <li><b>Happy path:</b> sign in &gt; "+ Custom construct" &gt; the "Draft with AI"
314
+ tab. Name: <code>Digital overwhelm</code>. Description: <i>"Feeling that
315
+ screens, notifications, and online demands exceed one's capacity to keep
316
+ up."</i> Press "Draft items". Expect in a few seconds: ~10 first-person,
317
+ positively-worded items in the textarea (no <code>(R)</code> items - by
318
+ design), a "0 of 20 used today" style counter, an amber "AI-generated 路 not
319
+ validated - drafted by &lt;model&gt;" notice, and sometimes short model notes
320
+ (e.g. which facets it covered).</li>
321
+ <li><b>Review is the point:</b> edit one item, delete a weak one, then Save.
322
+ The construct appears in the picker under "My custom constructs" with an
323
+ <b>AI-generated 路 not validated</b> tag - the tag stays even though you
324
+ edited, because the seed was AI (the item hash records your edits).</li>
325
+ <li><b>Library guardrail:</b> on the same tab, type <code>Satisfaction with
326
+ Life</code> as the name. Expect a warning that the library already has this
327
+ scale with validated items - use that instead of generating.</li>
328
+ <li><b>It follows your definition:</b> draft the same name twice with two
329
+ different descriptions (e.g. define "resilience" once as bouncing back from
330
+ setbacks, once as tolerating discomfort). The items should track YOUR
331
+ wording, not a generic textbook version - that is the feature working.</li>
332
+ <li><b>Vague input:</b> give a nonsense name (<code>Zorblex</code>) with a vague
333
+ description. Expect items anyway, plus model notes explaining it could not
334
+ identify a standard construct - refine the description and redraft.</li>
335
+ <li><b>Run + provenance:</b> run any corpus against your saved AI construct.
336
+ The results page shows a caution line; the metadata JSON download has
337
+ <code>source_type: "llm_generated"</code>, the drafting model + prompt
338
+ version + date, and a machine-readable cautionary note. This travels into
339
+ the reproduction script too.</li>
340
+ <li><b>Signed out:</b> the "Draft with AI" tab shows a sign-in nudge instead of
341
+ controls; the API refuses anonymous calls outright.</li>
342
+ </ol>
343
+ <div class="note">These items are drafted by an AI language model and have not been
344
+ psychometrically validated. Where a validated scale exists, prefer it; interpret
345
+ scores from AI-drafted constructs with appropriate caution. (Validation study -
346
+ AI-drafted vs. validated SWLS/MFQ items on the same texts - is planned before
347
+ public launch.)</div>
348
+
349
  <h2 id="warnings">4. Language, models, and warnings (Step 3 card + results)</h2>
350
  <p>Run each of these and open the results page; the amber warnings panel should show exactly:</p>
351
  <div class="tablewrap">