Spaces:
Running
Running
docs: AI item-drafting scenarios in /guide and MANUAL_TESTING.md
Browse filesClick-through test steps for the Draft-with-AI tab (happy path, review
labeling, library guardrail, vague input, provenance in metadata, daily
cap, signed-out/no-key behavior) + the 20/day row in the limits table.
- MANUAL_TESTING.md +30 -0
- backend/app/guide.html +45 -1
MANUAL_TESTING.md
CHANGED
|
@@ -88,6 +88,36 @@ with a "Sign in (top right)" message. Sign in and retry: accepted.
|
|
| 88 |
Nothing is saved until you review and press Save. Item files are never
|
| 89 |
retained on the server.
|
| 90 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 91 |
## 4. Language, models, and warnings (Step 3 card + results)
|
| 92 |
|
| 93 |
Run each of these and open the results page; the amber warnings panel should
|
|
|
|
| 88 |
Nothing is saved until you review and press Save. Item files are never
|
| 89 |
retained on the server.
|
| 90 |
|
| 91 |
+
### Custom construct, drafted with AI (new)
|
| 92 |
+
|
| 93 |
+
Needs a generation key on the instance (`GROQ_API_KEY` or `ANTHROPIC_API_KEY`);
|
| 94 |
+
without one the "Draft with AI" tab is hidden entirely. Signed-in only, 20/day
|
| 95 |
+
(tune locally with `CCR_USER_MAX_GENERATIONS_PER_DAY=2` to test the cap fast).
|
| 96 |
+
|
| 97 |
+
1. Happy path: sign in > "+ Custom construct" > "Draft with AI" tab. Enter a
|
| 98 |
+
name + a 1-2 sentence description, press "Draft items". Expect ~10
|
| 99 |
+
first-person, positively-worded items in the textarea (no (R) items - by
|
| 100 |
+
design), a "N of 20 used today" counter, the amber "AI-generated 路 not
|
| 101 |
+
validated - drafted by <model>" notice, and (sometimes) model notes.
|
| 102 |
+
2. Review-edit-save: edit one item, delete one, Save. Picker shows the
|
| 103 |
+
construct under "My custom constructs" with the "AI-generated 路 not
|
| 104 |
+
validated" tag; the selected-construct block shows the caution paragraph.
|
| 105 |
+
The tag persists despite the edits (seed was AI; item hash records edits).
|
| 106 |
+
3. Library guardrail: name it "Satisfaction with Life" - expect the warning
|
| 107 |
+
pointing at the existing library scale before you generate.
|
| 108 |
+
4. Vague input: nonsense name + vague description still returns items, with
|
| 109 |
+
model notes explaining the problem.
|
| 110 |
+
5. Run + provenance: run a corpus against the saved AI construct. Results page
|
| 111 |
+
shows the caution line; metadata JSON has `source_type: "llm_generated"`,
|
| 112 |
+
`generation` (model, prompt_version, generated_at), and `items_source_note`.
|
| 113 |
+
6. Cap: with `CCR_USER_MAX_GENERATIONS_PER_DAY=2`, the third draft returns a
|
| 114 |
+
friendly 429 ("resets at midnight UTC"); failed drafts do NOT burn quota.
|
| 115 |
+
7. Signed out: the tab shows a sign-in nudge; `POST
|
| 116 |
+
/api/constructs/generate-items` returns 401.
|
| 117 |
+
8. No key set: unset both keys - the tab disappears; the endpoint returns 503.
|
| 118 |
+
9. One-shot live check without the UI:
|
| 119 |
+
`cd backend && GROQ_API_KEY=... .venv/bin/python ../scripts/smoke_test_generation.py`
|
| 120 |
+
|
| 121 |
## 4. Language, models, and warnings (Step 3 card + results)
|
| 122 |
|
| 123 |
Run each of these and open the results page; the amber warnings panel should
|
backend/app/guide.html
CHANGED
|
@@ -181,6 +181,7 @@ what you hit first, not the file size.</p>
|
|
| 181 |
<tr><td>Rows per file</td><td>200</td><td>50,000</td><td>50,000</td></tr>
|
| 182 |
<tr><td>Runs per day</td><td>3, then sign in</td><td>unlimited</td><td>unlimited</td></tr>
|
| 183 |
<tr><td>Saved runs kept</td><td>none (file deleted after each run)</td><td>15</td><td>unlimited</td></tr>
|
|
|
|
| 184 |
</table>
|
| 185 |
</div>
|
| 186 |
<p>Every model below is available on all three, including signed out.</p>
|
|
@@ -291,7 +292,7 @@ are deleted after analysis. Upload
|
|
| 291 |
|
| 292 |
<h3>Custom construct, from file</h3>
|
| 293 |
<ol>
|
| 294 |
-
<li>"+ Custom construct" > "Upload
|
| 295 |
<li>Try <a href="/samples/construct_items_demo.csv" download>construct_items_demo.csv</a>
|
| 296 |
(<code>item,reverse</code> columns - 1/true/yes/R = reverse),
|
| 297 |
<a href="/samples/construct_items_marker_demo.csv" download>construct_items_marker_demo.csv</a>
|
|
@@ -302,6 +303,49 @@ are deleted after analysis. Upload
|
|
| 302 |
until you review and press Save. Item files are never retained on the server.</li>
|
| 303 |
</ol>
|
| 304 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 305 |
<h2 id="warnings">4. Language, models, and warnings (Step 3 card + results)</h2>
|
| 306 |
<p>Run each of these and open the results page; the amber warnings panel should show exactly:</p>
|
| 307 |
<div class="tablewrap">
|
|
|
|
| 181 |
<tr><td>Rows per file</td><td>200</td><td>50,000</td><td>50,000</td></tr>
|
| 182 |
<tr><td>Runs per day</td><td>3, then sign in</td><td>unlimited</td><td>unlimited</td></tr>
|
| 183 |
<tr><td>Saved runs kept</td><td>none (file deleted after each run)</td><td>15</td><td>unlimited</td></tr>
|
| 184 |
+
<tr><td>AI item drafts per day</td><td>0 (sign in to use)</td><td>20</td><td>20</td></tr>
|
| 185 |
</table>
|
| 186 |
</div>
|
| 187 |
<p>Every model below is available on all three, including signed out.</p>
|
|
|
|
| 292 |
|
| 293 |
<h3>Custom construct, from file</h3>
|
| 294 |
<ol>
|
| 295 |
+
<li>"+ Custom construct" > the "Upload CSV/XLSX" tab.</li>
|
| 296 |
<li>Try <a href="/samples/construct_items_demo.csv" download>construct_items_demo.csv</a>
|
| 297 |
(<code>item,reverse</code> columns - 1/true/yes/R = reverse),
|
| 298 |
<a href="/samples/construct_items_marker_demo.csv" download>construct_items_marker_demo.csv</a>
|
|
|
|
| 303 |
until you review and press Save. Item files are never retained on the server.</li>
|
| 304 |
</ol>
|
| 305 |
|
| 306 |
+
<h3 id="ai-draft">Custom construct, drafted with AI (new)</h3>
|
| 307 |
+
<p>For constructs with no validated questionnaire: the platform can draft candidate
|
| 308 |
+
items from the construct's name and a short explanation. The draft is a starting
|
| 309 |
+
point, <b>not</b> a validated scale - you review, edit, and delete before saving, and
|
| 310 |
+
everything the construct touches is labeled "AI-generated 路 not validated".
|
| 311 |
+
Signed-in users only, 20 drafts/day.</p>
|
| 312 |
+
<ol>
|
| 313 |
+
<li><b>Happy path:</b> sign in > "+ Custom construct" > the "Draft with AI"
|
| 314 |
+
tab. Name: <code>Digital overwhelm</code>. Description: <i>"Feeling that
|
| 315 |
+
screens, notifications, and online demands exceed one's capacity to keep
|
| 316 |
+
up."</i> Press "Draft items". Expect in a few seconds: ~10 first-person,
|
| 317 |
+
positively-worded items in the textarea (no <code>(R)</code> items - by
|
| 318 |
+
design), a "0 of 20 used today" style counter, an amber "AI-generated 路 not
|
| 319 |
+
validated - drafted by <model>" notice, and sometimes short model notes
|
| 320 |
+
(e.g. which facets it covered).</li>
|
| 321 |
+
<li><b>Review is the point:</b> edit one item, delete a weak one, then Save.
|
| 322 |
+
The construct appears in the picker under "My custom constructs" with an
|
| 323 |
+
<b>AI-generated 路 not validated</b> tag - the tag stays even though you
|
| 324 |
+
edited, because the seed was AI (the item hash records your edits).</li>
|
| 325 |
+
<li><b>Library guardrail:</b> on the same tab, type <code>Satisfaction with
|
| 326 |
+
Life</code> as the name. Expect a warning that the library already has this
|
| 327 |
+
scale with validated items - use that instead of generating.</li>
|
| 328 |
+
<li><b>It follows your definition:</b> draft the same name twice with two
|
| 329 |
+
different descriptions (e.g. define "resilience" once as bouncing back from
|
| 330 |
+
setbacks, once as tolerating discomfort). The items should track YOUR
|
| 331 |
+
wording, not a generic textbook version - that is the feature working.</li>
|
| 332 |
+
<li><b>Vague input:</b> give a nonsense name (<code>Zorblex</code>) with a vague
|
| 333 |
+
description. Expect items anyway, plus model notes explaining it could not
|
| 334 |
+
identify a standard construct - refine the description and redraft.</li>
|
| 335 |
+
<li><b>Run + provenance:</b> run any corpus against your saved AI construct.
|
| 336 |
+
The results page shows a caution line; the metadata JSON download has
|
| 337 |
+
<code>source_type: "llm_generated"</code>, the drafting model + prompt
|
| 338 |
+
version + date, and a machine-readable cautionary note. This travels into
|
| 339 |
+
the reproduction script too.</li>
|
| 340 |
+
<li><b>Signed out:</b> the "Draft with AI" tab shows a sign-in nudge instead of
|
| 341 |
+
controls; the API refuses anonymous calls outright.</li>
|
| 342 |
+
</ol>
|
| 343 |
+
<div class="note">These items are drafted by an AI language model and have not been
|
| 344 |
+
psychometrically validated. Where a validated scale exists, prefer it; interpret
|
| 345 |
+
scores from AI-drafted constructs with appropriate caution. (Validation study -
|
| 346 |
+
AI-drafted vs. validated SWLS/MFQ items on the same texts - is planned before
|
| 347 |
+
public launch.)</div>
|
| 348 |
+
|
| 349 |
<h2 id="warnings">4. Language, models, and warnings (Step 3 card + results)</h2>
|
| 350 |
<p>Run each of these and open the results page; the amber warnings panel should show exactly:</p>
|
| 351 |
<div class="tablewrap">
|