ROADMAP.md

Roadmap

Backlog (pick one at a time)

Working list as of 2026-07-31 - features, bugs, and chores in rough priority order. Detailed specs for the numbered features live in the sections below.

Features

• PsyEmbedding models in the picker (PI short-term ask): all four models in the hub collection are in the registry and live on dev - verified against huggingface.co/collections/Culture-and-Morality-Lab/psyembedding 2026-07-31.
• LLM-generated construct items (PI idea 2026-07-31): BUILT 2026-08-05 (see ITEM_GENERATION.md). Design note + PI-approved prompt and cautionary wording; POST /api/constructs/generate-items (signed-in, 20/day cap, Claude Haiku, preview-only into the existing review-and-save flow); provenance on construct + run metadata; badges in picker/results. Remaining before Friday launch: ANTHROPIC_API_KEY on the Space (billing), live smoke test, then the SWLS/MFQ validation run (correlate per-text CCR scores from generated vs. validated items).
• Anchor vectors / bipolar constructs (feature 2 below): SHIPPED 2026-08-22 (spec 0006). AV = C - C_opposite centroids with a cosine default and a dot-product toggle, both item sets entered or uploaded, bipolar results and export at output_schema_version 1.2. Live on dev; NOT yet on the lab production Space.
• Automatic chunking for over-limit rows (feature 1 below). PI re-sent the spec 2026-07-31; matches feature 1 (optional, warn about truncation without it, chunk-embedding vs chunk-similarity averaging still an open question for him).
• Dark mode for the app - /guide and /product already have it; the React app doesn't.
• Decide the default model with the PI: MiniLM (CCR reference) vs. a PsyEmbedding model ("prominent or default" was the PI's ask).

Bugs / fixes

• PsyEmbedding HF repos: DONE 2026-08-22. The missing 1_Pooling config was pushed to all four repos with lab credentials; models.yaml re-pins each to the fixed revision (one commit above the old pin, identical weights) and the pooling_fallback workaround is removed.
• PsyEmbedding benchmarks: DONE 2026-08-26. All SEVEN models were unbenchmarked, not just the four. Each now records short/medium/long seconds per 1k texts, measured on an Apple M4 Pro at 2 threads and labelled as such. Still worth a re-run on the Space before the numbers are shown to users: a shared vCPU is roughly 2-4x slower.

Updates / chores

• Open-source warning text from the PI: received and placed verbatim on the landing page (2026-07-31), along with his hero copy and "Who runs this" text.
• LICENSE: DONE. MIT, single-licensed (DECISIONS 2026-08-26); the two untracked drafts (a duplicate MIT and an Apache-2.0 alternative) are removed and README now states the license and separates it from the questionnaire items, which belong to their original authors.
• Construct library verification pass: DONE 2026-08-27 (spec 0007). Noor's review of all 525 items applied, and her follow-up answers resolved every open question except the two PI decisions; 88 of 94 constructs verified.
• Construct library: 6 constructs still unverified, all waiting on the PI: the IPIP "I" prefix (50 items, 5 constructs) and restoring the K10 stem (10 items). Everything else from the review is resolved.
• Durable source links: the Grit-S entries point at personal Dropbox URLs, dirty_dozen_* traded a working ResearchGate link for a paywalled PsycNET one (the reviewer's link carried a session token that could not be committed), cage_questionnaire uses a token-style USPSTF file URL, and mfq_care/mfq_fairness point at a third-party mirror of the cited paper. All four want a stable replacement.
• Verify Dr. Chen's maintainer pre-assignment exists on /admin and that she can sign in.
• Self-service password reset (currently admin-only; tied to the planned Supabase auth swap).
• Invite links: currently ON HOLD behind CCR_INVITES_ENABLED - decide keep/kill permanently.
• Celery + Redis job queue - only when multi-instance deployment or retry semantics are needed (see jobs.py docstring).

PI-requested features (Mohammad, 2026-07-18), grounded in Teitelbaum & Simchon (2025), Neural Text Embeddings in Psychological Research, Psychological Methods, https://doi.org/10.1037/met0000768.

1. Automatic text chunking for over-limit rows

Problem. Every model has a token window (MiniLM 256, the others 512 tokens = roughly 350-400 English words). Rows beyond it are silently truncated today; we only warn (TEXTS_MAYBE_TRUNCATED). In a typical upload most rows fit and a handful do not (e.g. 190 of 200 under the limit, 10 over).

Feature. An optional per-run "Split long texts into chunks" toggle (default OFF - never changes existing behavior silently).

• Detection: count tokens with the SELECTED model's own tokenizer (exact, not a word-count estimate). The Step 3 card shows the toggle only when the corpus has over-limit rows: "N rows exceed this model's 512-token window."
• Off (default): current behavior, plus the existing truncation warning, with hint text: "text beyond the model's window is ignored."
• On: each over-limit row is split into sequential chunks of at most max_seq_length tokens (example: 1,200 tokens -> 512 + 512 + 176). Rows within the limit are untouched.
• Row-level result from chunk results - two candidate aggregations (PI listed both; decide before implementation): a) average the chunk EMBEDDINGS (optionally length-weighted), then score the averaged embedding once - keeps one scoring path; b) score each chunk, then average the chunk SIMILARITIES per row.
• Warnings: chunked runs report TEXTS_CHUNKED (count + affected rows) instead of TEXTS_MAYBE_TRUNCATED for those rows.
• Reproducibility: chunking config (on/off, chunk size, aggregation) goes into run metadata AND the generated reproduction script, which must implement the identical split so exported scores reproduce offline.

Open questions for the PI

• Aggregation default: mean of embeddings vs mean of similarities?
• Length-weight the chunk average (176-token tail counts less) or plain mean?
• Any chunk overlap (e.g. 50 tokens) to avoid cutting sentences, or none?

2. Anchor vectors (bipolar constructs)

Problem. Plain CCR scores similarity to a single construct C. Constructs with a natural opposite (happiness vs sadness, internal vs external locus of control) are better measured along the direction BETWEEN the poles - this also cancels shared confounds like "questionnaire-ness" (both poles are worded as questionnaire items, so their difference subtracts that style component; see the paper's Appendix B).

Feature. Optional second item set on a run:

• C = centroid of the target construct's item embeddings
• C_opp = centroid of the opposing construct's item embeddings
• AV = C - C_opp
• loading = cos(T, AV) for each text embedding T

Higher = toward the target pole, negative = toward the opposing pole.

• UX: in the construct picker, an "Add contrasting construct (anchor vector)" option opens the SAME selection flows for the opposite pole (library / typed / file upload). Both item sets show side by side before running.
• Data model: Job gains an optional opposite_construct_id. Metadata records BOTH construct snapshots + item hashes and a scoring block {"method": "anchored_vector", "similarity": "cosine"}.
• Results page: the score is now bipolar - histogram centered on 0, negative scores meaningful (toward the opposite pole), top/bottom texts labeled "most " / "most ". Per-item loadings shown per pole.
• Reproduction script: embeds both item sets verbatim and reproduces AV math.
• Reverse-scored items: unchanged in v1 (the paper's footnote 27 suggests negating reverse items; our (R) flags already carry the information).

Open questions for the PI

• Similarity metric: cosine(T, AV) per the PI's formula; the paper found dot(T, AV) sometimes better (Appendix B). Config flag or fixed cosine?
• Should anchored runs also report the plain per-pole similarities (cos(T, C), cos(T, C_opp)) in the export for transparency?

3. Multi-construct runs - IMPLEMENTED

Problem. Each construct had to be run separately, which is slow when the goal is to see how two or more constructs are interrelated in the same texts.

Implemented (2026-07-23). A run now accepts up to 10 constructs (POST /api/jobs takes construct_ids; construct_id still works). The corpus is embedded ONCE and every construct is scored against the same document embeddings, so N constructs cost barely more than one - and the per-text scores are row-aligned by construction.

• Results add a "Construct interrelations" card: Pearson r between per-text CCR scores, plus a collapsible per-construct section (histogram, item loadings, top/bottom texts).
• Export: per-construct prefixed columns ({slug}_sim_item_N, {slug}_ccr_score) under output_schema_version 1.1; single-construct exports unchanged at 1.0.
• Metadata records every construct snapshot + item hash and the correlation matrix; the reproduction script embeds the corpus once, scores all constructs, and prints the same correlations.
• A multi-construct run counts once toward the anonymous/saved-run limits; the constructs-per-run cap (10) bounds export width, not compute.

Sequencing

Anchor vectors first (pure scoring change, no ingestion changes), then chunking (touches ingestion, warnings, cache keys - chunked and unchunked embeddings must not share a cache entry). Multi-construct runs (3) shipped first: no scoring or ingestion changes, and the embedding-reuse plumbing it added (one corpus pass, N scorings) is what chunking's cache work builds on.