Spaces:
Sleeping
Sleeping
feat: upload tiers with placeholder sign-in, project archive/delete, no-em-dash rule
Browse files- Anonymous caps (2 MB / 500 rows, env-tunable) with sign-in hints; demo
name-only session (HMAC cookie) lifts caps; get_current_user() is the
Phase 2 swap point for managed auth
- Archive/unarchive projects (reversible, Archived sidebar group)
- Permanent delete with typed-name confirmation; cascades datasets, runs,
uploaded + result files; logged without retaining text (design par. 9)
- Fix: job executor recreates after lifespan shutdown (dev reload broke jobs)
- Style: em dashes removed across 50 files; rule in CLAUDE.md + edit hook
- Tests: 39 total (tiers, tampered cookie, archive toggle, delete cascades)
This view is limited to 50 files because it contains too many changes. See raw diff
- .claude/agents/docs-sync-reviewer.md +7 -7
- .claude/agents/research-validity-reviewer.md +12 -12
- .claude/agents/test-parity-reviewer.md +2 -2
- .claude/commands/implement-spec.md +1 -1
- .claude/commands/review-scoring-contract.md +1 -1
- .claude/hooks/protect-golden.sh +1 -1
- .claude/hooks/py-syntax-check.sh +10 -0
- .claude/rules/frontend.md +3 -3
- .claude/rules/python-backend.md +1 -1
- .claude/rules/registries.md +2 -2
- .env.example +1 -1
- .gitignore +1 -1
- CHANGELOG.md +20 -5
- CLAUDE.md +8 -7
- DECISIONS.md +23 -7
- DEPLOY.md +8 -8
- Dockerfile +3 -3
- README.md +4 -4
- ROADMAP.md +6 -6
- WORKLOG.md +1 -1
- backend/app/auth_demo.py +64 -0
- backend/app/ccr.py +4 -4
- backend/app/construct_lib.py +3 -3
- backend/app/db.py +1 -1
- backend/app/ingest.py +4 -4
- backend/app/jobs.py +23 -6
- backend/app/main.py +124 -5
- backend/app/models.py +2 -1
- backend/app/registry.py +2 -2
- backend/app/reproducibility.py +3 -3
- backend/app/schemas.py +9 -0
- backend/app/warnings_engine.py +3 -3
- backend/static/assets/index-CHG17tHs.js +0 -0
- backend/static/assets/{index-BDrO2FRr.css → index-Cu39pU_-.css} +1 -1
- backend/static/assets/index-DOOkGzjl.js +0 -0
- backend/static/index.html +2 -2
- backend/tests/test_api.py +3 -3
- backend/tests/test_auth_tiers_and_lifecycle.py +134 -0
- backend/tests/test_ccr.py +1 -1
- docs/specs/0001-data-quality-warnings.md +7 -7
- docs/specs/0002-python-script-export.md +1 -1
- docs/specs/0003-model-registry-and-language-selection.md +2 -2
- docs/specs/0004-construct-library-yaml.md +2 -2
- docs/specs/0005-engine-extraction-and-parity.md +3 -3
- docs/specs/TEMPLATE.md +1 -1
- evals/README.md +4 -4
- frontend/index.html +1 -1
- frontend/src/App.jsx +101 -4
- frontend/src/ResultsView.jsx +4 -4
- frontend/src/Workspace.jsx +95 -9
.claude/agents/docs-sync-reviewer.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
| 1 |
---
|
| 2 |
name: docs-sync-reviewer
|
| 3 |
-
description: Keeps documentation truthful after code changes
|
| 4 |
tools: Read, Grep, Glob
|
| 5 |
model: haiku
|
| 6 |
---
|
|
@@ -9,13 +9,13 @@ You are the documentation-sync reviewer for the CCR Platform. Docs that lie are
|
|
| 9 |
no docs; a future lab member must be able to trust every README sentence.
|
| 10 |
|
| 11 |
After a change, check:
|
| 12 |
-
1. **README.md**
|
| 13 |
-
2. **CHANGELOG.md**
|
| 14 |
-
3. **DECISIONS.md**
|
| 15 |
entry (date, decision, why, rejected alternative)? CLAUDE.md hard rule.
|
| 16 |
-
4. **docs/specs/**
|
| 17 |
(draft → implemented) and note deviations in the spec file itself.
|
| 18 |
-
5. **CLAUDE.md**
|
| 19 |
|
| 20 |
Return a short list of exact edits needed (file + suggested text), or "docs are in sync."
|
| 21 |
-
Do not rewrite documents wholesale
|
|
|
|
| 1 |
---
|
| 2 |
name: docs-sync-reviewer
|
| 3 |
+
description: Keeps documentation truthful after code changes - README, CHANGELOG, DECISIONS, spec status, and design-doc references. Use after completing a feature or making any architecture-relevant choice.
|
| 4 |
tools: Read, Grep, Glob
|
| 5 |
model: haiku
|
| 6 |
---
|
|
|
|
| 9 |
no docs; a future lab member must be able to trust every README sentence.
|
| 10 |
|
| 11 |
After a change, check:
|
| 12 |
+
1. **README.md** - do quickstart commands still work as written? Are new features/flags mentioned?
|
| 13 |
+
2. **CHANGELOG.md** - is there an entry for user-visible changes (Keep-a-Changelog style)?
|
| 14 |
+
3. **DECISIONS.md** - was any architecture/infrastructure/vendor/model choice made without an
|
| 15 |
entry (date, decision, why, rejected alternative)? CLAUDE.md hard rule.
|
| 16 |
+
4. **docs/specs/** - does the implemented behavior match the spec? Mark spec status
|
| 17 |
(draft → implemented) and note deviations in the spec file itself.
|
| 18 |
+
5. **CLAUDE.md** - are commands, layout, and phase state still accurate?
|
| 19 |
|
| 20 |
Return a short list of exact edits needed (file + suggested text), or "docs are in sync."
|
| 21 |
+
Do not rewrite documents wholesale - smallest truthful diff.
|
.claude/agents/research-validity-reviewer.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
| 1 |
---
|
| 2 |
name: research-validity-reviewer
|
| 3 |
-
description: Reviews changes for research-validity risks
|
| 4 |
tools: Read, Grep, Glob, Bash
|
| 5 |
model: inherit
|
| 6 |
---
|
|
@@ -11,28 +11,28 @@ honesty: transparent warnings, reproducible runs, and never overstating what CCR
|
|
| 11 |
|
| 12 |
Review the changes you are pointed at for:
|
| 13 |
|
| 14 |
-
1. **Preprocessing drift**
|
| 15 |
row filtering that isn't reflected in run metadata AND documented. Silent preprocessing
|
| 16 |
changes invalidate cross-run comparisons.
|
| 17 |
-
2. **Metadata completeness**
|
| 18 |
construct snapshot + version + item_hash, language selection + detection summary,
|
| 19 |
preprocessing config, scoring.adjustment_strategy, output_schema_version, warnings.
|
| 20 |
-
3. **Registry bypasses**
|
| 21 |
language support) hardcoded anywhere outside packages/model_registry. E5-family models
|
| 22 |
without "query: " prefix handling is a correctness bug.
|
| 23 |
-
4. **Reverse scoring**
|
| 24 |
recorded adjustment_strategy, or aggregates that silently include/exclude reversed items.
|
| 25 |
V1 policy: raw + flags; exclude_reversed only when explicit; all-reversed must block.
|
| 26 |
-
5. **Language assumptions**
|
| 27 |
removed or softened; supported-language checks bypassed.
|
| 28 |
-
6. **Overconfident claims**
|
| 29 |
or "accuracy"; missing the relatedness-vs-stance caveat where results are presented.
|
| 30 |
-
7. **Reproducibility**
|
| 31 |
unpinned versions, missing prefix logic); golden/expected outputs edited without approval.
|
| 32 |
|
| 33 |
Return exactly three sections:
|
| 34 |
-
1. **Blocking issues**
|
| 35 |
-
2. **Non-blocking concerns**
|
| 36 |
-
3. **Suggested fixes**
|
| 37 |
|
| 38 |
-
Be specific and cite evidence. If you find nothing, say so plainly
|
|
|
|
| 1 |
---
|
| 2 |
name: research-validity-reviewer
|
| 3 |
+
description: Reviews changes for research-validity risks - hidden preprocessing changes, missing model/construct metadata, unsupported language assumptions, reverse-scoring mishandling, overconfident user-facing claims, reproducibility gaps. Use proactively after changes to the CCR engine, scoring, warnings, exports, or the model/construct registries.
|
| 4 |
tools: Read, Grep, Glob, Bash
|
| 5 |
model: inherit
|
| 6 |
---
|
|
|
|
| 11 |
|
| 12 |
Review the changes you are pointed at for:
|
| 13 |
|
| 14 |
+
1. **Preprocessing drift** - any change to text normalization, deduplication, truncation, or
|
| 15 |
row filtering that isn't reflected in run metadata AND documented. Silent preprocessing
|
| 16 |
changes invalidate cross-run comparisons.
|
| 17 |
+
2. **Metadata completeness** - every run must record: model id + pinned revision + usage_config,
|
| 18 |
construct snapshot + version + item_hash, language selection + detection summary,
|
| 19 |
preprocessing config, scoring.adjustment_strategy, output_schema_version, warnings.
|
| 20 |
+
3. **Registry bypasses** - model-specific behavior (prefixes, normalization, max length,
|
| 21 |
language support) hardcoded anywhere outside packages/model_registry. E5-family models
|
| 22 |
without "query: " prefix handling is a correctness bug.
|
| 23 |
+
4. **Reverse scoring** - any code that adjusts/negates reversed-item similarities beyond the
|
| 24 |
recorded adjustment_strategy, or aggregates that silently include/exclude reversed items.
|
| 25 |
V1 policy: raw + flags; exclude_reversed only when explicit; all-reversed must block.
|
| 26 |
+
5. **Language assumptions** - English-only logic applied to multilingual paths; warnings
|
| 27 |
removed or softened; supported-language checks bypassed.
|
| 28 |
+
6. **Overconfident claims** - UI/exports/docs text implying CCR measures stance, diagnoses,
|
| 29 |
or "accuracy"; missing the relatedness-vs-stance caveat where results are presented.
|
| 30 |
+
7. **Reproducibility** - generated scripts that would not run offline (platform credentials,
|
| 31 |
unpinned versions, missing prefix logic); golden/expected outputs edited without approval.
|
| 32 |
|
| 33 |
Return exactly three sections:
|
| 34 |
+
1. **Blocking issues** - must fix before merge (cite file:line).
|
| 35 |
+
2. **Non-blocking concerns** - should fix soon.
|
| 36 |
+
3. **Suggested fixes** - concrete, smallest-diff suggestions.
|
| 37 |
|
| 38 |
+
Be specific and cite evidence. If you find nothing, say so plainly - do not invent findings.
|
.claude/agents/test-parity-reviewer.md
CHANGED
|
@@ -10,7 +10,7 @@ You are the test & parity reviewer for the CCR Platform.
|
|
| 10 |
Ground rules you enforce:
|
| 11 |
- The published `ccr_wrapper` implementation is the correctness spec. Changes to similarity,
|
| 12 |
scoring, or preprocessing must keep (or add) parity coverage within float tolerance (atol),
|
| 13 |
-
with pinned model revisions on both paths
|
| 14 |
- Golden evals exist to catch silent drift: output shape, column names/order, metadata fields,
|
| 15 |
and warning behavior. `evals/expected_outputs/` is human-approved only.
|
| 16 |
- New behavior without tests in the same change is incomplete work, not a follow-up.
|
|
@@ -26,5 +26,5 @@ When reviewing, check:
|
|
| 26 |
4. Run the relevant test commands (cd backend && python -m pytest tests/ -q) and report results.
|
| 27 |
|
| 28 |
Return: (1) coverage gaps with proposed test names, (2) any weakened assertions,
|
| 29 |
-
(3) parity/golden risks, (4) test run results. Write new tests when asked
|
| 30 |
useful tests, placed beside existing suites, using existing fixtures.
|
|
|
|
| 10 |
Ground rules you enforce:
|
| 11 |
- The published `ccr_wrapper` implementation is the correctness spec. Changes to similarity,
|
| 12 |
scoring, or preprocessing must keep (or add) parity coverage within float tolerance (atol),
|
| 13 |
+
with pinned model revisions on both paths - never compare against a moving reference.
|
| 14 |
- Golden evals exist to catch silent drift: output shape, column names/order, metadata fields,
|
| 15 |
and warning behavior. `evals/expected_outputs/` is human-approved only.
|
| 16 |
- New behavior without tests in the same change is incomplete work, not a follow-up.
|
|
|
|
| 26 |
4. Run the relevant test commands (cd backend && python -m pytest tests/ -q) and report results.
|
| 27 |
|
| 28 |
Return: (1) coverage gaps with proposed test names, (2) any weakened assertions,
|
| 29 |
+
(3) parity/golden risks, (4) test run results. Write new tests when asked - smallest
|
| 30 |
useful tests, placed beside existing suites, using existing fixtures.
|
.claude/commands/implement-spec.md
CHANGED
|
@@ -11,5 +11,5 @@ Follow this loop strictly (CLAUDE.md workflow):
|
|
| 11 |
2. Restate the contract in 3 bullets: inputs, outputs/columns, warnings/edge cases. Flag anything the spec leaves undefined BEFORE coding.
|
| 12 |
3. Propose an implementation plan: files to change, tests to add, smallest viable diff. Wait for approval if the plan touches architecture rules.
|
| 13 |
4. Implement in small increments. Add/update tests in the same change.
|
| 14 |
-
5. Run: cd backend && python -m pytest tests/ -q
|
| 15 |
6. Summarize: changed files, test results, deviations from spec (update the spec's status/notes section), CHANGELOG entry, and whether a DECISIONS.md entry is needed.
|
|
|
|
| 11 |
2. Restate the contract in 3 bullets: inputs, outputs/columns, warnings/edge cases. Flag anything the spec leaves undefined BEFORE coding.
|
| 12 |
3. Propose an implementation plan: files to change, tests to add, smallest viable diff. Wait for approval if the plan touches architecture rules.
|
| 13 |
4. Implement in small increments. Add/update tests in the same change.
|
| 14 |
+
5. Run: cd backend && python -m pytest tests/ -q - plus registry validators if registries changed.
|
| 15 |
6. Summarize: changed files, test results, deviations from spec (update the spec's status/notes section), CHANGELOG entry, and whether a DECISIONS.md entry is needed.
|
.claude/commands/review-scoring-contract.md
CHANGED
|
@@ -1,5 +1,5 @@
|
|
| 1 |
---
|
| 2 |
-
description: Audit scoring/output contract
|
| 3 |
---
|
| 4 |
|
| 5 |
Audit the current scoring and output contract end to end:
|
|
|
|
| 1 |
---
|
| 2 |
+
description: Audit scoring/output contract - columns, adjustment_strategy, all-reversed block, metadata completeness
|
| 3 |
---
|
| 4 |
|
| 5 |
Audit the current scoring and output contract end to end:
|
.claude/hooks/protect-golden.sh
CHANGED
|
@@ -8,7 +8,7 @@ file_path="$(printf '%s' "$input" | python3 -c 'import sys,json; d=json.load(sys
|
|
| 8 |
|
| 9 |
case "$file_path" in
|
| 10 |
*"evals/expected_outputs/"*)
|
| 11 |
-
echo "BLOCKED: $file_path is a golden expected output. Golden files change only via explicit human approval
|
| 12 |
exit 2
|
| 13 |
;;
|
| 14 |
*"backend/data/"*)
|
|
|
|
| 8 |
|
| 9 |
case "$file_path" in
|
| 10 |
*"evals/expected_outputs/"*)
|
| 11 |
+
echo "BLOCKED: $file_path is a golden expected output. Golden files change only via explicit human approval - ask Deva to update it manually or approve a regeneration script." >&2
|
| 12 |
exit 2
|
| 13 |
;;
|
| 14 |
*"backend/data/"*)
|
.claude/hooks/py-syntax-check.sh
CHANGED
|
@@ -12,4 +12,14 @@ if [[ "$file_path" == *.py && -f "$file_path" ]]; then
|
|
| 12 |
exit 2
|
| 13 |
fi
|
| 14 |
fi
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 15 |
exit 0
|
|
|
|
| 12 |
exit 2
|
| 13 |
fi
|
| 14 |
fi
|
| 15 |
+
|
| 16 |
+
# Project style rule: no em dashes in any source/text file (CLAUDE.md hard rules).
|
| 17 |
+
case "$file_path" in
|
| 18 |
+
*.py|*.js|*.jsx|*.css|*.md|*.yaml|*.yml|*.sh|*.html)
|
| 19 |
+
if [[ -f "$file_path" ]] && grep -q $'\xe2\x80\x94' "$file_path"; then
|
| 20 |
+
echo "EM DASH found in $file_path. Project rule: no em dashes - use a hyphen or restructure the sentence." >&2
|
| 21 |
+
exit 2
|
| 22 |
+
fi
|
| 23 |
+
;;
|
| 24 |
+
esac
|
| 25 |
exit 0
|
.claude/rules/frontend.md
CHANGED
|
@@ -7,10 +7,10 @@ paths:
|
|
| 7 |
|
| 8 |
# Frontend rules
|
| 9 |
|
| 10 |
-
- Current stack: React 18 + Vite, JavaScript (TypeScript arrives in Phase 2 cleanup
|
| 11 |
-
- All server communication through frontend/src/api.js
|
| 12 |
- The UI never computes analysis values; it renders what the API returns. If a number needs computing, it belongs in the backend.
|
| 13 |
- Warnings from the API are always rendered (amber panel); never drop or reword warning text in the UI.
|
| 14 |
- Accessibility floor (design §19): label every input, keyboard-navigable controls, no color-only signals, charts need an accessible alternative.
|
| 15 |
- Keep the dependency count minimal (currently react + react-dom only); adding a UI library requires a DECISIONS.md entry.
|
| 16 |
-
- After frontend changes that should ship: npm run build (outputs to backend/static/)
|
|
|
|
| 7 |
|
| 8 |
# Frontend rules
|
| 9 |
|
| 10 |
+
- Current stack: React 18 + Vite, JavaScript (TypeScript arrives in Phase 2 cleanup - do not introduce it piecemeal now).
|
| 11 |
+
- All server communication through frontend/src/api.js - no fetch calls inside components.
|
| 12 |
- The UI never computes analysis values; it renders what the API returns. If a number needs computing, it belongs in the backend.
|
| 13 |
- Warnings from the API are always rendered (amber panel); never drop or reword warning text in the UI.
|
| 14 |
- Accessibility floor (design §19): label every input, keyboard-navigable controls, no color-only signals, charts need an accessible alternative.
|
| 15 |
- Keep the dependency count minimal (currently react + react-dom only); adding a UI library requires a DECISIONS.md entry.
|
| 16 |
+
- After frontend changes that should ship: npm run build (outputs to backend/static/) - the API serves the built SPA.
|
.claude/rules/python-backend.md
CHANGED
|
@@ -8,7 +8,7 @@ paths:
|
|
| 8 |
|
| 9 |
- Python 3.10+; type hints on public functions; docstrings explain WHY, not what.
|
| 10 |
- FastAPI: request/response shapes via Pydantic schemas; HTTPException with clear user-facing detail strings; validation errors are 400s, missing resources 404s, state conflicts 409s.
|
| 11 |
-
- All embedding/normalization/similarity math stays in the engine module
|
| 12 |
- Every user-facing failure path returns an actionable message (what happened + what to do), never a stack trace.
|
| 13 |
- Tests: pytest; use the deterministic fake embedder (model "fake-deterministic" / CCR_FAKE_EMBEDDINGS=1); no torch imports in the default suite; poll async jobs with a timeout helper, never sleep-and-hope.
|
| 14 |
- Warnings are structured objects with UPPER_SNAKE codes (design §12), never bare strings.
|
|
|
|
| 8 |
|
| 9 |
- Python 3.10+; type hints on public functions; docstrings explain WHY, not what.
|
| 10 |
- FastAPI: request/response shapes via Pydantic schemas; HTTPException with clear user-facing detail strings; validation errors are 400s, missing resources 404s, state conflicts 409s.
|
| 11 |
+
- All embedding/normalization/similarity math stays in the engine module - API handlers orchestrate, never compute.
|
| 12 |
- Every user-facing failure path returns an actionable message (what happened + what to do), never a stack trace.
|
| 13 |
- Tests: pytest; use the deterministic fake embedder (model "fake-deterministic" / CCR_FAKE_EMBEDDINGS=1); no torch imports in the default suite; poll async jobs with a timeout helper, never sleep-and-hope.
|
| 14 |
- Warnings are structured objects with UPPER_SNAKE codes (design §12), never bare strings.
|
.claude/rules/registries.md
CHANGED
|
@@ -7,9 +7,9 @@ paths:
|
|
| 7 |
# Registry & construct library rules
|
| 8 |
|
| 9 |
- models.yaml is the single source of model truth: id, provider_model_id, pinned revision, supported_languages OR supported_language_set, embedding_dimension, max_seq_length, usage_config (requires_prefix, prefix strings, normalize_embeddings), tiers, operational_config, user-facing warnings.
|
| 10 |
-
- The `pooling` field is DESCRIPTIVE only
|
| 11 |
- E5-family entries must set requires_prefix: true with "query: " on BOTH items and texts.
|
| 12 |
-
- supported_language_set values (e.g., xlm_roberta_100) must resolve to real ISO codes in language_sets.py
|
| 13 |
- Constructs are versioned append-only YAML: editing items/wording/flags creates a NEW version with a new item_hash; never mutate an existing version in place.
|
| 14 |
- Every construct carries: name, version, language, items (text + reverse_scored), citation, verification_status (draft|needs_verification|verified|archived). Item wordings must be flagged until verified verbatim against the original publication.
|
| 15 |
- After any change here, run both validators before considering the change done.
|
|
|
|
| 7 |
# Registry & construct library rules
|
| 8 |
|
| 9 |
- models.yaml is the single source of model truth: id, provider_model_id, pinned revision, supported_languages OR supported_language_set, embedding_dimension, max_seq_length, usage_config (requires_prefix, prefix strings, normalize_embeddings), tiers, operational_config, user-facing warnings.
|
| 10 |
+
- The `pooling` field is DESCRIPTIVE only - models load via their own sentence-transformers config; never reimplement pooling.
|
| 11 |
- E5-family entries must set requires_prefix: true with "query: " on BOTH items and texts.
|
| 12 |
+
- supported_language_set values (e.g., xlm_roberta_100) must resolve to real ISO codes in language_sets.py - a bare "multilingual" label is banned.
|
| 13 |
- Constructs are versioned append-only YAML: editing items/wording/flags creates a NEW version with a new item_hash; never mutate an existing version in place.
|
| 14 |
- Every construct carries: name, version, language, items (text + reverse_scored), citation, verification_status (draft|needs_verification|verified|archived). Item wordings must be flagged until verified verbatim against the original publication.
|
| 15 |
- After any change here, run both validators before considering the change done.
|
.env.example
CHANGED
|
@@ -6,7 +6,7 @@
|
|
| 6 |
# Row ceiling for uploads (default 100000; hosted demo uses 20000)
|
| 7 |
# CCR_MAX_ROWS=20000
|
| 8 |
|
| 9 |
-
# Force the deterministic fake embedder (tests/CI only
|
| 10 |
# CCR_FAKE_EMBEDDINGS=1
|
| 11 |
|
| 12 |
# ---- Phase 2 (not read yet; reserved names) ----
|
|
|
|
| 6 |
# Row ceiling for uploads (default 100000; hosted demo uses 20000)
|
| 7 |
# CCR_MAX_ROWS=20000
|
| 8 |
|
| 9 |
+
# Force the deterministic fake embedder (tests/CI only - never production)
|
| 10 |
# CCR_FAKE_EMBEDDINGS=1
|
| 11 |
|
| 12 |
# ---- Phase 2 (not read yet; reserved names) ----
|
.gitignore
CHANGED
|
@@ -4,7 +4,7 @@ __pycache__/
|
|
| 4 |
.venv/
|
| 5 |
.pytest_cache/
|
| 6 |
|
| 7 |
-
# runtime data (DB, uploaded corpora, results)
|
| 8 |
backend/data/
|
| 9 |
|
| 10 |
# node
|
|
|
|
| 4 |
.venv/
|
| 5 |
.pytest_cache/
|
| 6 |
|
| 7 |
+
# runtime data (DB, uploaded corpora, results) - never commit research data
|
| 8 |
backend/data/
|
| 9 |
|
| 10 |
# node
|
CHANGELOG.md
CHANGED
|
@@ -4,6 +4,21 @@ Format: [Keep a Changelog](https://keepachangelog.com/). User-visible changes on
|
|
| 4 |
|
| 5 |
## [Unreleased]
|
| 6 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 7 |
### Fixed
|
| 8 |
- Project sidebar redesigned for growing lists: always-visible search, recency groups
|
| 9 |
(Today / This week / Earlier), per-project run count + relative last-activity time,
|
|
@@ -19,7 +34,7 @@ Format: [Keep a Changelog](https://keepachangelog.com/). User-visible changes on
|
|
| 19 |
results dashboard with descriptive stats + per-item loadings + top/bottom texts,
|
| 20 |
data-quality warnings, CSV export mirroring ccr_wrapper shape, reproducibility metadata).
|
| 21 |
- Model registry (packages/model_registry) and versioned construct library
|
| 22 |
-
(packages/construct_library) with validators
|
| 23 |
- Claude Code project config: CLAUDE.md, path-scoped rules, 3 review agents, 4 commands,
|
| 24 |
golden-file protection + Python syntax hooks.
|
| 25 |
|
|
@@ -30,8 +45,8 @@ Format: [Keep a Changelog](https://keepachangelog.com/). User-visible changes on
|
|
| 30 |
warnings for language mismatch, uncertain detection, and model/language coverage
|
| 31 |
(multilingual model checked against its real ISO language set). (spec 0001/0003)
|
| 32 |
- Very-short-text warning (< 4 words) with affected-row samples. (spec 0001)
|
| 33 |
-
- Downloadable Python reproduction script per run
|
| 34 |
-
items, model revision, and E5 prefix logic
|
| 35 |
- Model registry drives the model dropdown, validation, prefixes, and metadata:
|
| 36 |
MiniLM (default), E5-Large-v2, Multilingual-E5-base. (spec 0003)
|
| 37 |
- Construct library loaded from versioned YAML files with item hashes, citations, and
|
|
@@ -40,9 +55,9 @@ Format: [Keep a Changelog](https://keepachangelog.com/). User-visible changes on
|
|
| 40 |
block, construct snapshot, model revision, environment pins.
|
| 41 |
|
| 42 |
### Changed
|
| 43 |
-
- All warnings are structured objects (code/severity/message/count/rows)
|
| 44 |
- /api/models response shape changed (registry ids like "all-minilm-l6-v2" replace provider ids).
|
| 45 |
-
- Delimiter detection restricted to real candidates (, ; tab |)
|
| 46 |
sentence CSVs being split on spaces.
|
| 47 |
|
| 48 |
### Removed
|
|
|
|
| 4 |
|
| 5 |
## [Unreleased]
|
| 6 |
|
| 7 |
+
### Added
|
| 8 |
+
- Anonymous vs signed-in upload tiers: anonymous uploads capped at 2 MB / 500 rows with a
|
| 9 |
+
clear sign-in hint; signing in lifts limits. Sign-in is a labeled placeholder (name only)
|
| 10 |
+
until managed auth arrives with lab accounts; tampered sessions are treated as anonymous.
|
| 11 |
+
- Project lifecycle: archive/unarchive (reversible, collapses into an Archived sidebar
|
| 12 |
+
group) and permanent delete with type-the-name confirmation. Delete cascades to datasets,
|
| 13 |
+
runs, uploaded files, and result files, and is logged without retaining any text.
|
| 14 |
+
|
| 15 |
+
### Changed
|
| 16 |
+
- Project-wide style rule: no em dashes in any project text (enforced by an edit hook).
|
| 17 |
+
|
| 18 |
+
### Fixed
|
| 19 |
+
- Job worker survives lifespan restarts (dev reload previously killed job submission
|
| 20 |
+
permanently).
|
| 21 |
+
|
| 22 |
### Fixed
|
| 23 |
- Project sidebar redesigned for growing lists: always-visible search, recency groups
|
| 24 |
(Today / This week / Earlier), per-project run count + relative last-activity time,
|
|
|
|
| 34 |
results dashboard with descriptive stats + per-item loadings + top/bottom texts,
|
| 35 |
data-quality warnings, CSV export mirroring ccr_wrapper shape, reproducibility metadata).
|
| 36 |
- Model registry (packages/model_registry) and versioned construct library
|
| 37 |
+
(packages/construct_library) with validators - not yet wired into the app (specs 0003/0004).
|
| 38 |
- Claude Code project config: CLAUDE.md, path-scoped rules, 3 review agents, 4 commands,
|
| 39 |
golden-file protection + Python syntax hooks.
|
| 40 |
|
|
|
|
| 45 |
warnings for language mismatch, uncertain detection, and model/language coverage
|
| 46 |
(multilingual model checked against its real ISO language set). (spec 0001/0003)
|
| 47 |
- Very-short-text warning (< 4 words) with affected-row samples. (spec 0001)
|
| 48 |
+
- Downloadable Python reproduction script per run - offline-runnable, embeds construct
|
| 49 |
+
items, model revision, and E5 prefix logic - plus pinned requirements file. (spec 0002)
|
| 50 |
- Model registry drives the model dropdown, validation, prefixes, and metadata:
|
| 51 |
MiniLM (default), E5-Large-v2, Multilingual-E5-base. (spec 0003)
|
| 52 |
- Construct library loaded from versioned YAML files with item hashes, citations, and
|
|
|
|
| 55 |
block, construct snapshot, model revision, environment pins.
|
| 56 |
|
| 57 |
### Changed
|
| 58 |
+
- All warnings are structured objects (code/severity/message/count/rows) - UI renders codes.
|
| 59 |
- /api/models response shape changed (registry ids like "all-minilm-l6-v2" replace provider ids).
|
| 60 |
+
- Delimiter detection restricted to real candidates (, ; tab |) - fixes single-column
|
| 61 |
sentence CSVs being split on spaces.
|
| 62 |
|
| 63 |
### Removed
|
CLAUDE.md
CHANGED
|
@@ -10,20 +10,20 @@ Current state: Phase 0 (visible features on the demo codebase). Roadmap: @ROADMA
|
|
| 10 |
|
| 11 |
## Commands
|
| 12 |
|
| 13 |
-
- Backend tests: `cd backend && python -m pytest tests/ -q` (fast
|
| 14 |
- Run app locally: `./run.sh` → http://127.0.0.1:8000
|
| 15 |
- Frontend build: `cd frontend && npm install && npm run build` (outputs to backend/static/)
|
| 16 |
- Frontend dev: `cd frontend && npm run dev` (proxies /api to :8000)
|
| 17 |
- Validate registries: `python packages/model_registry/validate_models.py && python packages/construct_library/validate_constructs.py`
|
| 18 |
- Real-model smoke test: `python scripts/verify_install.py` (downloads MiniLM once)
|
| 19 |
|
| 20 |
-
## Architecture rules (from design §15
|
| 21 |
|
| 22 |
- Analysis/NLP logic lives ONLY in the engine (today `backend/app/ccr.py`; Phase 1 extracts it to `packages/ccr_engine/`). Never in UI components or API route handlers.
|
| 23 |
- ALL model-specific behavior (prefixes, normalization, dims, max length, language support) goes through `packages/model_registry/models.yaml`. No hardcoded model behavior anywhere.
|
| 24 |
-
- Predefined constructs live ONLY in `packages/construct_library/constructs/*.yaml`
|
| 25 |
- Every analysis run must produce metadata + a reproduction script (design §14). Exports mirror `ccr_wrapper` output shape.
|
| 26 |
-
- MiniLM (all-MiniLM-L6-v2) is the default model
|
| 27 |
- The published `ccr_wrapper` implementation is the correctness spec (parity tests, design §16).
|
| 28 |
|
| 29 |
## Hard rules
|
|
@@ -31,10 +31,11 @@ Current state: Phase 0 (visible features on the demo codebase). Roadmap: @ROADMA
|
|
| 31 |
- Never silently change output column names (bump output_schema_version in metadata instead).
|
| 32 |
- Never remove or weaken warnings to make tests pass.
|
| 33 |
- Never log uploaded text content.
|
| 34 |
-
- Never edit files under `evals/expected_outputs/`
|
| 35 |
- No new infrastructure (Redis, GPU, new vendors, new services) without a DECISIONS.md entry.
|
| 36 |
- Reverse scoring: v1 exports raw similarities + flags only; adjustment_strategy is a recorded parameter (design §11). Do not invent adjustments.
|
| 37 |
- Do not restructure directories big-bang; follow the strangler steps (design §7).
|
|
|
|
| 38 |
|
| 39 |
## Workflow
|
| 40 |
|
|
@@ -45,10 +46,10 @@ Current state: Phase 0 (visible features on the demo codebase). Roadmap: @ROADMA
|
|
| 45 |
|
| 46 |
## PI working norms
|
| 47 |
|
| 48 |
-
- Mohammad (2026-07-09): "you can always suggest better options that I might not know about
|
| 49 |
take initiative (newer NLP models, lighter options, less expensive options)." Consequence:
|
| 50 |
when a decision involves a model, service, or cost, ALWAYS check for newer/lighter/cheaper
|
| 51 |
-
candidates and present the better option with evidence
|
| 52 |
New models enter via the registry + benchmark-before-adopt (golden/parity harness), never
|
| 53 |
by swapping defaults silently. Cost options favor free tiers and annual-friendly services.
|
| 54 |
|
|
|
|
| 10 |
|
| 11 |
## Commands
|
| 12 |
|
| 13 |
+
- Backend tests: `cd backend && python -m pytest tests/ -q` (fast - fake embedder, no torch needed)
|
| 14 |
- Run app locally: `./run.sh` → http://127.0.0.1:8000
|
| 15 |
- Frontend build: `cd frontend && npm install && npm run build` (outputs to backend/static/)
|
| 16 |
- Frontend dev: `cd frontend && npm run dev` (proxies /api to :8000)
|
| 17 |
- Validate registries: `python packages/model_registry/validate_models.py && python packages/construct_library/validate_constructs.py`
|
| 18 |
- Real-model smoke test: `python scripts/verify_install.py` (downloads MiniLM once)
|
| 19 |
|
| 20 |
+
## Architecture rules (from design §15 - non-negotiable)
|
| 21 |
|
| 22 |
- Analysis/NLP logic lives ONLY in the engine (today `backend/app/ccr.py`; Phase 1 extracts it to `packages/ccr_engine/`). Never in UI components or API route handlers.
|
| 23 |
- ALL model-specific behavior (prefixes, normalization, dims, max length, language support) goes through `packages/model_registry/models.yaml`. No hardcoded model behavior anywhere.
|
| 24 |
+
- Predefined constructs live ONLY in `packages/construct_library/constructs/*.yaml` - versioned, append-only; edits create new versions. (Migration from `backend/app/seed_constructs.py` is a Phase 0 task.)
|
| 25 |
- Every analysis run must produce metadata + a reproduction script (design §14). Exports mirror `ccr_wrapper` output shape.
|
| 26 |
+
- MiniLM (all-MiniLM-L6-v2) is the default model - it is the CCR reference model. E5-family models REQUIRE the "query: " prefix on both items and texts.
|
| 27 |
- The published `ccr_wrapper` implementation is the correctness spec (parity tests, design §16).
|
| 28 |
|
| 29 |
## Hard rules
|
|
|
|
| 31 |
- Never silently change output column names (bump output_schema_version in metadata instead).
|
| 32 |
- Never remove or weaken warnings to make tests pass.
|
| 33 |
- Never log uploaded text content.
|
| 34 |
+
- Never edit files under `evals/expected_outputs/` - golden outputs change only via explicit human approval (enforced by a PreToolUse hook).
|
| 35 |
- No new infrastructure (Redis, GPU, new vendors, new services) without a DECISIONS.md entry.
|
| 36 |
- Reverse scoring: v1 exports raw similarities + flags only; adjustment_strategy is a recorded parameter (design §11). Do not invent adjustments.
|
| 37 |
- Do not restructure directories big-bang; follow the strangler steps (design §7).
|
| 38 |
+
- No em dashes anywhere in project text: UI strings, docs, comments, commit messages, YAML. Use a hyphen or restructure the sentence (Deva's rule, 2026-07-09; enforced by a PostToolUse hook).
|
| 39 |
|
| 40 |
## Workflow
|
| 41 |
|
|
|
|
| 46 |
|
| 47 |
## PI working norms
|
| 48 |
|
| 49 |
+
- Mohammad (2026-07-09): "you can always suggest better options that I might not know about -
|
| 50 |
take initiative (newer NLP models, lighter options, less expensive options)." Consequence:
|
| 51 |
when a decision involves a model, service, or cost, ALWAYS check for newer/lighter/cheaper
|
| 52 |
+
candidates and present the better option with evidence - recommending beats asking.
|
| 53 |
New models enter via the registry + benchmark-before-adopt (golden/parity harness), never
|
| 54 |
by swapping defaults silently. Cost options favor free tiers and annual-friendly services.
|
| 55 |
|
DECISIONS.md
CHANGED
|
@@ -1,32 +1,48 @@
|
|
| 1 |
# Decisions Log
|
| 2 |
|
| 3 |
> One entry per architecture/infrastructure/vendor/model decision.
|
| 4 |
-
> Format: date
|
| 5 |
> CLAUDE.md hard rule: no new infrastructure without an entry here.
|
| 6 |
|
| 7 |
-
## 2026-07-09
|
| 8 |
Mainline moves to this repo (future home: Culture-and-Morality-Lab org). The public demo repo
|
| 9 |
-
and HF Space remain deployed and untouched
|
| 10 |
demo repo in place (would couple experiments to a live artifact). Revisit: after Phase 1, repo
|
| 11 |
transfers to the lab org (design §7 step 7).
|
| 12 |
|
| 13 |
-
## 2026-07-09
|
| 14 |
Durable queue via SELECT ... FOR UPDATE SKIP LOCKED + leases + sweeper; job state already in
|
| 15 |
the DB. Rejected: Redis/RQ (a third stateful service before pressure exists). Trigger to
|
| 16 |
revisit: multiple workers, scheduling needs, sustained queue depth.
|
| 17 |
|
| 18 |
-
## 2026-07-09
|
| 19 |
Rejected: custom password auth (hashing, reset flows, verification email, lockouts = hidden
|
| 20 |
security work). Staged: Google sign-in first (zero email infra), email/password second within
|
| 21 |
the same provider. Trigger: PI sign-off starts Phase 2.
|
| 22 |
|
| 23 |
-
## 2026-07-09
|
| 24 |
MiniLM is the CCR reference model → comparability with the published method; E5 models require
|
| 25 |
"query: " prefixes (encoded in registry usage_config). Rejected: e5-large-v2 as default
|
| 26 |
(slow on CPU, breaks anonymous inline tier; cross-family scores not comparable). Trigger:
|
| 27 |
lab validation study could change defaults.
|
| 28 |
|
| 29 |
-
## 2026-07-09
|
| 30 |
Aggregates exclude reversed items only when explicitly required; all-reversed blocks aggregate.
|
| 31 |
Rejected: sign-negation by default (silently invented psychometrics). Trigger: lab methods
|
| 32 |
decision after discussion with PI.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
# Decisions Log
|
| 2 |
|
| 3 |
> One entry per architecture/infrastructure/vendor/model decision.
|
| 4 |
+
> Format: date - decision - why - rejected alternative - revisit trigger.
|
| 5 |
> CLAUDE.md hard rule: no new infrastructure without an entry here.
|
| 6 |
|
| 7 |
+
## 2026-07-09 - Repo seeded from demo; demo stays live
|
| 8 |
Mainline moves to this repo (future home: Culture-and-Morality-Lab org). The public demo repo
|
| 9 |
+
and HF Space remain deployed and untouched - the PI links people to it. Rejected: evolving the
|
| 10 |
demo repo in place (would couple experiments to a live artifact). Revisit: after Phase 1, repo
|
| 11 |
transfers to the lab org (design §7 step 7).
|
| 12 |
|
| 13 |
+
## 2026-07-09 - Postgres jobs table over Redis/RQ (design §6)
|
| 14 |
Durable queue via SELECT ... FOR UPDATE SKIP LOCKED + leases + sweeper; job state already in
|
| 15 |
the DB. Rejected: Redis/RQ (a third stateful service before pressure exists). Trigger to
|
| 16 |
revisit: multiple workers, scheduling needs, sustained queue depth.
|
| 17 |
|
| 18 |
+
## 2026-07-09 - Managed auth (Supabase Auth recommended), staged Google-first (design §8, §4.1)
|
| 19 |
Rejected: custom password auth (hashing, reset flows, verification email, lockouts = hidden
|
| 20 |
security work). Staged: Google sign-in first (zero email infra), email/password second within
|
| 21 |
the same provider. Trigger: PI sign-off starts Phase 2.
|
| 22 |
|
| 23 |
+
## 2026-07-09 - MiniLM default; e5-large-v2 strong option; multilingual-E5 (design §13)
|
| 24 |
MiniLM is the CCR reference model → comparability with the published method; E5 models require
|
| 25 |
"query: " prefixes (encoded in registry usage_config). Rejected: e5-large-v2 as default
|
| 26 |
(slow on CPU, breaks anonymous inline tier; cross-family scores not comparable). Trigger:
|
| 27 |
lab validation study could change defaults.
|
| 28 |
|
| 29 |
+
## 2026-07-09 - Reverse scoring: raw + flags only in v1 (design §11)
|
| 30 |
Aggregates exclude reversed items only when explicitly required; all-reversed blocks aggregate.
|
| 31 |
Rejected: sign-negation by default (silently invented psychometrics). Trigger: lab methods
|
| 32 |
decision after discussion with PI.
|
| 33 |
+
|
| 34 |
+
## 2026-07-09 - Placeholder demo sign-in for tier testing (not security)
|
| 35 |
+
Anonymous caps (2 MB / 500 rows, env-tunable) enforce the design §5.1 tiers now; a
|
| 36 |
+
name-only signed-cookie session lifts them so the tiered UX is testable before Supabase
|
| 37 |
+
auth lands. get_current_user() is the single Phase 2 integration point. Rejected: waiting
|
| 38 |
+
for real auth (tiers untestable) and client-only gating (trivially bypassed).
|
| 39 |
+
|
| 40 |
+
## 2026-07-09 - Open archive/delete until accounts exist
|
| 41 |
+
Any visitor can archive/delete any project on the shared instance: there are no owners yet,
|
| 42 |
+
and mistake-cleanup matters more than protection on a demo. Delete requires typing the
|
| 43 |
+
project name, cascades to all files/rows, and is logged without text (design §9). Phase 2
|
| 44 |
+
adds ownership checks to the same endpoints.
|
| 45 |
+
|
| 46 |
+
## 2026-07-09 - No em dashes in project text (Deva)
|
| 47 |
+
Style rule across UI, docs, comments, commits. Enforced by a PostToolUse hook; swept 50
|
| 48 |
+
existing files.
|
DEPLOY.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
| 1 |
# Deploying the demo
|
| 2 |
|
| 3 |
-
## Option A
|
| 4 |
|
| 5 |
Free Docker Spaces: 2 vCPU / 16 GB RAM, no credit card. Bonus: the original
|
| 6 |
CCR online tool lives on HF Spaces, so the prototype sits where the CCR
|
|
@@ -34,19 +34,19 @@ community already works.
|
|
| 34 |
First build takes ~5–10 min (model bakes into the image). Watch the
|
| 35 |
build logs in the Space's "Logs" tab.
|
| 36 |
|
| 37 |
-
4. Optional hardening for the public instance
|
| 38 |
Variables, set `CCR_MAX_ROWS=20000` (tighter ceiling than the
|
| 39 |
100k default while strangers can reach it).
|
| 40 |
|
| 41 |
Notes:
|
| 42 |
-
- Storage is **ephemeral**
|
| 43 |
Fine for a demo; the email and the in-app welcome text both say so.
|
| 44 |
- Free Spaces sleep after ~48h without traffic. Visit the URL the
|
| 45 |
evening before and the morning of the interview so it's warm.
|
| 46 |
- The direct app URL (no HF frame) is
|
| 47 |
-
`https://<username>-ccr-platform.hf.space`
|
| 48 |
|
| 49 |
-
## Option B
|
| 50 |
|
| 51 |
```bash
|
| 52 |
gcloud run deploy ccr-platform \
|
|
@@ -62,7 +62,7 @@ gcloud run deploy ccr-platform \
|
|
| 62 |
|
| 63 |
- `--min-instances 1`: no cold starts while he plays with it (~a few
|
| 64 |
dollars for the week; delete the service after the process ends).
|
| 65 |
-
- `--max-instances 1`: SQLite + in-process queue assume one instance
|
| 66 |
documented demo trade-off, not an oversight.
|
| 67 |
|
| 68 |
## Pre-send checklist (either host)
|
|
@@ -72,9 +72,9 @@ gcloud run deploy ccr-platform \
|
|
| 72 |
Satisfaction with Life → Run → results render → Export CSV downloads.
|
| 73 |
3. Second run (Individualism) finishes in seconds (model + item cache warm).
|
| 74 |
4. Upload your own messy CSV (Excel export with a BOM, or semicolon-
|
| 75 |
-
delimited)
|
| 76 |
5. Upload rejects a bogus file (.txt/.exe) with a clean error.
|
| 77 |
-
6. Refresh the page
|
| 78 |
7. Morning of the interview: open the URL once (warm the instance),
|
| 79 |
re-run step 2 quickly.
|
| 80 |
|
|
|
|
| 1 |
# Deploying the demo
|
| 2 |
|
| 3 |
+
## Option A - Hugging Face Spaces (free, recommended for the demo)
|
| 4 |
|
| 5 |
Free Docker Spaces: 2 vCPU / 16 GB RAM, no credit card. Bonus: the original
|
| 6 |
CCR online tool lives on HF Spaces, so the prototype sits where the CCR
|
|
|
|
| 34 |
First build takes ~5–10 min (model bakes into the image). Watch the
|
| 35 |
build logs in the Space's "Logs" tab.
|
| 36 |
|
| 37 |
+
4. Optional hardening for the public instance - in Space Settings →
|
| 38 |
Variables, set `CCR_MAX_ROWS=20000` (tighter ceiling than the
|
| 39 |
100k default while strangers can reach it).
|
| 40 |
|
| 41 |
Notes:
|
| 42 |
+
- Storage is **ephemeral** - uploads/results vanish on restart or rebuild.
|
| 43 |
Fine for a demo; the email and the in-app welcome text both say so.
|
| 44 |
- Free Spaces sleep after ~48h without traffic. Visit the URL the
|
| 45 |
evening before and the morning of the interview so it's warm.
|
| 46 |
- The direct app URL (no HF frame) is
|
| 47 |
+
`https://<username>-ccr-platform.hf.space` - send that one.
|
| 48 |
|
| 49 |
+
## Option B - Google Cloud Run (few dollars, more "prod-like" URL)
|
| 50 |
|
| 51 |
```bash
|
| 52 |
gcloud run deploy ccr-platform \
|
|
|
|
| 62 |
|
| 63 |
- `--min-instances 1`: no cold starts while he plays with it (~a few
|
| 64 |
dollars for the week; delete the service after the process ends).
|
| 65 |
+
- `--max-instances 1`: SQLite + in-process queue assume one instance -
|
| 66 |
documented demo trade-off, not an oversight.
|
| 67 |
|
| 68 |
## Pre-send checklist (either host)
|
|
|
|
| 72 |
Satisfaction with Life → Run → results render → Export CSV downloads.
|
| 73 |
3. Second run (Individualism) finishes in seconds (model + item cache warm).
|
| 74 |
4. Upload your own messy CSV (Excel export with a BOM, or semicolon-
|
| 75 |
+
delimited) - parses, and any fallback is flagged in the UI.
|
| 76 |
5. Upload rejects a bogus file (.txt/.exe) with a clean error.
|
| 77 |
+
6. Refresh the page - SPA loads, project still listed.
|
| 78 |
7. Morning of the interview: open the URL once (warm the instance),
|
| 79 |
re-run step 2 quickly.
|
| 80 |
|
Dockerfile
CHANGED
|
@@ -1,4 +1,4 @@
|
|
| 1 |
-
# CCR Platform
|
| 2 |
# The embedding model is baked into the image so the first request
|
| 3 |
# doesn't trigger a ~90 MB download (critical for demo cold starts).
|
| 4 |
|
|
@@ -15,7 +15,7 @@ WORKDIR /srv
|
|
| 15 |
COPY backend/requirements.txt .
|
| 16 |
RUN pip install --no-cache-dir -r requirements.txt
|
| 17 |
|
| 18 |
-
# Pre-download ALL offered models into the image layer
|
| 19 |
# non-default model must not trigger a multi-hundred-MB download mid-job
|
| 20 |
# (looks like a hang). Make the cache usable by any runtime UID.
|
| 21 |
RUN python -c "from sentence_transformers import SentenceTransformer; \
|
|
@@ -28,7 +28,7 @@ RUN python -c "from sentence_transformers import SentenceTransformer; \
|
|
| 28 |
COPY backend/app ./app
|
| 29 |
COPY backend/static ./static
|
| 30 |
|
| 31 |
-
# Demo note: SQLite + uploads live on the container's ephemeral disk
|
| 32 |
# data resets on restart/redeploy. Acceptable for a demo; use
|
| 33 |
# Postgres + S3/GCS object storage before any real use.
|
| 34 |
EXPOSE 7860
|
|
|
|
| 1 |
+
# CCR Platform - Cloud Run / container deployment.
|
| 2 |
# The embedding model is baked into the image so the first request
|
| 3 |
# doesn't trigger a ~90 MB download (critical for demo cold starts).
|
| 4 |
|
|
|
|
| 15 |
COPY backend/requirements.txt .
|
| 16 |
RUN pip install --no-cache-dir -r requirements.txt
|
| 17 |
|
| 18 |
+
# Pre-download ALL offered models into the image layer - a user picking a
|
| 19 |
# non-default model must not trigger a multi-hundred-MB download mid-job
|
| 20 |
# (looks like a hang). Make the cache usable by any runtime UID.
|
| 21 |
RUN python -c "from sentence_transformers import SentenceTransformer; \
|
|
|
|
| 28 |
COPY backend/app ./app
|
| 29 |
COPY backend/static ./static
|
| 30 |
|
| 31 |
+
# Demo note: SQLite + uploads live on the container's ephemeral disk -
|
| 32 |
# data resets on restart/redeploy. Acceptable for a demo; use
|
| 33 |
# Postgres + S3/GCS object storage before any real use.
|
| 34 |
EXPOSE 7860
|
README.md
CHANGED
|
@@ -1,4 +1,4 @@
|
|
| 1 |
-
# CCR Platform
|
| 2 |
|
| 3 |
Web platform for **Contextualized Construct Representation (CCR)** psychological text analysis
|
| 4 |
([method](https://github.com/Ali-Omrani/CCR) · [EMNLP 2024](https://aclanthology.org/2024.emnlp-main.151/)).
|
|
@@ -20,7 +20,7 @@ reproducibility record.
|
|
| 20 |
```
|
| 21 |
|
| 22 |
```bash
|
| 23 |
-
# fast test suite (deterministic fake embedder
|
| 24 |
cd backend && pip install -r requirements-dev.txt && python -m pytest tests/ -q
|
| 25 |
|
| 26 |
# validate the registries
|
|
@@ -37,7 +37,7 @@ python scripts/verify_install.py
|
|
| 37 |
backend/ FastAPI app + tests (current working application)
|
| 38 |
frontend/ React/Vite SPA source (prebuilt copy served from backend/static/)
|
| 39 |
packages/
|
| 40 |
-
model_registry/ models.yaml
|
| 41 |
construct_library/ versioned construct YAMLs + validator (append-only)
|
| 42 |
ccr_engine/ Phase 1 target: pure analysis package (see spec 0005)
|
| 43 |
docs/
|
|
@@ -49,7 +49,7 @@ evals/ golden datasets + expected outputs (human-approved only)
|
|
| 49 |
|
| 50 |
## Working on this repo
|
| 51 |
|
| 52 |
-
Read `CLAUDE.md` first
|
| 53 |
hooks). Features start from `docs/specs/`; decisions land in `DECISIONS.md`; user-visible
|
| 54 |
changes land in `CHANGELOG.md`. The published `ccr_wrapper` implementation is the correctness
|
| 55 |
spec for the engine.
|
|
|
|
| 1 |
+
# CCR Platform - Culture and Morality Lab
|
| 2 |
|
| 3 |
Web platform for **Contextualized Construct Representation (CCR)** psychological text analysis
|
| 4 |
([method](https://github.com/Ali-Omrani/CCR) · [EMNLP 2024](https://aclanthology.org/2024.emnlp-main.151/)).
|
|
|
|
| 20 |
```
|
| 21 |
|
| 22 |
```bash
|
| 23 |
+
# fast test suite (deterministic fake embedder - no torch needed)
|
| 24 |
cd backend && pip install -r requirements-dev.txt && python -m pytest tests/ -q
|
| 25 |
|
| 26 |
# validate the registries
|
|
|
|
| 37 |
backend/ FastAPI app + tests (current working application)
|
| 38 |
frontend/ React/Vite SPA source (prebuilt copy served from backend/static/)
|
| 39 |
packages/
|
| 40 |
+
model_registry/ models.yaml - single source of model truth + validator
|
| 41 |
construct_library/ versioned construct YAMLs + validator (append-only)
|
| 42 |
ccr_engine/ Phase 1 target: pure analysis package (see spec 0005)
|
| 43 |
docs/
|
|
|
|
| 49 |
|
| 50 |
## Working on this repo
|
| 51 |
|
| 52 |
+
Read `CLAUDE.md` first - it carries the architecture rules and hard rules (enforced partly by
|
| 53 |
hooks). Features start from `docs/specs/`; decisions land in `DECISIONS.md`; user-visible
|
| 54 |
changes land in `CHANGELOG.md`. The published `ccr_wrapper` implementation is the correctness
|
| 55 |
spec for the engine.
|
ROADMAP.md
CHANGED
|
@@ -1,23 +1,23 @@
|
|
| 1 |
-
# Roadmap (from design doc §4 / §20
|
| 2 |
|
| 3 |
-
## Phase 0
|
| 4 |
-
- [x] Descriptive stats (mean/SD/min/max)
|
| 5 |
- [x] Spec 0001: short-text + language warnings (structured warning objects)
|
| 6 |
- [x] Spec 0002: offline-runnable Python script export + pinned requirements
|
| 7 |
- [x] Spec 0003: registry-driven models (E5 prefixes!) + language selection
|
| 8 |
- [x] Spec 0004: construct library from versioned YAML (assistant's questionnaires land here)
|
| 9 |
|
| 10 |
-
## Phase 1
|
| 11 |
- [ ] Spec 0005: extract packages/ccr_engine + golden evals + ccr_wrapper parity tests
|
| 12 |
- [ ] Pin model revisions in models.yaml (remove PIN_ME)
|
| 13 |
- [ ] Registry validation in CI
|
| 14 |
|
| 15 |
-
## Phase 2
|
| 16 |
- [ ] Supabase Auth (Google first, then email/password) · projects · Postgres metadata + jobs
|
| 17 |
- [ ] Object storage behind the existing storage interface · TTL cleanup (anonymous 48h)
|
| 18 |
- [ ] Anonymous caps per design §5.1 starter table
|
| 19 |
|
| 20 |
-
## Phase 3
|
| 21 |
- [ ] Repo → Culture-and-Morality-Lab org · staging + production hosting (Option B)
|
| 22 |
- [ ] Backups, monitoring, privacy wording (lab-provided), verified-constructs workflow
|
| 23 |
|
|
|
|
| 1 |
+
# Roadmap (from design doc §4 / §20 - capacity: one person, ≤20 hrs/week)
|
| 2 |
|
| 3 |
+
## Phase 0 - Visible progress (now)
|
| 4 |
+
- [x] Descriptive stats (mean/SD/min/max) - shipped in demo codebase
|
| 5 |
- [x] Spec 0001: short-text + language warnings (structured warning objects)
|
| 6 |
- [x] Spec 0002: offline-runnable Python script export + pinned requirements
|
| 7 |
- [x] Spec 0003: registry-driven models (E5 prefixes!) + language selection
|
| 8 |
- [x] Spec 0004: construct library from versioned YAML (assistant's questionnaires land here)
|
| 9 |
|
| 10 |
+
## Phase 1 - Engine hardening (weeks 3–5)
|
| 11 |
- [ ] Spec 0005: extract packages/ccr_engine + golden evals + ccr_wrapper parity tests
|
| 12 |
- [ ] Pin model revisions in models.yaml (remove PIN_ME)
|
| 13 |
- [ ] Registry validation in CI
|
| 14 |
|
| 15 |
+
## Phase 2 - Persistence + auth (weeks 6–9, AFTER §22 sign-off)
|
| 16 |
- [ ] Supabase Auth (Google first, then email/password) · projects · Postgres metadata + jobs
|
| 17 |
- [ ] Object storage behind the existing storage interface · TTL cleanup (anonymous 48h)
|
| 18 |
- [ ] Anonymous caps per design §5.1 starter table
|
| 19 |
|
| 20 |
+
## Phase 3 - Production-lite launch
|
| 21 |
- [ ] Repo → Culture-and-Morality-Lab org · staging + production hosting (Option B)
|
| 22 |
- [ ] Backups, monitoring, privacy wording (lab-provided), verified-constructs workflow
|
| 23 |
|
WORKLOG.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
| 1 |
# Worklog pointer
|
| 2 |
|
| 3 |
Hours and task-level logging live in the private lab worklog
|
| 4 |
-
(`../CAM-Lab/worklog.md` on Deva's machine
|
| 5 |
|
| 6 |
This file exists so repo docs can reference a stable path; keep private details out of the repo.
|
|
|
|
| 1 |
# Worklog pointer
|
| 2 |
|
| 3 |
Hours and task-level logging live in the private lab worklog
|
| 4 |
+
(`../CAM-Lab/worklog.md` on Deva's machine - not committed here).
|
| 5 |
|
| 6 |
This file exists so repo docs can reference a stable path; keep private details out of the repo.
|
backend/app/auth_demo.py
ADDED
|
@@ -0,0 +1,64 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Placeholder sign-in for tier testing (design doc §4.1 staged rollout).
|
| 2 |
+
|
| 3 |
+
This is deliberately NOT security. It exists so the anonymous-vs-signed-in
|
| 4 |
+
upload tiers have real UX and enforcement before the managed auth provider
|
| 5 |
+
(Supabase: Google first, then email/password) lands in Phase 2. The swap
|
| 6 |
+
plan: get_current_user() is the only integration point - Phase 2 replaces
|
| 7 |
+
its body with provider JWT verification and nothing else changes.
|
| 8 |
+
|
| 9 |
+
Session: HMAC-signed cookie. Secret comes from CCR_SESSION_SECRET or is
|
| 10 |
+
random per process (restart signs everyone out - acceptable for a demo).
|
| 11 |
+
"""
|
| 12 |
+
|
| 13 |
+
from __future__ import annotations
|
| 14 |
+
|
| 15 |
+
import base64
|
| 16 |
+
import hashlib
|
| 17 |
+
import hmac
|
| 18 |
+
import os
|
| 19 |
+
import secrets
|
| 20 |
+
|
| 21 |
+
from fastapi import Request
|
| 22 |
+
|
| 23 |
+
COOKIE_NAME = "ccr_demo_session"
|
| 24 |
+
_SECRET = (os.environ.get("CCR_SESSION_SECRET") or secrets.token_hex(32)).encode()
|
| 25 |
+
|
| 26 |
+
# Tier limits; env-overridable so tests and the hosted demo can tune them.
|
| 27 |
+
ANON_MAX_BYTES_DEFAULT = 2 * 1024 * 1024
|
| 28 |
+
ANON_MAX_ROWS_DEFAULT = 500
|
| 29 |
+
|
| 30 |
+
|
| 31 |
+
def anon_max_bytes() -> int:
|
| 32 |
+
return int(os.environ.get("CCR_ANON_MAX_BYTES", ANON_MAX_BYTES_DEFAULT))
|
| 33 |
+
|
| 34 |
+
|
| 35 |
+
def anon_max_rows() -> int:
|
| 36 |
+
return int(os.environ.get("CCR_ANON_MAX_ROWS", ANON_MAX_ROWS_DEFAULT))
|
| 37 |
+
|
| 38 |
+
|
| 39 |
+
def _sign(payload: bytes) -> str:
|
| 40 |
+
return hmac.new(_SECRET, payload, hashlib.sha256).hexdigest()
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
def create_token(name: str) -> str:
|
| 44 |
+
payload = base64.urlsafe_b64encode(name.encode()).decode()
|
| 45 |
+
return f"{payload}.{_sign(payload.encode())}"
|
| 46 |
+
|
| 47 |
+
|
| 48 |
+
def verify_token(token: str | None) -> str | None:
|
| 49 |
+
"""Return the signed-in name, or None for anonymous/invalid tokens."""
|
| 50 |
+
if not token or "." not in token:
|
| 51 |
+
return None
|
| 52 |
+
payload, signature = token.rsplit(".", 1)
|
| 53 |
+
if not hmac.compare_digest(signature, _sign(payload.encode())):
|
| 54 |
+
return None
|
| 55 |
+
try:
|
| 56 |
+
return base64.urlsafe_b64decode(payload.encode()).decode()
|
| 57 |
+
except Exception:
|
| 58 |
+
return None
|
| 59 |
+
|
| 60 |
+
|
| 61 |
+
def get_current_user(request: Request) -> dict | None:
|
| 62 |
+
"""Phase 2 integration point: replace body with provider JWT verification."""
|
| 63 |
+
name = verify_token(request.cookies.get(COOKIE_NAME))
|
| 64 |
+
return {"name": name, "tier": "member"} if name else None
|
backend/app/ccr.py
CHANGED
|
@@ -1,4 +1,4 @@
|
|
| 1 |
-
"""CCR engine
|
| 2 |
|
| 3 |
Method (Atari, Omrani et al.): embed validated questionnaire items and the
|
| 4 |
texts to be analyzed with a contextual sentence-embedding model, then take
|
|
@@ -7,7 +7,7 @@ similarities are the text's "loadings" on the construct; their mean is the
|
|
| 7 |
overall CCR score.
|
| 8 |
|
| 9 |
The embedding model is injected behind a small interface so that:
|
| 10 |
-
* production uses sentence-transformers (local, pinned, reproducible
|
| 11 |
corpora never leave the machine), and
|
| 12 |
* tests/CI use a deterministic hash-based embedder with no ML dependency.
|
| 13 |
"""
|
|
@@ -126,7 +126,7 @@ class CCRResult:
|
|
| 126 |
|
| 127 |
|
| 128 |
# Item-set embeddings are tiny and constantly reused (same construct run
|
| 129 |
-
# against many corpora)
|
| 130 |
_item_embedding_cache: dict[tuple[str, str], np.ndarray] = {}
|
| 131 |
|
| 132 |
|
|
@@ -150,7 +150,7 @@ def run_ccr(
|
|
| 150 |
) -> CCRResult:
|
| 151 |
"""Compute CCR loadings: cosine(text, item) for every text × item pair.
|
| 152 |
|
| 153 |
-
Prefixes come from the model registry's usage_config
|
| 154 |
"query: " on BOTH sides for symmetric similarity. Prefixed strings feed the
|
| 155 |
encoder only; raw wording is what gets hashed and exported.
|
| 156 |
"""
|
|
|
|
| 1 |
+
"""CCR engine - Contextualized Construct Representations.
|
| 2 |
|
| 3 |
Method (Atari, Omrani et al.): embed validated questionnaire items and the
|
| 4 |
texts to be analyzed with a contextual sentence-embedding model, then take
|
|
|
|
| 7 |
overall CCR score.
|
| 8 |
|
| 9 |
The embedding model is injected behind a small interface so that:
|
| 10 |
+
* production uses sentence-transformers (local, pinned, reproducible -
|
| 11 |
corpora never leave the machine), and
|
| 12 |
* tests/CI use a deterministic hash-based embedder with no ML dependency.
|
| 13 |
"""
|
|
|
|
| 126 |
|
| 127 |
|
| 128 |
# Item-set embeddings are tiny and constantly reused (same construct run
|
| 129 |
+
# against many corpora) - cache them per (model, exact item wording).
|
| 130 |
_item_embedding_cache: dict[tuple[str, str], np.ndarray] = {}
|
| 131 |
|
| 132 |
|
|
|
|
| 150 |
) -> CCRResult:
|
| 151 |
"""Compute CCR loadings: cosine(text, item) for every text × item pair.
|
| 152 |
|
| 153 |
+
Prefixes come from the model registry's usage_config - E5-family models require
|
| 154 |
"query: " on BOTH sides for symmetric similarity. Prefixed strings feed the
|
| 155 |
encoder only; raw wording is what gets hashed and exported.
|
| 156 |
"""
|
backend/app/construct_lib.py
CHANGED
|
@@ -1,11 +1,11 @@
|
|
| 1 |
-
"""Construct library loader
|
| 2 |
|
| 3 |
Source of truth is the versioned YAML files (spec 0004, design doc §10.1). Rules:
|
| 4 |
-
* append-only: (construct_id, version) is immutable
|
| 5 |
items is a hard error, never a silent update;
|
| 6 |
* item_hash uses the REFERENCE implementation from validate_constructs.py (loaded
|
| 7 |
by file path) so validator, seeder, and metadata always agree;
|
| 8 |
-
* verification_status flows to the UI
|
| 9 |
|
| 10 |
New questionnaires from the lab land as new YAML files; `python packages/construct_library/
|
| 11 |
validate_constructs.py` first, then restart the app (or call sync) to pick them up.
|
|
|
|
| 1 |
+
"""Construct library loader - seeds the DB from packages/construct_library/constructs/.
|
| 2 |
|
| 3 |
Source of truth is the versioned YAML files (spec 0004, design doc §10.1). Rules:
|
| 4 |
+
* append-only: (construct_id, version) is immutable - same version with changed
|
| 5 |
items is a hard error, never a silent update;
|
| 6 |
* item_hash uses the REFERENCE implementation from validate_constructs.py (loaded
|
| 7 |
by file path) so validator, seeder, and metadata always agree;
|
| 8 |
+
* verification_status flows to the UI - unverified wording is visibly flagged.
|
| 9 |
|
| 10 |
New questionnaires from the lab land as new YAML files; `python packages/construct_library/
|
| 11 |
validate_constructs.py` first, then restart the app (or call sync) to pick them up.
|
backend/app/db.py
CHANGED
|
@@ -1,4 +1,4 @@
|
|
| 1 |
-
"""Database setup
|
| 2 |
|
| 3 |
SQLite is a deliberate choice for this deployment size (single-node, few
|
| 4 |
concurrent writers). The models use no SQLite-specific features, so moving
|
|
|
|
| 1 |
+
"""Database setup - SQLite via SQLAlchemy.
|
| 2 |
|
| 3 |
SQLite is a deliberate choice for this deployment size (single-node, few
|
| 4 |
concurrent writers). The models use no SQLite-specific features, so moving
|
backend/app/ingest.py
CHANGED
|
@@ -1,4 +1,4 @@
|
|
| 1 |
-
"""Corpus ingestion
|
| 2 |
|
| 3 |
Researchers upload CSVs exported from Qualtrics, Excel, R, SPSS, and
|
| 4 |
scrapers: BOMs, latin-1 encodings, semicolon/tab delimiters, ragged rows.
|
|
@@ -34,7 +34,7 @@ def load_corpus(path: str | Path) -> tuple[pd.DataFrame, dict]:
|
|
| 34 |
"""Parse CSV/XLSX into a DataFrame.
|
| 35 |
|
| 36 |
Returns (df, parse_info) where parse_info records the format,
|
| 37 |
-
encoding, and delimiter actually used
|
| 38 |
echoed into every run's reproducibility metadata.
|
| 39 |
"""
|
| 40 |
p = Path(path)
|
|
@@ -85,7 +85,7 @@ _DELIMITER_CANDIDATES = (",", ";", "\t", "|")
|
|
| 85 |
def _detect_delimiter(p: Path, encoding: str) -> str:
|
| 86 |
"""Pick the candidate delimiter most consistent across the first lines.
|
| 87 |
|
| 88 |
-
Single-column files (no candidate present) default to ','
|
| 89 |
of a delimiter-free file yields one column, which is exactly right.
|
| 90 |
"""
|
| 91 |
try:
|
|
@@ -111,7 +111,7 @@ def _validate(df: pd.DataFrame) -> pd.DataFrame:
|
|
| 111 |
raise IngestError("The file parsed but contains no data rows.")
|
| 112 |
if len(df) > max_rows():
|
| 113 |
raise IngestError(
|
| 114 |
-
f"File has {len(df):,} rows
|
| 115 |
f"{max_rows():,}-row limit. Split the corpus or run locally."
|
| 116 |
)
|
| 117 |
df.columns = [str(c) for c in df.columns]
|
|
|
|
| 1 |
+
"""Corpus ingestion - tolerant of real-world research files.
|
| 2 |
|
| 3 |
Researchers upload CSVs exported from Qualtrics, Excel, R, SPSS, and
|
| 4 |
scrapers: BOMs, latin-1 encodings, semicolon/tab delimiters, ragged rows.
|
|
|
|
| 34 |
"""Parse CSV/XLSX into a DataFrame.
|
| 35 |
|
| 36 |
Returns (df, parse_info) where parse_info records the format,
|
| 37 |
+
encoding, and delimiter actually used - stored with the corpus and
|
| 38 |
echoed into every run's reproducibility metadata.
|
| 39 |
"""
|
| 40 |
p = Path(path)
|
|
|
|
| 85 |
def _detect_delimiter(p: Path, encoding: str) -> str:
|
| 86 |
"""Pick the candidate delimiter most consistent across the first lines.
|
| 87 |
|
| 88 |
+
Single-column files (no candidate present) default to ',' - a comma parse
|
| 89 |
of a delimiter-free file yields one column, which is exactly right.
|
| 90 |
"""
|
| 91 |
try:
|
|
|
|
| 111 |
raise IngestError("The file parsed but contains no data rows.")
|
| 112 |
if len(df) > max_rows():
|
| 113 |
raise IngestError(
|
| 114 |
+
f"File has {len(df):,} rows - above this instance's "
|
| 115 |
f"{max_rows():,}-row limit. Split the corpus or run locally."
|
| 116 |
)
|
| 117 |
df.columns = [str(c) for c in df.columns]
|
backend/app/jobs.py
CHANGED
|
@@ -1,6 +1,6 @@
|
|
| 1 |
"""Background job runner.
|
| 2 |
|
| 3 |
-
Jobs run on a dedicated single-worker executor
|
| 4 |
embedding is CPU-bound, so running jobs sequentially protects the instance's
|
| 5 |
memory and keeps per-job throughput predictable, while job state lives in
|
| 6 |
the DB (queued → running → completed/failed) so the API and UI never depend
|
|
@@ -42,15 +42,32 @@ TOP_N = 10
|
|
| 42 |
SNIPPET_LEN = 220
|
| 43 |
|
| 44 |
# Single worker: sequential jobs, bounded memory. See module docstring.
|
| 45 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 46 |
|
| 47 |
|
| 48 |
def submit_job(job_id: str) -> None:
|
| 49 |
-
|
| 50 |
|
| 51 |
|
| 52 |
def shutdown_executor() -> None:
|
| 53 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 54 |
|
| 55 |
|
| 56 |
def recover_orphaned_jobs() -> int:
|
|
@@ -129,7 +146,7 @@ def run_job(job_id: str) -> None:
|
|
| 129 |
progress_cb=progress, item_prefix=item_prefix, text_prefix=text_prefix,
|
| 130 |
)
|
| 131 |
|
| 132 |
-
# Structured data-quality warnings (spec 0001)
|
| 133 |
W = warnings_engine.warning
|
| 134 |
warnings: list[dict] = []
|
| 135 |
if dropped:
|
|
@@ -141,7 +158,7 @@ def run_job(job_id: str) -> None:
|
|
| 141 |
if n_dupes:
|
| 142 |
warnings.append(W(
|
| 143 |
"DUPLICATE_TEXTS", "warning",
|
| 144 |
-
f"{n_dupes} duplicate text(s) detected
|
| 145 |
"deduplicate upstream if unintended.", count=n_dupes,
|
| 146 |
))
|
| 147 |
short = warnings_engine.short_text_warning(texts)
|
|
|
|
| 1 |
"""Background job runner.
|
| 2 |
|
| 3 |
+
Jobs run on a dedicated single-worker executor - a deliberate right-sizing:
|
| 4 |
embedding is CPU-bound, so running jobs sequentially protects the instance's
|
| 5 |
memory and keeps per-job throughput predictable, while job state lives in
|
| 6 |
the DB (queued → running → completed/failed) so the API and UI never depend
|
|
|
|
| 42 |
SNIPPET_LEN = 220
|
| 43 |
|
| 44 |
# Single worker: sequential jobs, bounded memory. See module docstring.
|
| 45 |
+
# Created lazily and re-creatable: a lifespan shutdown (dev reload, test
|
| 46 |
+
# client closing) must not permanently kill job submission for the process.
|
| 47 |
+
import threading
|
| 48 |
+
|
| 49 |
+
_executor: ThreadPoolExecutor | None = None
|
| 50 |
+
_executor_lock = threading.Lock()
|
| 51 |
+
|
| 52 |
+
|
| 53 |
+
def _get_executor() -> ThreadPoolExecutor:
|
| 54 |
+
global _executor
|
| 55 |
+
with _executor_lock:
|
| 56 |
+
if _executor is None:
|
| 57 |
+
_executor = ThreadPoolExecutor(max_workers=1, thread_name_prefix="ccr-job")
|
| 58 |
+
return _executor
|
| 59 |
|
| 60 |
|
| 61 |
def submit_job(job_id: str) -> None:
|
| 62 |
+
_get_executor().submit(_run_job_logged, job_id)
|
| 63 |
|
| 64 |
|
| 65 |
def shutdown_executor() -> None:
|
| 66 |
+
global _executor
|
| 67 |
+
with _executor_lock:
|
| 68 |
+
if _executor is not None:
|
| 69 |
+
_executor.shutdown(wait=False, cancel_futures=True)
|
| 70 |
+
_executor = None
|
| 71 |
|
| 72 |
|
| 73 |
def recover_orphaned_jobs() -> int:
|
|
|
|
| 146 |
progress_cb=progress, item_prefix=item_prefix, text_prefix=text_prefix,
|
| 147 |
)
|
| 148 |
|
| 149 |
+
# Structured data-quality warnings (spec 0001) - objects, never bare strings.
|
| 150 |
W = warnings_engine.warning
|
| 151 |
warnings: list[dict] = []
|
| 152 |
if dropped:
|
|
|
|
| 158 |
if n_dupes:
|
| 159 |
warnings.append(W(
|
| 160 |
"DUPLICATE_TEXTS", "warning",
|
| 161 |
+
f"{n_dupes} duplicate text(s) detected - each is scored independently; "
|
| 162 |
"deduplicate upstream if unintended.", count=n_dupes,
|
| 163 |
))
|
| 164 |
short = warnings_engine.short_text_warning(texts)
|
backend/app/main.py
CHANGED
|
@@ -1,4 +1,4 @@
|
|
| 1 |
-
"""CCR Platform
|
| 2 |
|
| 3 |
Single deployable: serves the JSON API under /api and the prebuilt React
|
| 4 |
dashboard as static files at /. Local-first by design: corpora, embeddings,
|
|
@@ -13,12 +13,13 @@ import json
|
|
| 13 |
from contextlib import asynccontextmanager
|
| 14 |
from pathlib import Path
|
| 15 |
|
| 16 |
-
from fastapi import Depends, FastAPI, HTTPException, UploadFile
|
| 17 |
from fastapi.middleware.cors import CORSMiddleware
|
| 18 |
from fastapi.responses import FileResponse, JSONResponse, PlainTextResponse
|
| 19 |
from fastapi.staticfiles import StaticFiles
|
| 20 |
from sqlalchemy.orm import Session
|
| 21 |
|
|
|
|
| 22 |
from . import jobs as jobs_module
|
| 23 |
from . import registry
|
| 24 |
from .ccr import FAKE_MODEL_NAME
|
|
@@ -31,10 +32,12 @@ from .schemas import (
|
|
| 31 |
ConstructCreate,
|
| 32 |
ConstructOut,
|
| 33 |
CorpusOut,
|
|
|
|
| 34 |
JobCreate,
|
| 35 |
JobOut,
|
| 36 |
ProjectCreate,
|
| 37 |
ProjectOut,
|
|
|
|
| 38 |
)
|
| 39 |
|
| 40 |
MAX_UPLOAD_BYTES = 25 * 1024 * 1024 # sane lab-scale ceiling; raise deliberately
|
|
@@ -133,7 +136,7 @@ def health():
|
|
| 133 |
|
| 134 |
@app.get("/api/models")
|
| 135 |
def list_models():
|
| 136 |
-
"""Model options from the registry (spec 0003)
|
| 137 |
return [
|
| 138 |
{
|
| 139 |
"id": m.id,
|
|
@@ -153,10 +156,47 @@ def list_languages():
|
|
| 153 |
return SELECTABLE_LANGUAGES
|
| 154 |
|
| 155 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 156 |
# --------------------------------------------------------------- projects
|
| 157 |
@app.get("/api/projects", response_model=list[ProjectOut])
|
| 158 |
def list_projects(db: Session = Depends(get_db)):
|
| 159 |
-
"""Projects ordered by last activity (latest run, else creation)
|
| 160 |
project a researcher wants is almost always the one they last worked on."""
|
| 161 |
from sqlalchemy import func
|
| 162 |
|
|
@@ -180,6 +220,7 @@ def list_projects(db: Session = Depends(get_db)):
|
|
| 180 |
created_at=p.created_at,
|
| 181 |
last_activity_at=last or p.created_at,
|
| 182 |
n_runs=count,
|
|
|
|
| 183 |
)
|
| 184 |
)
|
| 185 |
out.sort(key=lambda x: x.last_activity_at, reverse=True)
|
|
@@ -198,9 +239,65 @@ def create_project(body: ProjectCreate, db: Session = Depends(get_db)):
|
|
| 198 |
created_at=project.created_at,
|
| 199 |
last_activity_at=project.created_at,
|
| 200 |
n_runs=0,
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 201 |
)
|
| 202 |
|
| 203 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 204 |
# ----------------------------------------------------------------- corpora
|
| 205 |
@app.get("/api/projects/{project_id}/corpora", response_model=list[CorpusOut])
|
| 206 |
def list_corpora(project_id: str, db: Session = Depends(get_db)):
|
|
@@ -227,7 +324,12 @@ def list_corpora(project_id: str, db: Session = Depends(get_db)):
|
|
| 227 |
|
| 228 |
|
| 229 |
@app.post("/api/projects/{project_id}/corpora", response_model=CorpusOut, status_code=201)
|
| 230 |
-
async def upload_corpus(
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 231 |
_get_or_404(db, Project, project_id)
|
| 232 |
|
| 233 |
suffix = Path(file.filename or "upload.csv").suffix.lower()
|
|
@@ -238,6 +340,15 @@ async def upload_corpus(project_id: str, file: UploadFile, db: Session = Depends
|
|
| 238 |
if len(payload) > MAX_UPLOAD_BYTES:
|
| 239 |
raise HTTPException(413, "File exceeds the 25 MB upload limit.")
|
| 240 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 241 |
corpus = Corpus(
|
| 242 |
project_id=project_id, filename=file.filename, path="", n_rows=0, columns_json="[]"
|
| 243 |
)
|
|
@@ -251,6 +362,14 @@ async def upload_corpus(project_id: str, file: UploadFile, db: Session = Depends
|
|
| 251 |
dest.unlink(missing_ok=True)
|
| 252 |
raise HTTPException(400, str(exc)) from exc
|
| 253 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 254 |
corpus.n_rows = int(len(df))
|
| 255 |
corpus.columns_json = json.dumps(list(df.columns))
|
| 256 |
corpus.parse_info_json = json.dumps(parse_info)
|
|
|
|
| 1 |
+
"""CCR Platform - FastAPI application.
|
| 2 |
|
| 3 |
Single deployable: serves the JSON API under /api and the prebuilt React
|
| 4 |
dashboard as static files at /. Local-first by design: corpora, embeddings,
|
|
|
|
| 13 |
from contextlib import asynccontextmanager
|
| 14 |
from pathlib import Path
|
| 15 |
|
| 16 |
+
from fastapi import Depends, FastAPI, HTTPException, Response, UploadFile
|
| 17 |
from fastapi.middleware.cors import CORSMiddleware
|
| 18 |
from fastapi.responses import FileResponse, JSONResponse, PlainTextResponse
|
| 19 |
from fastapi.staticfiles import StaticFiles
|
| 20 |
from sqlalchemy.orm import Session
|
| 21 |
|
| 22 |
+
from . import auth_demo
|
| 23 |
from . import jobs as jobs_module
|
| 24 |
from . import registry
|
| 25 |
from .ccr import FAKE_MODEL_NAME
|
|
|
|
| 32 |
ConstructCreate,
|
| 33 |
ConstructOut,
|
| 34 |
CorpusOut,
|
| 35 |
+
DemoLogin,
|
| 36 |
JobCreate,
|
| 37 |
JobOut,
|
| 38 |
ProjectCreate,
|
| 39 |
ProjectOut,
|
| 40 |
+
ProjectPatch,
|
| 41 |
)
|
| 42 |
|
| 43 |
MAX_UPLOAD_BYTES = 25 * 1024 * 1024 # sane lab-scale ceiling; raise deliberately
|
|
|
|
| 136 |
|
| 137 |
@app.get("/api/models")
|
| 138 |
def list_models():
|
| 139 |
+
"""Model options from the registry (spec 0003) - never hardcoded."""
|
| 140 |
return [
|
| 141 |
{
|
| 142 |
"id": m.id,
|
|
|
|
| 156 |
return SELECTABLE_LANGUAGES
|
| 157 |
|
| 158 |
|
| 159 |
+
# ---------------------------------------------------- demo auth (placeholder)
|
| 160 |
+
@app.get("/api/auth/me")
|
| 161 |
+
def auth_me(user: dict | None = Depends(auth_demo.get_current_user)):
|
| 162 |
+
return {
|
| 163 |
+
"signed_in": user is not None,
|
| 164 |
+
"name": user["name"] if user else None,
|
| 165 |
+
"demo": True,
|
| 166 |
+
"limits": (
|
| 167 |
+
{"max_bytes": MAX_UPLOAD_BYTES, "max_rows": None}
|
| 168 |
+
if user
|
| 169 |
+
else {"max_bytes": auth_demo.anon_max_bytes(), "max_rows": auth_demo.anon_max_rows()}
|
| 170 |
+
),
|
| 171 |
+
}
|
| 172 |
+
|
| 173 |
+
|
| 174 |
+
@app.post("/api/auth/demo/login")
|
| 175 |
+
def demo_login(body: DemoLogin, response: Response):
|
| 176 |
+
"""Placeholder session for tier testing. Replaced by managed auth in Phase 2."""
|
| 177 |
+
name = body.name.strip()
|
| 178 |
+
if not name:
|
| 179 |
+
raise HTTPException(400, "Name cannot be empty.")
|
| 180 |
+
response.set_cookie(
|
| 181 |
+
auth_demo.COOKIE_NAME,
|
| 182 |
+
auth_demo.create_token(name),
|
| 183 |
+
httponly=True,
|
| 184 |
+
samesite="lax",
|
| 185 |
+
max_age=7 * 24 * 3600,
|
| 186 |
+
)
|
| 187 |
+
return {"signed_in": True, "name": name, "demo": True}
|
| 188 |
+
|
| 189 |
+
|
| 190 |
+
@app.post("/api/auth/logout")
|
| 191 |
+
def logout(response: Response):
|
| 192 |
+
response.delete_cookie(auth_demo.COOKIE_NAME)
|
| 193 |
+
return {"signed_in": False}
|
| 194 |
+
|
| 195 |
+
|
| 196 |
# --------------------------------------------------------------- projects
|
| 197 |
@app.get("/api/projects", response_model=list[ProjectOut])
|
| 198 |
def list_projects(db: Session = Depends(get_db)):
|
| 199 |
+
"""Projects ordered by last activity (latest run, else creation) - the
|
| 200 |
project a researcher wants is almost always the one they last worked on."""
|
| 201 |
from sqlalchemy import func
|
| 202 |
|
|
|
|
| 220 |
created_at=p.created_at,
|
| 221 |
last_activity_at=last or p.created_at,
|
| 222 |
n_runs=count,
|
| 223 |
+
archived=bool(p.archived),
|
| 224 |
)
|
| 225 |
)
|
| 226 |
out.sort(key=lambda x: x.last_activity_at, reverse=True)
|
|
|
|
| 239 |
created_at=project.created_at,
|
| 240 |
last_activity_at=project.created_at,
|
| 241 |
n_runs=0,
|
| 242 |
+
archived=False,
|
| 243 |
+
)
|
| 244 |
+
|
| 245 |
+
|
| 246 |
+
@app.patch("/api/projects/{project_id}", response_model=ProjectOut)
|
| 247 |
+
def patch_project(project_id: str, body: ProjectPatch, db: Session = Depends(get_db)):
|
| 248 |
+
"""Archive/unarchive - reversible, no data loss. Archived projects collapse
|
| 249 |
+
into the sidebar's Archived section and keep all datasets and runs."""
|
| 250 |
+
project = _get_or_404(db, Project, project_id)
|
| 251 |
+
if body.archived is not None:
|
| 252 |
+
project.archived = bool(body.archived)
|
| 253 |
+
db.commit()
|
| 254 |
+
return ProjectOut(
|
| 255 |
+
id=project.id,
|
| 256 |
+
name=project.name,
|
| 257 |
+
description=project.description,
|
| 258 |
+
created_at=project.created_at,
|
| 259 |
+
last_activity_at=project.created_at,
|
| 260 |
+
n_runs=0,
|
| 261 |
+
archived=bool(project.archived),
|
| 262 |
)
|
| 263 |
|
| 264 |
|
| 265 |
+
@app.delete("/api/projects/{project_id}", status_code=204)
|
| 266 |
+
def delete_project(project_id: str, db: Session = Depends(get_db)):
|
| 267 |
+
"""Permanent delete: removes the project, its datasets, runs, uploaded
|
| 268 |
+
files, and result files. Logged without retaining any uploaded text
|
| 269 |
+
(design doc §9). Ownership checks arrive with real accounts in Phase 2;
|
| 270 |
+
until then the shared instance is deliberately open."""
|
| 271 |
+
import logging
|
| 272 |
+
|
| 273 |
+
project = _get_or_404(db, Project, project_id)
|
| 274 |
+
corpora = db.query(Corpus).filter_by(project_id=project_id).all()
|
| 275 |
+
jobs = db.query(Job).filter_by(project_id=project_id).all()
|
| 276 |
+
|
| 277 |
+
n_files = 0
|
| 278 |
+
for corpus in corpora:
|
| 279 |
+
if corpus.path and Path(corpus.path).exists():
|
| 280 |
+
Path(corpus.path).unlink(missing_ok=True)
|
| 281 |
+
n_files += 1
|
| 282 |
+
for job in jobs:
|
| 283 |
+
if job.result_path and Path(job.result_path).exists():
|
| 284 |
+
Path(job.result_path).unlink(missing_ok=True)
|
| 285 |
+
n_files += 1
|
| 286 |
+
|
| 287 |
+
for job in jobs:
|
| 288 |
+
db.delete(job)
|
| 289 |
+
for corpus in corpora:
|
| 290 |
+
db.delete(corpus)
|
| 291 |
+
db.delete(project)
|
| 292 |
+
db.commit()
|
| 293 |
+
|
| 294 |
+
logging.getLogger("ccr.projects").info(
|
| 295 |
+
"project deleted: id=%s name=%r corpora=%d runs=%d files_removed=%d",
|
| 296 |
+
project_id, project.name, len(corpora), len(jobs), n_files,
|
| 297 |
+
)
|
| 298 |
+
return Response(status_code=204)
|
| 299 |
+
|
| 300 |
+
|
| 301 |
# ----------------------------------------------------------------- corpora
|
| 302 |
@app.get("/api/projects/{project_id}/corpora", response_model=list[CorpusOut])
|
| 303 |
def list_corpora(project_id: str, db: Session = Depends(get_db)):
|
|
|
|
| 324 |
|
| 325 |
|
| 326 |
@app.post("/api/projects/{project_id}/corpora", response_model=CorpusOut, status_code=201)
|
| 327 |
+
async def upload_corpus(
|
| 328 |
+
project_id: str,
|
| 329 |
+
file: UploadFile,
|
| 330 |
+
db: Session = Depends(get_db),
|
| 331 |
+
user: dict | None = Depends(auth_demo.get_current_user),
|
| 332 |
+
):
|
| 333 |
_get_or_404(db, Project, project_id)
|
| 334 |
|
| 335 |
suffix = Path(file.filename or "upload.csv").suffix.lower()
|
|
|
|
| 340 |
if len(payload) > MAX_UPLOAD_BYTES:
|
| 341 |
raise HTTPException(413, "File exceeds the 25 MB upload limit.")
|
| 342 |
|
| 343 |
+
# Tier gate (design §5.1): anonymous users get strict caps; signing in
|
| 344 |
+
# lifts them. Demo sign-in for now; managed auth replaces it in Phase 2.
|
| 345 |
+
if user is None and len(payload) > auth_demo.anon_max_bytes():
|
| 346 |
+
mb = auth_demo.anon_max_bytes() // (1024 * 1024)
|
| 347 |
+
raise HTTPException(
|
| 348 |
+
413,
|
| 349 |
+
f"Anonymous uploads are limited to {mb} MB. Sign in (top right) to upload larger files.",
|
| 350 |
+
)
|
| 351 |
+
|
| 352 |
corpus = Corpus(
|
| 353 |
project_id=project_id, filename=file.filename, path="", n_rows=0, columns_json="[]"
|
| 354 |
)
|
|
|
|
| 362 |
dest.unlink(missing_ok=True)
|
| 363 |
raise HTTPException(400, str(exc)) from exc
|
| 364 |
|
| 365 |
+
if user is None and len(df) > auth_demo.anon_max_rows():
|
| 366 |
+
dest.unlink(missing_ok=True)
|
| 367 |
+
raise HTTPException(
|
| 368 |
+
400,
|
| 369 |
+
f"Anonymous uploads are limited to {auth_demo.anon_max_rows():,} rows "
|
| 370 |
+
f"(this file has {len(df):,}). Sign in (top right) to upload larger corpora.",
|
| 371 |
+
)
|
| 372 |
+
|
| 373 |
corpus.n_rows = int(len(df))
|
| 374 |
corpus.columns_json = json.dumps(list(df.columns))
|
| 375 |
corpus.parse_info_json = json.dumps(parse_info)
|
backend/app/models.py
CHANGED
|
@@ -2,7 +2,7 @@
|
|
| 2 |
|
| 3 |
IDs are UUID strings (portable across SQLite/Postgres). JSON-ish payloads
|
| 4 |
(column lists, construct items, job metadata/summaries) are stored as JSON
|
| 5 |
-
text
|
| 6 |
"""
|
| 7 |
|
| 8 |
import uuid
|
|
@@ -28,6 +28,7 @@ class Project(Base):
|
|
| 28 |
id: Mapped[str] = mapped_column(String(32), primary_key=True, default=_uuid)
|
| 29 |
name: Mapped[str] = mapped_column(String(200))
|
| 30 |
description: Mapped[str] = mapped_column(Text, default="")
|
|
|
|
| 31 |
created_at: Mapped[str] = mapped_column(String(32), default=_now)
|
| 32 |
|
| 33 |
|
|
|
|
| 2 |
|
| 3 |
IDs are UUID strings (portable across SQLite/Postgres). JSON-ish payloads
|
| 4 |
(column lists, construct items, job metadata/summaries) are stored as JSON
|
| 5 |
+
text - they are read-mostly blobs, not queried relationally.
|
| 6 |
"""
|
| 7 |
|
| 8 |
import uuid
|
|
|
|
| 28 |
id: Mapped[str] = mapped_column(String(32), primary_key=True, default=_uuid)
|
| 29 |
name: Mapped[str] = mapped_column(String(200))
|
| 30 |
description: Mapped[str] = mapped_column(Text, default="")
|
| 31 |
+
archived: Mapped[bool] = mapped_column(Boolean, default=False)
|
| 32 |
created_at: Mapped[str] = mapped_column(String(32), default=_now)
|
| 33 |
|
| 34 |
|
backend/app/registry.py
CHANGED
|
@@ -1,4 +1,4 @@
|
|
| 1 |
-
"""Model registry loader
|
| 2 |
|
| 3 |
Single source of model truth (design doc §13): the UI dropdown, backend validation,
|
| 4 |
prefix handling, language-support warnings, run metadata, and generated reproduction
|
|
@@ -57,7 +57,7 @@ class ModelConfig:
|
|
| 57 |
return None if self.revision in ("PIN_ME", "", None) else self.revision
|
| 58 |
|
| 59 |
def supports_language(self, iso_code: str) -> bool:
|
| 60 |
-
if not self.supported_languages: # unknown coverage
|
| 61 |
return True
|
| 62 |
return iso_code.lower() in self.supported_languages
|
| 63 |
|
|
|
|
| 1 |
+
"""Model registry loader - the app-side reader of packages/model_registry/models.yaml.
|
| 2 |
|
| 3 |
Single source of model truth (design doc §13): the UI dropdown, backend validation,
|
| 4 |
prefix handling, language-support warnings, run metadata, and generated reproduction
|
|
|
|
| 57 |
return None if self.revision in ("PIN_ME", "", None) else self.revision
|
| 58 |
|
| 59 |
def supports_language(self, iso_code: str) -> bool:
|
| 60 |
+
if not self.supported_languages: # unknown coverage - never block/warn on it
|
| 61 |
return True
|
| 62 |
return iso_code.lower() in self.supported_languages
|
| 63 |
|
backend/app/reproducibility.py
CHANGED
|
@@ -1,7 +1,7 @@
|
|
| 1 |
"""Reproduction-script generation (spec 0002, design doc §14).
|
| 2 |
|
| 3 |
-
The generated script is built ONLY from the run's stored metadata
|
| 4 |
-
state
|
| 5 |
input CSV + Python + internet for the (pinned) model download. No platform credentials.
|
| 6 |
"""
|
| 7 |
|
|
@@ -108,7 +108,7 @@ def main(csv_path: str) -> None:
|
|
| 108 |
|
| 109 |
work.to_csv("reproduced_results.csv", index=False)
|
| 110 |
print(f"Wrote reproduced_results.csv ({{len(work)}} rows, {{sims.shape[1]}} items).")
|
| 111 |
-
print("Compare against the platform export
|
| 112 |
|
| 113 |
|
| 114 |
if __name__ == "__main__":
|
|
|
|
| 1 |
"""Reproduction-script generation (spec 0002, design doc §14).
|
| 2 |
|
| 3 |
+
The generated script is built ONLY from the run's stored metadata - never from live
|
| 4 |
+
state - so it reproduces what actually ran. It must be runnable outside the platform:
|
| 5 |
input CSV + Python + internet for the (pinned) model download. No platform credentials.
|
| 6 |
"""
|
| 7 |
|
|
|
|
| 108 |
|
| 109 |
work.to_csv("reproduced_results.csv", index=False)
|
| 110 |
print(f"Wrote reproduced_results.csv ({{len(work)}} rows, {{sims.shape[1]}} items).")
|
| 111 |
+
print("Compare against the platform export - values should match to ~1e-5.")
|
| 112 |
|
| 113 |
|
| 114 |
if __name__ == "__main__":
|
backend/app/schemas.py
CHANGED
|
@@ -15,6 +15,15 @@ class ProjectOut(BaseModel):
|
|
| 15 |
created_at: str
|
| 16 |
last_activity_at: str = "" # latest run creation, else project creation
|
| 17 |
n_runs: int = 0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 18 |
|
| 19 |
|
| 20 |
class CorpusOut(BaseModel):
|
|
|
|
| 15 |
created_at: str
|
| 16 |
last_activity_at: str = "" # latest run creation, else project creation
|
| 17 |
n_runs: int = 0
|
| 18 |
+
archived: bool = False
|
| 19 |
+
|
| 20 |
+
|
| 21 |
+
class ProjectPatch(BaseModel):
|
| 22 |
+
archived: bool | None = None
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
class DemoLogin(BaseModel):
|
| 26 |
+
name: str = Field(min_length=1, max_length=80)
|
| 27 |
|
| 28 |
|
| 29 |
class CorpusOut(BaseModel):
|
backend/app/warnings_engine.py
CHANGED
|
@@ -1,12 +1,12 @@
|
|
| 1 |
"""Structured data-quality warnings (spec 0001, design doc §12).
|
| 2 |
|
| 3 |
-
Every warning is an object
|
| 4 |
never a bare string. Codes are UPPER_SNAKE and stable: downstream notebooks and the UI
|
| 5 |
key off them. Severity: "info" (status, not a problem) | "warning" (proceed with care).
|
| 6 |
Language detection is corpus-level only; short texts are exactly where detection is
|
| 7 |
unreliable, so uncertainty is reported instead of guessed away.
|
| 8 |
|
| 9 |
-
Deviation from spec 0001 (recorded there): langdetect instead of lingua
|
| 10 |
~1 MB vs ~100 MB wheels; seeded for determinism. Upgrade path preserved by recording
|
| 11 |
detector + version in metadata.
|
| 12 |
"""
|
|
@@ -75,7 +75,7 @@ def detect_corpus_language(texts: list[str], selected: str) -> tuple[LanguageRes
|
|
| 75 |
except Exception:
|
| 76 |
detector_version = "unknown"
|
| 77 |
|
| 78 |
-
DetectorFactory.seed = 0 # determinism
|
| 79 |
|
| 80 |
detectable = [t for t in texts if len(t.split()) >= DETECT_MIN_TOKENS][:DETECT_SAMPLE_MAX]
|
| 81 |
warnings: list[dict] = []
|
|
|
|
| 1 |
"""Structured data-quality warnings (spec 0001, design doc §12).
|
| 2 |
|
| 3 |
+
Every warning is an object - {code, severity, message, count?, affected_rows_sample?} -
|
| 4 |
never a bare string. Codes are UPPER_SNAKE and stable: downstream notebooks and the UI
|
| 5 |
key off them. Severity: "info" (status, not a problem) | "warning" (proceed with care).
|
| 6 |
Language detection is corpus-level only; short texts are exactly where detection is
|
| 7 |
unreliable, so uncertainty is reported instead of guessed away.
|
| 8 |
|
| 9 |
+
Deviation from spec 0001 (recorded there): langdetect instead of lingua - pure-Python,
|
| 10 |
~1 MB vs ~100 MB wheels; seeded for determinism. Upgrade path preserved by recording
|
| 11 |
detector + version in metadata.
|
| 12 |
"""
|
|
|
|
| 75 |
except Exception:
|
| 76 |
detector_version = "unknown"
|
| 77 |
|
| 78 |
+
DetectorFactory.seed = 0 # determinism - same corpus, same result, every run
|
| 79 |
|
| 80 |
detectable = [t for t in texts if len(t.split()) >= DETECT_MIN_TOKENS][:DETECT_SAMPLE_MAX]
|
| 81 |
warnings: list[dict] = []
|
backend/static/assets/index-CHG17tHs.js
DELETED
|
The diff for this file is too large to render.
See raw diff
|
|
|
backend/static/assets/{index-BDrO2FRr.css → index-Cu39pU_-.css}
RENAMED
|
@@ -1 +1 @@
|
|
| 1 |
-
:root{--maroon: #7a1f3d;--maroon-dark: #5e1730;--ink: #1d2129;--muted: #667085;--line: #e5e7eb;--bg: #f7f7f8;--card: #ffffff;--ok: #157f3d;--err: #b42318;--accent-soft: #f6ebef;--control-height: 40px}*{box-sizing:border-box}body{margin:0;font-family:-apple-system,BlinkMacSystemFont,Segoe UI,Roboto,Helvetica Neue,sans-serif;color:var(--ink);background:var(--bg);font-size:14.5px;line-height:1.5}.app{display:flex;flex-direction:column;min-height:100vh}.header{background:var(--maroon);color:#fff;padding:14px 28px;display:flex;align-items:baseline;flex-wrap:wrap;gap:14px}.header h1{flex:0 0 auto;font-size:17px;margin:0;font-weight:650;letter-spacing:.2px;white-space:nowrap}.header .sub{flex:1 1 280px;min-width:0;font-size:12.5px;opacity:.85}.layout{display:flex;flex:1;min-width:0;min-height:0}.sidebar{width:250px;background:var(--card);border-right:1px solid var(--line);padding:18px 14px;flex-shrink:0;display:flex;flex-direction:column;min-height:0}.sidebar h2{font-size:11.5px;text-transform:uppercase;letter-spacing:.7px;color:var(--muted);margin:0 0 10px 4px;display:flex;align-items:center;gap:8px}.sidebar h2 .count{background:var(--bg);border:1px solid var(--line);border-radius:999px;padding:0 8px;font-size:10.5px;letter-spacing:0;color:var(--muted)}.sidebar-filter{width:100%;padding:7px 10px;margin-bottom:10px;border:1px solid var(--line);border-radius:8px;font:inherit;font-size:13px;background:#fff;color:var(--ink)}.sidebar-filter:focus{outline:none;border-color:var(--maroon)}.project-list{flex:1;min-height:0;overflow-y:auto;margin:0 -4px;padding:0 4px 4px}.group-label{font-size:10.5px;text-transform:uppercase;letter-spacing:.6px;color:var(--muted);margin:10px 4px 5px}div:first-child>.group-label{margin-top:2px}.project-item{display:block;width:100%;text-align:left;padding:9px 12px;margin-bottom:5px;border:1px solid transparent;border-radius:8px;background:none;cursor:pointer;font:inherit;color:var(--ink)}.project-item:hover{background:var(--bg)}.project-item.active{background:var(--accent-soft);border-color:var(--maroon);font-weight:600}.project-item .project-name{display:block;overflow:hidden;text-overflow:ellipsis;white-space:nowrap}.project-item .date{display:block;font-size:11.5px;color:var(--muted);font-weight:400}.project-create{margin-top:12px;padding-top:12px;border-top:1px solid var(--line)}.project-create>button{width:100%}.main{flex:1;padding:22px 28px;overflow-y:auto;min-width:0}.card{background:var(--card);border:1px solid var(--line);border-radius:8px;padding:18px 20px;margin-bottom:16px}.card h3{margin:0 0 4px;font-size:15px}.card .hint{color:var(--muted);font-size:12.5px;margin:0 0 12px}.step-badge{display:inline-flex;align-items:center;justify-content:center;width:21px;height:21px;border-radius:50%;background:var(--maroon);color:#fff;font-size:12px;font-weight:700;margin-right:8px;vertical-align:-3px}button{white-space:nowrap}button.primary{background:var(--maroon);color:#fff;border:none;min-height:var(--control-height);display:inline-flex;align-items:center;justify-content:center;padding:0 18px;border-radius:8px;font:inherit;font-weight:600;cursor:pointer;line-height:1.2}button.primary:hover{background:var(--maroon-dark)}button.primary:disabled{background:#c9ccd1;cursor:not-allowed}button.ghost{background:none;border:1px solid var(--line);color:var(--ink);min-height:var(--control-height);display:inline-flex;align-items:center;justify-content:center;padding:0 14px;border-radius:8px;font:inherit;cursor:pointer;line-height:1.2}button.ghost:hover{border-color:var(--maroon);color:var(--maroon)}button.linkish{background:none;border:none;color:var(--maroon);font:inherit;cursor:pointer;padding:0;text-decoration:underline}input[type=text],textarea,select{width:100%;padding:8px 10px;border:1px solid var(--line);border-radius:8px;font:inherit;background:#fff;color:var(--ink)}input[type=text],select{height:var(--control-height)}button:focus-visible,input:focus-visible,textarea:focus-visible,select:focus-visible{outline:2px solid rgba(122,31,61,.38);outline-offset:2px}input[type=file]{display:block;max-width:100%;margin-top:6px;font-size:13px;color:var(--muted)}input[type=file]::file-selector-button{background:#fff;border:1px solid var(--line);color:var(--ink);padding:7px 14px;border-radius:8px;font:inherit;font-size:13px;cursor:pointer;margin-right:10px}input[type=file]::file-selector-button:hover{border-color:var(--maroon);color:var(--maroon)}.row>button{align-self:flex-end;margin-bottom:1px}textarea{resize:vertical}label.field{display:block;margin-bottom:10px;font-size:13px;font-weight:600}label.field>*{margin-top:4px;font-weight:400}.row{display:flex;gap:14px;flex-wrap:wrap}.row>*{min-width:0}.row>.grow{flex:1;min-width:min(220px,100%)}.language-control{min-width:170px}.model-control{min-width:260px}.run-settings{display:grid;grid-template-columns:minmax(160px,230px) minmax(320px,720px) max-content;justify-content:start;gap:14px;align-items:end}.run-settings .field{margin-bottom:0}.run-button{min-width:180px;height:var(--control-height)}.construct-row{display:grid;grid-template-columns:minmax(320px,1120px) max-content;justify-content:start;align-items:end;gap:14px}.results-toolbar{display:flex;align-items:center;justify-content:space-between;gap:14px;margin-bottom:14px;flex-wrap:wrap}.result-actions{justify-content:flex-end}.result-actions a{display:inline-flex;text-decoration:none}.pill{display:inline-block;padding:2px 10px;border-radius:999px;font-size:11.5px;font-weight:600}.pill.completed{background:#e6f4ea;color:var(--ok)}.pill.running{background:#fff3e0;color:#b45309}.pill.queued{background:#eef2f7;color:var(--muted)}.pill.failed{background:#fdecea;color:var(--err)}.progress-track{background:var(--line);border-radius:999px;height:7px;overflow:hidden}.progress-fill{background:var(--maroon);height:100%;transition:width .4s ease}.warnings{background:#fff8e6;border:1px solid #f2dfa8;color:#7a5b00;border-radius:8px;padding:10px 14px}.error-banner{background:#fdecea;color:var(--err);border:1px solid #f5c6c0;padding:10px 14px;border-radius:8px;margin-bottom:14px;font-size:13px}.table-wrap{width:100%;overflow-x:auto}table.docs{width:100%;border-collapse:collapse;font-size:13px}table.docs th{text-align:left;color:var(--muted);font-size:11.5px;text-transform:uppercase;letter-spacing:.5px;padding:6px 8px;border-bottom:1px solid var(--line)}table.docs td{padding:7px 8px;border-bottom:1px solid var(--bg);vertical-align:top;overflow-wrap:anywhere}table.docs td.score{font-variant-numeric:tabular-nums;font-weight:600;white-space:nowrap}.stat-grid{display:flex;gap:12px;flex-wrap:wrap;margin-bottom:4px}.stat{flex:1;min-width:110px;background:var(--bg);border-radius:8px;padding:10px 14px}.stat .v{font-size:20px;font-weight:700;font-variant-numeric:tabular-nums}.stat .k{font-size:11.5px;color:var(--muted);text-transform:uppercase;letter-spacing:.5px}.item-bar-row{display:flex;align-items:center;gap:10px;margin-bottom:7px}.item-bar-label{flex:1;font-size:12.5px;min-width:0;overflow-wrap:anywhere}.item-bar-track{flex:1.2;background:var(--bg);border-radius:999px;height:10px}.item-bar-fill{background:var(--maroon);opacity:.85;height:100%;border-radius:999px}.item-bar-val{width:52px;text-align:right;font-variant-numeric:tabular-nums;font-size:12.5px;font-weight:600}.construct-items{margin:8px 0 0;padding-left:20px;color:var(--muted);font-size:12.5px}.construct-items li{margin-bottom:2px}.meta-footer{font-size:12px;color:var(--muted);background:var(--bg);border-radius:8px;padding:10px 14px;margin-top:14px;font-variant-numeric:tabular-nums}.meta-footer code{font-size:11.5px}.muted{color:var(--muted)}.small{font-size:12.5px}.mt{margin-top:12px}@media (max-width: 820px){body{font-size:14px}.header{padding:12px 16px;align-items:flex-start;gap:2px 12px}.header h1{font-size:16px}.header .sub{flex:1 1 210px;font-size:12px;line-height:1.35}.layout{display:block}.sidebar{width:100%;border-right:0;border-bottom:1px solid var(--line);padding:14px}.project-list{max-height:220px;margin-right:0;flex:none}.project-create{border-top:0}.main{width:100%;padding:16px 14px 28px;overflow:visible}.card{padding:16px;margin-bottom:14px}.row,.run-settings,.construct-row,.results-toolbar,.result-actions{gap:10px}.row,.results-toolbar,.result-actions{flex-direction:column;align-items:stretch}.run-settings,.construct-row{grid-template-columns:1fr}.row>.grow,.construct-row>.grow,.language-control,.model-control{width:100%;min-width:0}.main .row>button,.main .row>a,.main .row>a>button,.results-toolbar>button{align-self:stretch;margin-bottom:0;width:100%}.run-button{width:100%;min-width:0}.stat-grid{display:grid;grid-template-columns:repeat(2,minmax(0,1fr))}.stat{min-width:0}.item-bar-row{display:grid;grid-template-columns:minmax(0,1fr) 56px;gap:6px 10px}.item-bar-track{grid-column:1 / -1;width:100%}.item-bar-val{width:auto}table.docs{min-width:520px}}@media (max-width: 460px){.header .sub{flex-basis:100%}.stat-grid{grid-template-columns:1fr}}
|
|
|
|
| 1 |
+
:root{--maroon: #7a1f3d;--maroon-dark: #5e1730;--ink: #1d2129;--muted: #667085;--line: #e5e7eb;--bg: #f7f7f8;--card: #ffffff;--ok: #157f3d;--err: #b42318;--accent-soft: #f6ebef;--control-height: 40px}*{box-sizing:border-box}body{margin:0;font-family:-apple-system,BlinkMacSystemFont,Segoe UI,Roboto,Helvetica Neue,sans-serif;color:var(--ink);background:var(--bg);font-size:14.5px;line-height:1.5}.app{display:flex;flex-direction:column;min-height:100vh}.header{background:var(--maroon);color:#fff;padding:14px 28px;display:flex;align-items:baseline;flex-wrap:wrap;gap:14px}.header h1{flex:0 0 auto;font-size:17px;margin:0;font-weight:650;letter-spacing:.2px;white-space:nowrap}.header .sub{flex:1 1 280px;min-width:0;font-size:12.5px;opacity:.85}.layout{display:flex;flex:1;min-width:0;min-height:0}.sidebar{width:250px;background:var(--card);border-right:1px solid var(--line);padding:18px 14px;flex-shrink:0;display:flex;flex-direction:column;min-height:0}.sidebar h2{font-size:11.5px;text-transform:uppercase;letter-spacing:.7px;color:var(--muted);margin:0 0 10px 4px;display:flex;align-items:center;gap:8px}.sidebar h2 .count{background:var(--bg);border:1px solid var(--line);border-radius:999px;padding:0 8px;font-size:10.5px;letter-spacing:0;color:var(--muted)}.sidebar-filter{width:100%;padding:7px 10px;margin-bottom:10px;border:1px solid var(--line);border-radius:8px;font:inherit;font-size:13px;background:#fff;color:var(--ink)}.sidebar-filter:focus{outline:none;border-color:var(--maroon)}.project-list{flex:1;min-height:0;overflow-y:auto;margin:0 -4px;padding:0 4px 4px}.group-label{font-size:10.5px;text-transform:uppercase;letter-spacing:.6px;color:var(--muted);margin:10px 4px 5px}div:first-child>.group-label{margin-top:2px}.project-item{display:block;width:100%;text-align:left;padding:9px 12px;margin-bottom:5px;border:1px solid transparent;border-radius:8px;background:none;cursor:pointer;font:inherit;color:var(--ink)}.project-item:hover{background:var(--bg)}.project-item.active{background:var(--accent-soft);border-color:var(--maroon);font-weight:600}.project-item .project-name{display:block;overflow:hidden;text-overflow:ellipsis;white-space:nowrap}.project-item .date{display:block;font-size:11.5px;color:var(--muted);font-weight:400}.project-create{margin-top:12px;padding-top:12px;border-top:1px solid var(--line)}.project-create>button{width:100%}.main{flex:1;padding:22px 28px;overflow-y:auto;min-width:0}.card{background:var(--card);border:1px solid var(--line);border-radius:8px;padding:18px 20px;margin-bottom:16px}.card h3{margin:0 0 4px;font-size:15px}.card .hint{color:var(--muted);font-size:12.5px;margin:0 0 12px}.step-badge{display:inline-flex;align-items:center;justify-content:center;width:21px;height:21px;border-radius:50%;background:var(--maroon);color:#fff;font-size:12px;font-weight:700;margin-right:8px;vertical-align:-3px}button{white-space:nowrap}button.primary{background:var(--maroon);color:#fff;border:none;min-height:var(--control-height);display:inline-flex;align-items:center;justify-content:center;padding:0 18px;border-radius:8px;font:inherit;font-weight:600;cursor:pointer;line-height:1.2}button.primary:hover{background:var(--maroon-dark)}button.primary:disabled{background:#c9ccd1;cursor:not-allowed}button.ghost{background:none;border:1px solid var(--line);color:var(--ink);min-height:var(--control-height);display:inline-flex;align-items:center;justify-content:center;padding:0 14px;border-radius:8px;font:inherit;cursor:pointer;line-height:1.2}button.ghost:hover{border-color:var(--maroon);color:var(--maroon)}button.linkish{background:none;border:none;color:var(--maroon);font:inherit;cursor:pointer;padding:0;text-decoration:underline}input[type=text],textarea,select{width:100%;padding:8px 10px;border:1px solid var(--line);border-radius:8px;font:inherit;background:#fff;color:var(--ink)}input[type=text],select{height:var(--control-height)}button:focus-visible,input:focus-visible,textarea:focus-visible,select:focus-visible{outline:2px solid rgba(122,31,61,.38);outline-offset:2px}input[type=file]{display:block;max-width:100%;margin-top:6px;font-size:13px;color:var(--muted)}input[type=file]::file-selector-button{background:#fff;border:1px solid var(--line);color:var(--ink);padding:7px 14px;border-radius:8px;font:inherit;font-size:13px;cursor:pointer;margin-right:10px}input[type=file]::file-selector-button:hover{border-color:var(--maroon);color:var(--maroon)}.row>button{align-self:flex-end;margin-bottom:1px}textarea{resize:vertical}label.field{display:block;margin-bottom:10px;font-size:13px;font-weight:600}label.field>*{margin-top:4px;font-weight:400}.row{display:flex;gap:14px;flex-wrap:wrap}.row>*{min-width:0}.row>.grow{flex:1;min-width:min(220px,100%)}.language-control{min-width:170px}.model-control{min-width:260px}.run-settings{display:grid;grid-template-columns:minmax(160px,230px) minmax(320px,720px) max-content;justify-content:start;gap:14px;align-items:end}.run-settings .field{margin-bottom:0}.run-button{min-width:180px;height:var(--control-height)}.construct-row{display:grid;grid-template-columns:minmax(320px,1120px) max-content;justify-content:start;align-items:end;gap:14px}.results-toolbar{display:flex;align-items:center;justify-content:space-between;gap:14px;margin-bottom:14px;flex-wrap:wrap}.result-actions{justify-content:flex-end}.result-actions a{display:inline-flex;text-decoration:none}.pill{display:inline-block;padding:2px 10px;border-radius:999px;font-size:11.5px;font-weight:600}.pill.completed{background:#e6f4ea;color:var(--ok)}.pill.running{background:#fff3e0;color:#b45309}.pill.queued{background:#eef2f7;color:var(--muted)}.pill.failed{background:#fdecea;color:var(--err)}.progress-track{background:var(--line);border-radius:999px;height:7px;overflow:hidden}.progress-fill{background:var(--maroon);height:100%;transition:width .4s ease}.warnings{background:#fff8e6;border:1px solid #f2dfa8;color:#7a5b00;border-radius:8px;padding:10px 14px}.error-banner{background:#fdecea;color:var(--err);border:1px solid #f5c6c0;padding:10px 14px;border-radius:8px;margin-bottom:14px;font-size:13px}.table-wrap{width:100%;overflow-x:auto}table.docs{width:100%;border-collapse:collapse;font-size:13px}table.docs th{text-align:left;color:var(--muted);font-size:11.5px;text-transform:uppercase;letter-spacing:.5px;padding:6px 8px;border-bottom:1px solid var(--line)}table.docs td{padding:7px 8px;border-bottom:1px solid var(--bg);vertical-align:top;overflow-wrap:anywhere}table.docs td.score{font-variant-numeric:tabular-nums;font-weight:600;white-space:nowrap}.stat-grid{display:flex;gap:12px;flex-wrap:wrap;margin-bottom:4px}.stat{flex:1;min-width:110px;background:var(--bg);border-radius:8px;padding:10px 14px}.stat .v{font-size:20px;font-weight:700;font-variant-numeric:tabular-nums}.stat .k{font-size:11.5px;color:var(--muted);text-transform:uppercase;letter-spacing:.5px}.item-bar-row{display:flex;align-items:center;gap:10px;margin-bottom:7px}.item-bar-label{flex:1;font-size:12.5px;min-width:0;overflow-wrap:anywhere}.item-bar-track{flex:1.2;background:var(--bg);border-radius:999px;height:10px}.item-bar-fill{background:var(--maroon);opacity:.85;height:100%;border-radius:999px}.item-bar-val{width:52px;text-align:right;font-variant-numeric:tabular-nums;font-size:12.5px;font-weight:600}.construct-items{margin:8px 0 0;padding-left:20px;color:var(--muted);font-size:12.5px}.construct-items li{margin-bottom:2px}.meta-footer{font-size:12px;color:var(--muted);background:var(--bg);border-radius:8px;padding:10px 14px;margin-top:14px;font-variant-numeric:tabular-nums}.meta-footer code{font-size:11.5px}.muted{color:var(--muted)}.small{font-size:12.5px}.mt{margin-top:12px}.header-auth{margin-left:auto;display:flex;align-items:center;gap:10px;color:#fff}.header-btn{background:#ffffff1f;color:#fff;border:1px solid rgba(255,255,255,.45);padding:5px 14px;border-radius:7px;font:inherit;font-size:13px;cursor:pointer}.header-btn:hover{background:#ffffff38}.project-header{display:flex;align-items:center;justify-content:space-between;gap:12px;margin-bottom:14px;flex-wrap:wrap}.project-title{font-size:17px;font-weight:650;margin-right:10px}button.danger{color:var(--err);border-color:#f0c4be}button.danger:hover{color:var(--err);border-color:var(--err)}button.danger-solid{background:var(--err)}button.danger-solid:hover{background:#93261b}button.danger-solid:disabled{background:#c9ccd1}.modal-backdrop{position:fixed;top:0;right:0;bottom:0;left:0;background:#14161a73;display:flex;align-items:center;justify-content:center;z-index:40;padding:16px}.modal{background:var(--card);border-radius:10px;padding:22px 24px;width:100%;max-width:430px;box-shadow:0 12px 40px #0000002e}.modal h3{margin:0 0 6px}@media (max-width: 820px){body{font-size:14px}.header{padding:12px 16px;align-items:flex-start;gap:2px 12px}.header h1{font-size:16px}.header .sub{flex:1 1 210px;font-size:12px;line-height:1.35}.layout{display:block}.sidebar{width:100%;border-right:0;border-bottom:1px solid var(--line);padding:14px}.project-list{max-height:220px;margin-right:0;flex:none}.project-create{border-top:0}.main{width:100%;padding:16px 14px 28px;overflow:visible}.card{padding:16px;margin-bottom:14px}.row,.run-settings,.construct-row,.results-toolbar,.result-actions{gap:10px}.row,.results-toolbar,.result-actions{flex-direction:column;align-items:stretch}.run-settings,.construct-row{grid-template-columns:1fr}.row>.grow,.construct-row>.grow,.language-control,.model-control{width:100%;min-width:0}.main .row>button,.main .row>a,.main .row>a>button,.results-toolbar>button{align-self:stretch;margin-bottom:0;width:100%}.run-button{width:100%;min-width:0}.stat-grid{display:grid;grid-template-columns:repeat(2,minmax(0,1fr))}.stat{min-width:0}.item-bar-row{display:grid;grid-template-columns:minmax(0,1fr) 56px;gap:6px 10px}.item-bar-track{grid-column:1 / -1;width:100%}.item-bar-val{width:auto}table.docs{min-width:520px}}@media (max-width: 460px){.header .sub{flex-basis:100%}.stat-grid{grid-template-columns:1fr}}
|
backend/static/assets/index-DOOkGzjl.js
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
backend/static/index.html
CHANGED
|
@@ -4,8 +4,8 @@
|
|
| 4 |
<meta charset="UTF-8" />
|
| 5 |
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
| 6 |
<title>CCR Platform — Contextualized Construct Representations</title>
|
| 7 |
-
<script type="module" crossorigin src="/assets/index-
|
| 8 |
-
<link rel="stylesheet" crossorigin href="/assets/index-
|
| 9 |
</head>
|
| 10 |
<body>
|
| 11 |
<div id="root"></div>
|
|
|
|
| 4 |
<meta charset="UTF-8" />
|
| 5 |
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
| 6 |
<title>CCR Platform — Contextualized Construct Representations</title>
|
| 7 |
+
<script type="module" crossorigin src="/assets/index-DOOkGzjl.js"></script>
|
| 8 |
+
<link rel="stylesheet" crossorigin href="/assets/index-Cu39pU_-.css">
|
| 9 |
</head>
|
| 10 |
<body>
|
| 11 |
<div id="root"></div>
|
backend/tests/test_api.py
CHANGED
|
@@ -1,8 +1,8 @@
|
|
| 1 |
"""End-to-end API tests: project -> upload -> job -> results -> export.
|
| 2 |
|
| 3 |
Run against the fake embedding backend (model_name = 'fake-deterministic'),
|
| 4 |
-
exercising the full pipeline
|
| 5 |
-
(worker thread + polling), results summary, warnings, and export shape
|
| 6 |
without ML dependencies.
|
| 7 |
"""
|
| 8 |
|
|
@@ -81,7 +81,7 @@ def test_health(client):
|
|
| 81 |
def test_seed_constructs_present(client):
|
| 82 |
names = {c["name"] for c in client.get("/api/constructs").json()}
|
| 83 |
assert "Satisfaction with Life" in names
|
| 84 |
-
assert "Moral Foundations
|
| 85 |
|
| 86 |
|
| 87 |
def test_corpus_upload_parses_columns(flow):
|
|
|
|
| 1 |
"""End-to-end API tests: project -> upload -> job -> results -> export.
|
| 2 |
|
| 3 |
Run against the fake embedding backend (model_name = 'fake-deterministic'),
|
| 4 |
+
exercising the full pipeline - tolerant upload parsing, the real job queue
|
| 5 |
+
(worker thread + polling), results summary, warnings, and export shape -
|
| 6 |
without ML dependencies.
|
| 7 |
"""
|
| 8 |
|
|
|
|
| 81 |
def test_seed_constructs_present(client):
|
| 82 |
names = {c["name"] for c in client.get("/api/constructs").json()}
|
| 83 |
assert "Satisfaction with Life" in names
|
| 84 |
+
assert "Moral Foundations - Care" in names
|
| 85 |
|
| 86 |
|
| 87 |
def test_corpus_upload_parses_columns(flow):
|
backend/tests/test_auth_tiers_and_lifecycle.py
ADDED
|
@@ -0,0 +1,134 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Demo sign-in tiers (anonymous caps lift on sign-in) and project lifecycle
|
| 2 |
+
(archive is reversible; delete cascades to files and rows)."""
|
| 3 |
+
|
| 4 |
+
import io
|
| 5 |
+
import time
|
| 6 |
+
from pathlib import Path
|
| 7 |
+
|
| 8 |
+
import pytest
|
| 9 |
+
from fastapi.testclient import TestClient
|
| 10 |
+
|
| 11 |
+
from app.main import app
|
| 12 |
+
|
| 13 |
+
|
| 14 |
+
def upload(client, project_id, name, payload: bytes):
|
| 15 |
+
return client.post(
|
| 16 |
+
f"/api/projects/{project_id}/corpora",
|
| 17 |
+
files={"file": (name, io.BytesIO(payload), "text/csv")},
|
| 18 |
+
)
|
| 19 |
+
|
| 20 |
+
|
| 21 |
+
def wait_for_job(client, job_id, timeout=10.0):
|
| 22 |
+
deadline = time.time() + timeout
|
| 23 |
+
while time.time() < deadline:
|
| 24 |
+
job = client.get(f"/api/jobs/{job_id}").json()
|
| 25 |
+
if job["status"] in ("completed", "failed"):
|
| 26 |
+
return job
|
| 27 |
+
time.sleep(0.05)
|
| 28 |
+
raise TimeoutError(job_id)
|
| 29 |
+
|
| 30 |
+
|
| 31 |
+
@pytest.fixture()
|
| 32 |
+
def client():
|
| 33 |
+
with TestClient(app) as c:
|
| 34 |
+
yield c
|
| 35 |
+
|
| 36 |
+
|
| 37 |
+
def csv_rows(n: int) -> bytes:
|
| 38 |
+
return ("text\n" + "\n".join(f"sample sentence number {i} here" for i in range(n))).encode()
|
| 39 |
+
|
| 40 |
+
|
| 41 |
+
# ------------------------------------------------------------------- tiers
|
| 42 |
+
def test_anonymous_row_cap_and_signin_lifts_it(client, monkeypatch):
|
| 43 |
+
monkeypatch.setenv("CCR_ANON_MAX_ROWS", "5")
|
| 44 |
+
project = client.post("/api/projects", json={"name": "Tiers"}).json()
|
| 45 |
+
|
| 46 |
+
me = client.get("/api/auth/me").json()
|
| 47 |
+
assert me["signed_in"] is False and me["limits"]["max_rows"] == 5
|
| 48 |
+
|
| 49 |
+
resp = upload(client, project["id"], "big.csv", csv_rows(10))
|
| 50 |
+
assert resp.status_code == 400
|
| 51 |
+
assert "Sign in" in resp.json()["detail"]
|
| 52 |
+
|
| 53 |
+
assert client.post("/api/auth/demo/login", json={"name": "Deva"}).status_code == 200
|
| 54 |
+
me = client.get("/api/auth/me").json()
|
| 55 |
+
assert me["signed_in"] is True and me["name"] == "Deva"
|
| 56 |
+
|
| 57 |
+
resp = upload(client, project["id"], "big.csv", csv_rows(10))
|
| 58 |
+
assert resp.status_code == 201, resp.json()
|
| 59 |
+
|
| 60 |
+
client.post("/api/auth/logout")
|
| 61 |
+
assert client.get("/api/auth/me").json()["signed_in"] is False
|
| 62 |
+
|
| 63 |
+
|
| 64 |
+
def test_anonymous_size_cap(client, monkeypatch):
|
| 65 |
+
monkeypatch.setenv("CCR_ANON_MAX_BYTES", "200")
|
| 66 |
+
project = client.post("/api/projects", json={"name": "SizeCap"}).json()
|
| 67 |
+
resp = upload(client, project["id"], "big.csv", csv_rows(50))
|
| 68 |
+
assert resp.status_code == 413
|
| 69 |
+
assert "Sign in" in resp.json()["detail"]
|
| 70 |
+
|
| 71 |
+
|
| 72 |
+
def test_tampered_session_cookie_is_anonymous(client):
|
| 73 |
+
client.cookies.set("ccr_demo_session", "aGFja2Vy.badsignature")
|
| 74 |
+
assert client.get("/api/auth/me").json()["signed_in"] is False
|
| 75 |
+
|
| 76 |
+
|
| 77 |
+
# --------------------------------------------------------------- lifecycle
|
| 78 |
+
def test_archive_toggle_is_reversible(client):
|
| 79 |
+
project = client.post("/api/projects", json={"name": "Archivable"}).json()
|
| 80 |
+
assert project["archived"] is False
|
| 81 |
+
|
| 82 |
+
patched = client.patch(f"/api/projects/{project['id']}", json={"archived": True}).json()
|
| 83 |
+
assert patched["archived"] is True
|
| 84 |
+
listed = next(p for p in client.get("/api/projects").json() if p["id"] == project["id"])
|
| 85 |
+
assert listed["archived"] is True
|
| 86 |
+
|
| 87 |
+
patched = client.patch(f"/api/projects/{project['id']}", json={"archived": False}).json()
|
| 88 |
+
assert patched["archived"] is False
|
| 89 |
+
|
| 90 |
+
|
| 91 |
+
def test_delete_cascades_rows_and_files(client):
|
| 92 |
+
project = client.post("/api/projects", json={"name": "Doomed"}).json()
|
| 93 |
+
corpus = upload(client, project["id"], "corpus.csv", csv_rows(6)).json()
|
| 94 |
+
|
| 95 |
+
constructs = client.get("/api/constructs").json()
|
| 96 |
+
swls = next(c for c in constructs if c["name"] == "Satisfaction with Life")
|
| 97 |
+
job = client.post(
|
| 98 |
+
"/api/jobs",
|
| 99 |
+
json={
|
| 100 |
+
"project_id": project["id"],
|
| 101 |
+
"corpus_id": corpus["id"],
|
| 102 |
+
"construct_id": swls["id"],
|
| 103 |
+
"text_column": "text",
|
| 104 |
+
"model_name": "fake-deterministic",
|
| 105 |
+
},
|
| 106 |
+
).json()
|
| 107 |
+
job = wait_for_job(client, job["id"])
|
| 108 |
+
assert job["status"] == "completed"
|
| 109 |
+
|
| 110 |
+
# capture file paths before deletion
|
| 111 |
+
results = client.get(f"/api/jobs/{job['id']}/results")
|
| 112 |
+
assert results.status_code == 200
|
| 113 |
+
|
| 114 |
+
resp = client.delete(f"/api/projects/{project['id']}")
|
| 115 |
+
assert resp.status_code == 204
|
| 116 |
+
|
| 117 |
+
assert client.get(f"/api/jobs/{job['id']}").status_code == 404
|
| 118 |
+
assert all(p["id"] != project["id"] for p in client.get("/api/projects").json())
|
| 119 |
+
# corpora listing for the deleted project 404s
|
| 120 |
+
assert client.get(f"/api/projects/{project['id']}/corpora").status_code == 404
|
| 121 |
+
|
| 122 |
+
|
| 123 |
+
def test_delete_removes_files_on_disk(client, tmp_path):
|
| 124 |
+
import os
|
| 125 |
+
|
| 126 |
+
data_dir = Path(os.environ["CCR_DATA_DIR"])
|
| 127 |
+
project = client.post("/api/projects", json={"name": "FileCheck"}).json()
|
| 128 |
+
before = set((data_dir / "corpora").glob("*"))
|
| 129 |
+
upload(client, project["id"], "corpus.csv", csv_rows(6))
|
| 130 |
+
created = set((data_dir / "corpora").glob("*")) - before
|
| 131 |
+
assert len(created) == 1
|
| 132 |
+
|
| 133 |
+
client.delete(f"/api/projects/{project['id']}")
|
| 134 |
+
assert not created.pop().exists()
|
backend/tests/test_ccr.py
CHANGED
|
@@ -1,4 +1,4 @@
|
|
| 1 |
-
"""Unit tests for the CCR engine (deterministic fake backend
|
| 2 |
|
| 3 |
import numpy as np
|
| 4 |
import pytest
|
|
|
|
| 1 |
+
"""Unit tests for the CCR engine (deterministic fake backend - no torch)."""
|
| 2 |
|
| 3 |
import numpy as np
|
| 4 |
import pytest
|
docs/specs/0001-data-quality-warnings.md
CHANGED
|
@@ -1,13 +1,13 @@
|
|
| 1 |
-
# Spec 0001
|
| 2 |
|
| 3 |
**Status:** implemented (2026-07-09)
|
| 4 |
**Phase:** 0 **Design doc ref:** §12 (language), §11 pipeline step 3 (Text QA)
|
| 5 |
-
**Requested by:** Mohammad (Slack brief: "proper warnings
|
| 6 |
|
| 7 |
## Problem
|
| 8 |
Researchers upload corpora with rows that embed unreliably (very short texts) or that don't
|
| 9 |
match the language they selected. Today the platform warns about empties, duplicates, and
|
| 10 |
-
truncation-length texts, but not about short texts or language mismatch
|
| 11 |
problems reach published results.
|
| 12 |
|
| 13 |
## Contract
|
|
@@ -25,7 +25,7 @@ problems reach published results.
|
|
| 25 |
|
| 26 |
**Language detection design:**
|
| 27 |
- Corpus-level only: sample up to 200 rows with ≥ 5 tokens; if < 20 such rows → LANGUAGE_UNCERTAIN.
|
| 28 |
-
- Library: `lingua-language-detector` (pure-Python wheels, no torch)
|
| 29 |
- Never block on language; user may proceed (design §12 user-override rule).
|
| 30 |
|
| 31 |
**Warning object schema (all warnings migrate to this shape):**
|
|
@@ -51,12 +51,12 @@ splitting (future spec).
|
|
| 51 |
- golden: sample_corpus.csv against SWLS produces byte-stable warning set
|
| 52 |
|
| 53 |
## Implementation notes
|
| 54 |
-
- Warning construction centralizes in a new `warnings.py` module (engine-side, pure)
|
| 55 |
extraction step toward packages/ccr_engine.
|
| 56 |
- Detection runs once at job start on the parsed dataframe; result flows into metadata + warnings.
|
| 57 |
-
- UI: no new components
|
| 58 |
|
| 59 |
## Deviations (filled after implementation)
|
| 60 |
-
- Detector: `langdetect` (seeded, deterministic) instead of lingua
|
| 61 |
- LANGUAGE_MISMATCH threshold implemented as majority-share >= 0.70 across sampled rows.
|
| 62 |
- MODEL_NOTE (info) added: registry user_warnings surface per run (not in original spec).
|
|
|
|
| 1 |
+
# Spec 0001 - Data-quality warnings: very short texts + likely wrong language
|
| 2 |
|
| 3 |
**Status:** implemented (2026-07-09)
|
| 4 |
**Phase:** 0 **Design doc ref:** §12 (language), §11 pipeline step 3 (Text QA)
|
| 5 |
+
**Requested by:** Mohammad (Slack brief: "proper warnings - wrong language, super short texts, super long texts")
|
| 6 |
|
| 7 |
## Problem
|
| 8 |
Researchers upload corpora with rows that embed unreliably (very short texts) or that don't
|
| 9 |
match the language they selected. Today the platform warns about empties, duplicates, and
|
| 10 |
+
truncation-length texts, but not about short texts or language mismatch - so silent validity
|
| 11 |
problems reach published results.
|
| 12 |
|
| 13 |
## Contract
|
|
|
|
| 25 |
|
| 26 |
**Language detection design:**
|
| 27 |
- Corpus-level only: sample up to 200 rows with ≥ 5 tokens; if < 20 such rows → LANGUAGE_UNCERTAIN.
|
| 28 |
+
- Library: `lingua-language-detector` (pure-Python wheels, no torch) - add to requirements; record library+version in metadata.
|
| 29 |
- Never block on language; user may proceed (design §12 user-override rule).
|
| 30 |
|
| 31 |
**Warning object schema (all warnings migrate to this shape):**
|
|
|
|
| 51 |
- golden: sample_corpus.csv against SWLS produces byte-stable warning set
|
| 52 |
|
| 53 |
## Implementation notes
|
| 54 |
+
- Warning construction centralizes in a new `warnings.py` module (engine-side, pure) - first
|
| 55 |
extraction step toward packages/ccr_engine.
|
| 56 |
- Detection runs once at job start on the parsed dataframe; result flows into metadata + warnings.
|
| 57 |
+
- UI: no new components - existing warnings panel renders the message strings.
|
| 58 |
|
| 59 |
## Deviations (filled after implementation)
|
| 60 |
+
- Detector: `langdetect` (seeded, deterministic) instead of lingua - ~1 MB pure-Python vs ~100 MB wheels; detector + version recorded in metadata so an upgrade stays traceable.
|
| 61 |
- LANGUAGE_MISMATCH threshold implemented as majority-share >= 0.70 across sampled rows.
|
| 62 |
- MODEL_NOTE (info) added: registry user_warnings surface per run (not in original spec).
|
docs/specs/0002-python-script-export.md
CHANGED
|
@@ -1,4 +1,4 @@
|
|
| 1 |
-
# Spec 0002
|
| 2 |
|
| 3 |
**Status:** implemented (2026-07-09)
|
| 4 |
**Phase:** 0 **Design doc ref:** §14
|
|
|
|
| 1 |
+
# Spec 0002 - Reproduction script export (offline-runnable)
|
| 2 |
|
| 3 |
**Status:** implemented (2026-07-09)
|
| 4 |
**Phase:** 0 **Design doc ref:** §14
|
docs/specs/0003-model-registry-and-language-selection.md
CHANGED
|
@@ -1,4 +1,4 @@
|
|
| 1 |
-
# Spec 0003
|
| 2 |
|
| 3 |
**Status:** implemented (2026-07-09)
|
| 4 |
**Phase:** 0 **Design doc ref:** §12, §13
|
|
@@ -12,5 +12,5 @@ into spec-0001 warnings. Add e5-large-v2 and multilingual-e5-base to the UI via
|
|
| 12 |
Validator (`validate_models.py`) runs in tests. Pin revisions before Phase 1 exit.
|
| 13 |
|
| 14 |
**Key tests:** loader validation errors, prefix application for E5 (assert embedded strings
|
| 15 |
-
receive "query: "
|
| 16 |
records registry snapshot, MiniLM stays default.
|
|
|
|
| 1 |
+
# Spec 0003 - Registry-driven models + language selection (brief)
|
| 2 |
|
| 3 |
**Status:** implemented (2026-07-09)
|
| 4 |
**Phase:** 0 **Design doc ref:** §12, §13
|
|
|
|
| 12 |
Validator (`validate_models.py`) runs in tests. Pin revisions before Phase 1 exit.
|
| 13 |
|
| 14 |
**Key tests:** loader validation errors, prefix application for E5 (assert embedded strings
|
| 15 |
+
receive "query: " - testable via fake backend capturing inputs), /api/models shape, metadata
|
| 16 |
records registry snapshot, MiniLM stays default.
|
docs/specs/0004-construct-library-yaml.md
CHANGED
|
@@ -1,4 +1,4 @@
|
|
| 1 |
-
# Spec 0004
|
| 2 |
|
| 3 |
**Status:** implemented (2026-07-09)
|
| 4 |
**Phase:** 0/1 **Design doc ref:** §10.1, §11
|
|
@@ -7,7 +7,7 @@
|
|
| 7 |
Replace `backend/app/seed_constructs.py` seeding with `packages/construct_library/constructs/*.yaml`
|
| 8 |
as the source of truth: loader validates via the same rules as `validate_constructs.py`,
|
| 9 |
computes item_hash with the reference algorithm, seeds/updates DB rows keyed
|
| 10 |
-
(construct_id, version)
|
| 11 |
the full construct snapshot. UI shows verification_status (verified prominently; others with
|
| 12 |
a "wording not yet verified" chip). New questionnaires from the lab = new YAML files + validator
|
| 13 |
run; item wordings marked needs_verification until checked verbatim against the publication.
|
|
|
|
| 1 |
+
# Spec 0004 - Construct library from YAML (brief)
|
| 2 |
|
| 3 |
**Status:** implemented (2026-07-09)
|
| 4 |
**Phase:** 0/1 **Design doc ref:** §10.1, §11
|
|
|
|
| 7 |
Replace `backend/app/seed_constructs.py` seeding with `packages/construct_library/constructs/*.yaml`
|
| 8 |
as the source of truth: loader validates via the same rules as `validate_constructs.py`,
|
| 9 |
computes item_hash with the reference algorithm, seeds/updates DB rows keyed
|
| 10 |
+
(construct_id, version) - append-only, never mutate an existing version. Run metadata embeds
|
| 11 |
the full construct snapshot. UI shows verification_status (verified prominently; others with
|
| 12 |
a "wording not yet verified" chip). New questionnaires from the lab = new YAML files + validator
|
| 13 |
run; item wordings marked needs_verification until checked verbatim against the publication.
|
docs/specs/0005-engine-extraction-and-parity.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
| 1 |
-
# Spec 0005
|
| 2 |
|
| 3 |
-
**Status:** draft (Phase 1 opener
|
| 4 |
**Phase:** 1 **Design doc ref:** §7 (strangler steps 1–3), §10, §16
|
| 5 |
|
| 6 |
Extract the pure analysis package per the strangler plan: move similarity/scoring/warnings/
|
|
@@ -8,7 +8,7 @@ stats/reproducibility logic from backend/app into `packages/ccr_engine/` with th
|
|
| 8 |
`run_ccr_analysis(...)` interface (design §10); backend becomes an orchestration layer. Freeze
|
| 9 |
behavior first with golden datasets (evals/golden_datasets → evals/expected_outputs, human-
|
| 10 |
approved). Then add reference parity: same corpus + items through the published `ccr_wrapper`
|
| 11 |
-
(pyccr) and through ccr_engine
|
| 12 |
revisions on BOTH paths, run in the same container as production. Document any expected
|
| 13 |
differences. Editable-install workspace wiring so backend imports the package (no sys.path hacks).
|
| 14 |
|
|
|
|
| 1 |
+
# Spec 0005 - ccr_engine extraction + reference parity tests (brief)
|
| 2 |
|
| 3 |
+
**Status:** draft (Phase 1 opener - refine after Phase 0 ships)
|
| 4 |
**Phase:** 1 **Design doc ref:** §7 (strangler steps 1–3), §10, §16
|
| 5 |
|
| 6 |
Extract the pure analysis package per the strangler plan: move similarity/scoring/warnings/
|
|
|
|
| 8 |
`run_ccr_analysis(...)` interface (design §10); backend becomes an orchestration layer. Freeze
|
| 9 |
behavior first with golden datasets (evals/golden_datasets → evals/expected_outputs, human-
|
| 10 |
approved). Then add reference parity: same corpus + items through the published `ccr_wrapper`
|
| 11 |
+
(pyccr) and through ccr_engine - similarities match within atol (1e-5 target), pinned model
|
| 12 |
revisions on BOTH paths, run in the same container as production. Document any expected
|
| 13 |
differences. Editable-install workspace wiring so backend imports the package (no sys.path hacks).
|
| 14 |
|
docs/specs/TEMPLATE.md
CHANGED
|
@@ -1,4 +1,4 @@
|
|
| 1 |
-
# Spec NNNN
|
| 2 |
|
| 3 |
**Status:** draft | approved | implemented (+ deviations noted below)
|
| 4 |
**Phase:** 0 | 1 | 2 | 3 **Design doc ref:** §<n>
|
|
|
|
| 1 |
+
# Spec NNNN - <feature name>
|
| 2 |
|
| 3 |
**Status:** draft | approved | implemented (+ deviations noted below)
|
| 4 |
**Phase:** 0 | 1 | 2 | 3 **Design doc ref:** §<n>
|
evals/README.md
CHANGED
|
@@ -1,10 +1,10 @@
|
|
| 1 |
# Golden evals
|
| 2 |
|
| 3 |
-
Purpose: catch silent research-behavior drift
|
| 4 |
-
fields, warning codes
|
| 5 |
|
| 6 |
-
- `golden_datasets/`
|
| 7 |
-
- `expected_outputs/`
|
| 8 |
Claude Code cannot edit files here. To update after an approved behavior change, regenerate
|
| 9 |
manually and review the diff yourself before committing.
|
| 10 |
- Runner arrives with spec 0005 (engine extraction freezes behavior first). Until then, the
|
|
|
|
| 1 |
# Golden evals
|
| 2 |
|
| 3 |
+
Purpose: catch silent research-behavior drift - output shape, column names/order, metadata
|
| 4 |
+
fields, warning codes - per design doc §16.
|
| 5 |
|
| 6 |
+
- `golden_datasets/` - frozen input corpora (never edit rows in place; add new files).
|
| 7 |
+
- `expected_outputs/` - human-approved expected results. **Protected by a PreToolUse hook:**
|
| 8 |
Claude Code cannot edit files here. To update after an approved behavior change, regenerate
|
| 9 |
manually and review the diff yourself before committing.
|
| 10 |
- Runner arrives with spec 0005 (engine extraction freezes behavior first). Until then, the
|
frontend/index.html
CHANGED
|
@@ -3,7 +3,7 @@
|
|
| 3 |
<head>
|
| 4 |
<meta charset="UTF-8" />
|
| 5 |
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
| 6 |
-
<title>CCR Platform
|
| 7 |
</head>
|
| 8 |
<body>
|
| 9 |
<div id="root"></div>
|
|
|
|
| 3 |
<head>
|
| 4 |
<meta charset="UTF-8" />
|
| 5 |
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
| 6 |
+
<title>CCR Platform - Contextualized Construct Representations</title>
|
| 7 |
</head>
|
| 8 |
<body>
|
| 9 |
<div id="root"></div>
|
frontend/src/App.jsx
CHANGED
|
@@ -16,12 +16,17 @@ function relativeTime(iso) {
|
|
| 16 |
}
|
| 17 |
|
| 18 |
function groupProjects(projects) {
|
| 19 |
-
// Buckets by last activity: Today / This week / Earlier
|
|
|
|
| 20 |
// sorted by last activity (backend), so group order falls out naturally.
|
| 21 |
const now = Date.now();
|
| 22 |
const DAY = 86400000;
|
| 23 |
-
const groups = { Today: [], "This week": [], Earlier: [] };
|
| 24 |
for (const p of projects) {
|
|
|
|
|
|
|
|
|
|
|
|
|
| 25 |
const iso = p.last_activity_at || p.created_at;
|
| 26 |
const t = new Date(iso.endsWith("Z") || iso.includes("+") ? iso : iso + "Z").getTime();
|
| 27 |
const age = now - t;
|
|
@@ -39,14 +44,41 @@ export default function App() {
|
|
| 39 |
const [newName, setNewName] = useState("");
|
| 40 |
const [filter, setFilter] = useState("");
|
| 41 |
const [error, setError] = useState("");
|
|
|
|
|
|
|
|
|
|
| 42 |
|
| 43 |
const loadProjects = () =>
|
| 44 |
api.listProjects().then(setProjects).catch((e) => setError(e.message));
|
|
|
|
| 45 |
|
| 46 |
useEffect(() => {
|
| 47 |
loadProjects();
|
|
|
|
| 48 |
}, []);
|
| 49 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 50 |
useEffect(() => {
|
| 51 |
if (projects.length === 0) {
|
| 52 |
setSelectedId(null);
|
|
@@ -84,8 +116,64 @@ export default function App() {
|
|
| 84 |
<span className="sub">
|
| 85 |
Contextualized Construct Representations · theory-driven psychological text analysis
|
| 86 |
</span>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 87 |
</header>
|
| 88 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 89 |
<div className="layout">
|
| 90 |
<aside className="sidebar">
|
| 91 |
<h2>
|
|
@@ -158,7 +246,16 @@ export default function App() {
|
|
| 158 |
</div>
|
| 159 |
)}
|
| 160 |
{selected ? (
|
| 161 |
-
<Workspace
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 162 |
) : (
|
| 163 |
<div className="card">
|
| 164 |
<h3>Welcome</h3>
|
|
@@ -168,7 +265,7 @@ export default function App() {
|
|
| 168 |
score distributions, and a reproducibility record for every run.
|
| 169 |
</p>
|
| 170 |
<p className="small muted">
|
| 171 |
-
Self-contained by design: embeddings run on this server itself
|
| 172 |
third-party AI APIs. Demo instance: storage is ephemeral and may reset;
|
| 173 |
please don't upload sensitive or identifiable data.
|
| 174 |
</p>
|
|
|
|
| 16 |
}
|
| 17 |
|
| 18 |
function groupProjects(projects) {
|
| 19 |
+
// Buckets by last activity: Today / This week / Earlier, with archived
|
| 20 |
+
// projects collapsed into their own group at the bottom. Projects arrive
|
| 21 |
// sorted by last activity (backend), so group order falls out naturally.
|
| 22 |
const now = Date.now();
|
| 23 |
const DAY = 86400000;
|
| 24 |
+
const groups = { Today: [], "This week": [], Earlier: [], Archived: [] };
|
| 25 |
for (const p of projects) {
|
| 26 |
+
if (p.archived) {
|
| 27 |
+
groups.Archived.push(p);
|
| 28 |
+
continue;
|
| 29 |
+
}
|
| 30 |
const iso = p.last_activity_at || p.created_at;
|
| 31 |
const t = new Date(iso.endsWith("Z") || iso.includes("+") ? iso : iso + "Z").getTime();
|
| 32 |
const age = now - t;
|
|
|
|
| 44 |
const [newName, setNewName] = useState("");
|
| 45 |
const [filter, setFilter] = useState("");
|
| 46 |
const [error, setError] = useState("");
|
| 47 |
+
const [auth, setAuth] = useState(null);
|
| 48 |
+
const [showLogin, setShowLogin] = useState(false);
|
| 49 |
+
const [loginName, setLoginName] = useState("");
|
| 50 |
|
| 51 |
const loadProjects = () =>
|
| 52 |
api.listProjects().then(setProjects).catch((e) => setError(e.message));
|
| 53 |
+
const loadAuth = () => api.authMe().then(setAuth).catch(() => {});
|
| 54 |
|
| 55 |
useEffect(() => {
|
| 56 |
loadProjects();
|
| 57 |
+
loadAuth();
|
| 58 |
}, []);
|
| 59 |
|
| 60 |
+
async function handleLogin(e) {
|
| 61 |
+
e.preventDefault();
|
| 62 |
+
if (!loginName.trim()) return;
|
| 63 |
+
try {
|
| 64 |
+
await api.demoLogin(loginName.trim());
|
| 65 |
+
setShowLogin(false);
|
| 66 |
+
setLoginName("");
|
| 67 |
+
await loadAuth();
|
| 68 |
+
} catch (err) {
|
| 69 |
+
setError(err.message);
|
| 70 |
+
}
|
| 71 |
+
}
|
| 72 |
+
|
| 73 |
+
async function handleLogout() {
|
| 74 |
+
try {
|
| 75 |
+
await api.logout();
|
| 76 |
+
await loadAuth();
|
| 77 |
+
} catch (err) {
|
| 78 |
+
setError(err.message);
|
| 79 |
+
}
|
| 80 |
+
}
|
| 81 |
+
|
| 82 |
useEffect(() => {
|
| 83 |
if (projects.length === 0) {
|
| 84 |
setSelectedId(null);
|
|
|
|
| 116 |
<span className="sub">
|
| 117 |
Contextualized Construct Representations · theory-driven psychological text analysis
|
| 118 |
</span>
|
| 119 |
+
<span className="header-auth">
|
| 120 |
+
{auth?.signed_in ? (
|
| 121 |
+
<>
|
| 122 |
+
<span className="small">Hi, {auth.name}</span>
|
| 123 |
+
<button className="header-btn" onClick={handleLogout}>
|
| 124 |
+
Sign out
|
| 125 |
+
</button>
|
| 126 |
+
</>
|
| 127 |
+
) : (
|
| 128 |
+
<button className="header-btn" onClick={() => setShowLogin(true)}>
|
| 129 |
+
Sign in
|
| 130 |
+
</button>
|
| 131 |
+
)}
|
| 132 |
+
</span>
|
| 133 |
</header>
|
| 134 |
|
| 135 |
+
{showLogin && (
|
| 136 |
+
<div className="modal-backdrop" onClick={() => setShowLogin(false)}>
|
| 137 |
+
<div className="modal" onClick={(e) => e.stopPropagation()}>
|
| 138 |
+
<h3>Sign in</h3>
|
| 139 |
+
<p className="hint">
|
| 140 |
+
Signing in lifts the anonymous upload limits
|
| 141 |
+
{auth?.limits?.max_rows
|
| 142 |
+
? ` (currently ${Math.round(auth.limits.max_bytes / 1048576)} MB / ${auth.limits.max_rows.toLocaleString()} rows per file)`
|
| 143 |
+
: ""}
|
| 144 |
+
.
|
| 145 |
+
</p>
|
| 146 |
+
<button className="ghost" disabled title="Arrives with lab accounts">
|
| 147 |
+
Sign in with Google (coming soon)
|
| 148 |
+
</button>
|
| 149 |
+
<form onSubmit={handleLogin} className="mt">
|
| 150 |
+
<label className="field">
|
| 151 |
+
Your name (placeholder sign-in for now)
|
| 152 |
+
<input
|
| 153 |
+
type="text"
|
| 154 |
+
autoFocus
|
| 155 |
+
value={loginName}
|
| 156 |
+
onChange={(e) => setLoginName(e.target.value)}
|
| 157 |
+
placeholder="e.g. Mohammad"
|
| 158 |
+
/>
|
| 159 |
+
</label>
|
| 160 |
+
<div className="row">
|
| 161 |
+
<button className="primary" type="submit" disabled={!loginName.trim()}>
|
| 162 |
+
Continue
|
| 163 |
+
</button>
|
| 164 |
+
<button className="ghost" type="button" onClick={() => setShowLogin(false)}>
|
| 165 |
+
Cancel
|
| 166 |
+
</button>
|
| 167 |
+
</div>
|
| 168 |
+
</form>
|
| 169 |
+
<p className="small muted mt">
|
| 170 |
+
This is a temporary placeholder so larger uploads can be tested. Real
|
| 171 |
+
sign-in (Google + university account) arrives with lab accounts.
|
| 172 |
+
</p>
|
| 173 |
+
</div>
|
| 174 |
+
</div>
|
| 175 |
+
)}
|
| 176 |
+
|
| 177 |
<div className="layout">
|
| 178 |
<aside className="sidebar">
|
| 179 |
<h2>
|
|
|
|
| 246 |
</div>
|
| 247 |
)}
|
| 248 |
{selected ? (
|
| 249 |
+
<Workspace
|
| 250 |
+
key={selected.id}
|
| 251 |
+
project={selected}
|
| 252 |
+
auth={auth}
|
| 253 |
+
onProjectChanged={loadProjects}
|
| 254 |
+
onProjectDeleted={() => {
|
| 255 |
+
setSelectedId(null);
|
| 256 |
+
loadProjects();
|
| 257 |
+
}}
|
| 258 |
+
/>
|
| 259 |
) : (
|
| 260 |
<div className="card">
|
| 261 |
<h3>Welcome</h3>
|
|
|
|
| 265 |
score distributions, and a reproducibility record for every run.
|
| 266 |
</p>
|
| 267 |
<p className="small muted">
|
| 268 |
+
Self-contained by design: embeddings run on this server itself - no
|
| 269 |
third-party AI APIs. Demo instance: storage is ephemeral and may reset;
|
| 270 |
please don't upload sensitive or identifiable data.
|
| 271 |
</p>
|
frontend/src/ResultsView.jsx
CHANGED
|
@@ -73,7 +73,7 @@ export default function ResultsView({ jobId, onBack }) {
|
|
| 73 |
<li key={i}>
|
| 74 |
{typeof w === "string" ? w : (
|
| 75 |
<>
|
| 76 |
-
<code style={{ fontSize: 11 }}>{w.code}</code>
|
| 77 |
</>
|
| 78 |
)}
|
| 79 |
</li>
|
|
@@ -91,7 +91,7 @@ export default function ResultsView({ jobId, onBack }) {
|
|
| 91 |
<div className="card">
|
| 92 |
<h3>Per-item mean loadings</h3>
|
| 93 |
<p className="hint">
|
| 94 |
-
Mean similarity of the corpus to each scale item
|
| 95 |
items drive the construct signal.
|
| 96 |
</p>
|
| 97 |
{summary.item_means.map((m, i) => (
|
|
@@ -122,7 +122,7 @@ export default function ResultsView({ jobId, onBack }) {
|
|
| 122 |
</div>
|
| 123 |
|
| 124 |
<div className="meta-footer">
|
| 125 |
-
<strong>Reproducibility record</strong>
|
| 126 |
{metadata.embedding_dim}) · items hash: <code>{metadata.items_sha256_16}</code> ·
|
| 127 |
text column: <code>{metadata.text_column}</code> · run:{" "}
|
| 128 |
{metadata.started_at} → {metadata.finished_at} ({metadata.duration_seconds}s) ·
|
|
@@ -130,7 +130,7 @@ export default function ResultsView({ jobId, onBack }) {
|
|
| 130 |
{metadata.sentence_transformers &&
|
| 131 |
` · sentence-transformers ${metadata.sentence_transformers}`}
|
| 132 |
<div className="mt small">
|
| 133 |
-
Construct reference: {metadata.construct_reference || "
|
| 134 |
</div>
|
| 135 |
</div>
|
| 136 |
</>
|
|
|
|
| 73 |
<li key={i}>
|
| 74 |
{typeof w === "string" ? w : (
|
| 75 |
<>
|
| 76 |
+
<code style={{ fontSize: 11 }}>{w.code}</code> - {w.message}
|
| 77 |
</>
|
| 78 |
)}
|
| 79 |
</li>
|
|
|
|
| 91 |
<div className="card">
|
| 92 |
<h3>Per-item mean loadings</h3>
|
| 93 |
<p className="hint">
|
| 94 |
+
Mean similarity of the corpus to each scale item - a face-validity check on which
|
| 95 |
items drive the construct signal.
|
| 96 |
</p>
|
| 97 |
{summary.item_means.map((m, i) => (
|
|
|
|
| 122 |
</div>
|
| 123 |
|
| 124 |
<div className="meta-footer">
|
| 125 |
+
<strong>Reproducibility record</strong> - model: <code>{metadata.model}</code> (dim{" "}
|
| 126 |
{metadata.embedding_dim}) · items hash: <code>{metadata.items_sha256_16}</code> ·
|
| 127 |
text column: <code>{metadata.text_column}</code> · run:{" "}
|
| 128 |
{metadata.started_at} → {metadata.finished_at} ({metadata.duration_seconds}s) ·
|
|
|
|
| 130 |
{metadata.sentence_transformers &&
|
| 131 |
` · sentence-transformers ${metadata.sentence_transformers}`}
|
| 132 |
<div className="mt small">
|
| 133 |
+
Construct reference: {metadata.construct_reference || "-"}
|
| 134 |
</div>
|
| 135 |
</div>
|
| 136 |
</>
|
frontend/src/Workspace.jsx
CHANGED
|
@@ -2,7 +2,7 @@ import { useCallback, useEffect, useRef, useState } from "react";
|
|
| 2 |
import { api } from "./api.js";
|
| 3 |
import ResultsView from "./ResultsView.jsx";
|
| 4 |
|
| 5 |
-
export default function Workspace({ project }) {
|
| 6 |
const [corpora, setCorpora] = useState([]);
|
| 7 |
const [constructs, setConstructs] = useState([]);
|
| 8 |
const [models, setModels] = useState([]);
|
|
@@ -19,8 +19,29 @@ export default function Workspace({ project }) {
|
|
| 19 |
const [error, setError] = useState("");
|
| 20 |
const [showNewConstruct, setShowNewConstruct] = useState(false);
|
| 21 |
const [viewJobId, setViewJobId] = useState(null);
|
|
|
|
|
|
|
| 22 |
const fileRef = useRef(null);
|
| 23 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 24 |
const refreshJobs = useCallback(
|
| 25 |
() => api.listJobs(project.id).then(setJobs).catch(() => {}),
|
| 26 |
[project.id]
|
|
@@ -113,13 +134,78 @@ export default function Workspace({ project }) {
|
|
| 113 |
</div>
|
| 114 |
)}
|
| 115 |
|
| 116 |
-
{/*
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 117 |
<div className="card">
|
| 118 |
<h3>
|
| 119 |
<span className="step-badge">1</span>Corpus
|
| 120 |
</h3>
|
| 121 |
<p className="hint">
|
| 122 |
Upload a CSV or XLSX file, then choose the column containing the text to analyze.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 123 |
</p>
|
| 124 |
<div className="row">
|
| 125 |
<div className="grow">
|
|
@@ -139,7 +225,7 @@ export default function Workspace({ project }) {
|
|
| 139 |
<label className="field">
|
| 140 |
Corpus
|
| 141 |
<select value={corpusId} onChange={(e) => setCorpusId(e.target.value)}>
|
| 142 |
-
<option value="">
|
| 143 |
{corpora.map((c) => (
|
| 144 |
<option key={c.id} value={c.id}>
|
| 145 |
{c.filename} ({c.n_rows.toLocaleString()} rows)
|
|
@@ -156,7 +242,7 @@ export default function Workspace({ project }) {
|
|
| 156 |
onChange={(e) => setTextColumn(e.target.value)}
|
| 157 |
disabled={!corpus}
|
| 158 |
>
|
| 159 |
-
<option value="">
|
| 160 |
{corpus?.columns.map((col) => (
|
| 161 |
<option key={col} value={col}>
|
| 162 |
{col}
|
|
@@ -172,7 +258,7 @@ export default function Workspace({ project }) {
|
|
| 172 |
)}
|
| 173 |
</div>
|
| 174 |
|
| 175 |
-
{/* Step 2
|
| 176 |
<div className="card">
|
| 177 |
<h3>
|
| 178 |
<span className="step-badge">2</span>Construct
|
|
@@ -184,7 +270,7 @@ export default function Workspace({ project }) {
|
|
| 184 |
<div className="construct-row">
|
| 185 |
<div className="grow">
|
| 186 |
<select value={constructId} onChange={(e) => setConstructId(e.target.value)}>
|
| 187 |
-
<option value="">
|
| 188 |
{constructs.map((c) => (
|
| 189 |
<option key={c.id} value={c.id}>
|
| 190 |
{c.name} ({c.items.length} items{c.is_seed ? ", library" : ", custom"})
|
|
@@ -232,7 +318,7 @@ export default function Workspace({ project }) {
|
|
| 232 |
)}
|
| 233 |
</div>
|
| 234 |
|
| 235 |
-
{/* Step 3
|
| 236 |
<div className="card">
|
| 237 |
<h3>
|
| 238 |
<span className="step-badge">3</span>Language, model & run
|
|
@@ -240,7 +326,7 @@ export default function Workspace({ project }) {
|
|
| 240 |
<p className="hint">
|
| 241 |
Embeddings run locally via sentence-transformers; model and language are recorded
|
| 242 |
in the run metadata. If the corpus doesn't match the selected language or the
|
| 243 |
-
model doesn't support it, you'll get a warning
|
| 244 |
</p>
|
| 245 |
<div className="run-settings">
|
| 246 |
<label className="field language-control">
|
|
@@ -380,7 +466,7 @@ function NewConstructForm({ onCreated, onError }) {
|
|
| 380 |
</div>
|
| 381 |
</div>
|
| 382 |
<label className="field">
|
| 383 |
-
Scale items
|
| 384 |
<textarea rows={5} value={itemsText} onChange={(e) => setItemsText(e.target.value)} />
|
| 385 |
</label>
|
| 386 |
<button className="primary" type="submit" disabled={saving}>
|
|
|
|
| 2 |
import { api } from "./api.js";
|
| 3 |
import ResultsView from "./ResultsView.jsx";
|
| 4 |
|
| 5 |
+
export default function Workspace({ project, auth, onProjectChanged, onProjectDeleted }) {
|
| 6 |
const [corpora, setCorpora] = useState([]);
|
| 7 |
const [constructs, setConstructs] = useState([]);
|
| 8 |
const [models, setModels] = useState([]);
|
|
|
|
| 19 |
const [error, setError] = useState("");
|
| 20 |
const [showNewConstruct, setShowNewConstruct] = useState(false);
|
| 21 |
const [viewJobId, setViewJobId] = useState(null);
|
| 22 |
+
const [confirmDelete, setConfirmDelete] = useState(false);
|
| 23 |
+
const [deleteText, setDeleteText] = useState("");
|
| 24 |
const fileRef = useRef(null);
|
| 25 |
|
| 26 |
+
async function toggleArchive() {
|
| 27 |
+
try {
|
| 28 |
+
await api.patchProject(project.id, { archived: !project.archived });
|
| 29 |
+
onProjectChanged?.();
|
| 30 |
+
} catch (err) {
|
| 31 |
+
setError(err.message);
|
| 32 |
+
}
|
| 33 |
+
}
|
| 34 |
+
|
| 35 |
+
async function handleDelete() {
|
| 36 |
+
try {
|
| 37 |
+
await api.deleteProject(project.id);
|
| 38 |
+
setConfirmDelete(false);
|
| 39 |
+
onProjectDeleted?.();
|
| 40 |
+
} catch (err) {
|
| 41 |
+
setError(err.message);
|
| 42 |
+
}
|
| 43 |
+
}
|
| 44 |
+
|
| 45 |
const refreshJobs = useCallback(
|
| 46 |
() => api.listJobs(project.id).then(setJobs).catch(() => {}),
|
| 47 |
[project.id]
|
|
|
|
| 134 |
</div>
|
| 135 |
)}
|
| 136 |
|
| 137 |
+
{/* Project header + actions */}
|
| 138 |
+
<div className="project-header">
|
| 139 |
+
<div>
|
| 140 |
+
<span className="project-title">{project.name}</span>
|
| 141 |
+
{project.archived && <span className="pill queued">archived</span>}
|
| 142 |
+
</div>
|
| 143 |
+
<div className="row">
|
| 144 |
+
<button className="ghost" onClick={toggleArchive}>
|
| 145 |
+
{project.archived ? "Unarchive" : "Archive"}
|
| 146 |
+
</button>
|
| 147 |
+
<button className="ghost danger" onClick={() => setConfirmDelete(true)}>
|
| 148 |
+
Delete
|
| 149 |
+
</button>
|
| 150 |
+
</div>
|
| 151 |
+
</div>
|
| 152 |
+
|
| 153 |
+
{confirmDelete && (
|
| 154 |
+
<div className="modal-backdrop" onClick={() => setConfirmDelete(false)}>
|
| 155 |
+
<div className="modal" onClick={(e) => e.stopPropagation()}>
|
| 156 |
+
<h3>Delete "{project.name}"?</h3>
|
| 157 |
+
<p className="hint">
|
| 158 |
+
This permanently deletes {corpora.length} dataset{corpora.length === 1 ? "" : "s"},{" "}
|
| 159 |
+
{jobs.length} run{jobs.length === 1 ? "" : "s"}, and all uploaded and result files.
|
| 160 |
+
This cannot be undone. If you might need it later, use Archive instead.
|
| 161 |
+
</p>
|
| 162 |
+
<label className="field">
|
| 163 |
+
Type the project name to confirm
|
| 164 |
+
<input
|
| 165 |
+
type="text"
|
| 166 |
+
autoFocus
|
| 167 |
+
value={deleteText}
|
| 168 |
+
onChange={(e) => setDeleteText(e.target.value)}
|
| 169 |
+
placeholder={project.name}
|
| 170 |
+
/>
|
| 171 |
+
</label>
|
| 172 |
+
<div className="row">
|
| 173 |
+
<button
|
| 174 |
+
className="primary danger-solid"
|
| 175 |
+
disabled={deleteText !== project.name}
|
| 176 |
+
onClick={handleDelete}
|
| 177 |
+
>
|
| 178 |
+
Delete permanently
|
| 179 |
+
</button>
|
| 180 |
+
<button
|
| 181 |
+
className="ghost"
|
| 182 |
+
onClick={() => {
|
| 183 |
+
setConfirmDelete(false);
|
| 184 |
+
setDeleteText("");
|
| 185 |
+
}}
|
| 186 |
+
>
|
| 187 |
+
Cancel
|
| 188 |
+
</button>
|
| 189 |
+
</div>
|
| 190 |
+
</div>
|
| 191 |
+
</div>
|
| 192 |
+
)}
|
| 193 |
+
|
| 194 |
+
{/* Step 1 - corpus */}
|
| 195 |
<div className="card">
|
| 196 |
<h3>
|
| 197 |
<span className="step-badge">1</span>Corpus
|
| 198 |
</h3>
|
| 199 |
<p className="hint">
|
| 200 |
Upload a CSV or XLSX file, then choose the column containing the text to analyze.
|
| 201 |
+
{auth && !auth.signed_in && auth.limits?.max_rows && (
|
| 202 |
+
<>
|
| 203 |
+
{" "}
|
| 204 |
+
Anonymous limit: {Math.round(auth.limits.max_bytes / 1048576)} MB /{" "}
|
| 205 |
+
{auth.limits.max_rows.toLocaleString()} rows per file; sign in (top right) for
|
| 206 |
+
larger uploads.
|
| 207 |
+
</>
|
| 208 |
+
)}
|
| 209 |
</p>
|
| 210 |
<div className="row">
|
| 211 |
<div className="grow">
|
|
|
|
| 225 |
<label className="field">
|
| 226 |
Corpus
|
| 227 |
<select value={corpusId} onChange={(e) => setCorpusId(e.target.value)}>
|
| 228 |
+
<option value="">- select -</option>
|
| 229 |
{corpora.map((c) => (
|
| 230 |
<option key={c.id} value={c.id}>
|
| 231 |
{c.filename} ({c.n_rows.toLocaleString()} rows)
|
|
|
|
| 242 |
onChange={(e) => setTextColumn(e.target.value)}
|
| 243 |
disabled={!corpus}
|
| 244 |
>
|
| 245 |
+
<option value="">- select -</option>
|
| 246 |
{corpus?.columns.map((col) => (
|
| 247 |
<option key={col} value={col}>
|
| 248 |
{col}
|
|
|
|
| 258 |
)}
|
| 259 |
</div>
|
| 260 |
|
| 261 |
+
{/* Step 2 - construct */}
|
| 262 |
<div className="card">
|
| 263 |
<h3>
|
| 264 |
<span className="step-badge">2</span>Construct
|
|
|
|
| 270 |
<div className="construct-row">
|
| 271 |
<div className="grow">
|
| 272 |
<select value={constructId} onChange={(e) => setConstructId(e.target.value)}>
|
| 273 |
+
<option value="">- select construct -</option>
|
| 274 |
{constructs.map((c) => (
|
| 275 |
<option key={c.id} value={c.id}>
|
| 276 |
{c.name} ({c.items.length} items{c.is_seed ? ", library" : ", custom"})
|
|
|
|
| 318 |
)}
|
| 319 |
</div>
|
| 320 |
|
| 321 |
+
{/* Step 3 - language, model + run */}
|
| 322 |
<div className="card">
|
| 323 |
<h3>
|
| 324 |
<span className="step-badge">3</span>Language, model & run
|
|
|
|
| 326 |
<p className="hint">
|
| 327 |
Embeddings run locally via sentence-transformers; model and language are recorded
|
| 328 |
in the run metadata. If the corpus doesn't match the selected language or the
|
| 329 |
+
model doesn't support it, you'll get a warning - never a silent result.
|
| 330 |
</p>
|
| 331 |
<div className="run-settings">
|
| 332 |
<label className="field language-control">
|
|
|
|
| 466 |
</div>
|
| 467 |
</div>
|
| 468 |
<label className="field">
|
| 469 |
+
Scale items - one per line, verbatim from the validated instrument
|
| 470 |
<textarea rows={5} value={itemsText} onChange={(e) => setItemsText(e.target.value)} />
|
| 471 |
</label>
|
| 472 |
<button className="primary" type="submit" disabled={saving}>
|