docs/PRD.md

Product Requirement Document (PRD) — forecaster-agent

Version: 0.11 (Self-Evolving Transition Recommendations)
Last updated: 2026-06-24

An autonomous AI × economy forecasting system that generates, calibrates, and publishes falsifiable predictions — with explicit limits on historical extrapolation.

---

1. Product Vision

Build a self-correcting forecasting system for how AI technology and the economy co-evolve. Every prediction is dated, judgeable, and Brier-scored; the scoreboard feeds the next cycle.

Core objectives

Objective	Definition of done
Falsifiability	Every forecast has resolution_date + resolution_criteria
Self-calibration	Mean Brier + 10-decile reliability curve in registry.scoreboard()
Evidence-backed	Signals from ingest; evolution prior from citable case library
Honest extrapolation	OOD Mahalanobis signal widens confidence when scenario is unprecedented
Crowd wisdom (Phase 2)	Anti-anchoring contributions filtered by soundness × novelty

---

2. System Architecture (as implemented)

┌─────────────────────────────────────────────────────────────────────┐
│                         Orchestrator (`loop.py`)                    │
│                                                                     │
│  RESOLVE ──► SCORE      INGEST signals                             │
│  (judge due  (Brier)      (arXiv·RSS·Tavily·FRED)                  │
│  predictions)   │               │                                   │
│                 │    track      │   signals                         │
│                 └──► record ──► EVOLUTION PRIOR                    │
│                                    │    (PCA/GMM/OOD)               │
│                                    ▼                                │
│                               FORECAST ◄── track record             │
│                                    │                                │
│                              [CROWD GATE]  ← Phase 2, not in loop  │
│                                    │                                │
│                                DEDUP ──► REGISTRY (SQLite)         │
│                                    │                                │
│                               PUBLISH ──► site · webhook · git     │
│                           (review gate)                             │
└─────────────────────────────────────────────────────────────────────┘

Parallel surface: `dashboard.py` (Streamlit) + `services/read_model.py` (read API seam)

2.1 Forecaster loop (loop.py, forecast.py, registry.py, publish.py)

Capability	Status
Signal ingestion	✅ arXiv, RSS, Tavily*, FRED*
Evolution prior in forecast prompt	✅ since v0.5
Falsifiable prediction generation	✅
Due prediction judging	✅
Brier scoreboard	✅
SQLite registry	✅ data/forecaster.db
Review gate (require_review)	✅ default true
Crowd gate in loop	🔧 Phase 2

* Requires API key in environment.

2.2 Crowd gate (crowd.py, services/crowd_service.py)

Capability	Status
Soundness + novelty gates	✅ offline-tested
Reputation from Brier	✅
Anti-anchoring REST submit	✅ GET/POST /v1/predictions/{id}/...
Crowd gate in loop	✅ process_open_prediction_crowds()
Contribution resolve on prediction resolve	✅
Discord/Telegram bot adapters	🔧 Phase 2b

2.3 Job evolution agent (evolution.py)

Capability	Status
15+ citable historical cases	✅
PCA + Bayesian GMM	✅
Mahalanobis OOD	✅
EvolutionPrior.to_prompt_context()	✅ wired into forecast.generate_predictions()

2.4 Job Radar (job_radar.py, dashboard.py)

Capability	Status
~80 occupation profiles, 13 industries	✅ v0.10
Hybrid RAG (α·structural + β·semantic)	✅
Search retrieval (combined_similarity: semantic embed + lexical blend)	✅ v0.10
Transition recommendation (compute_transition_paths: skill 40% + overlap 15% + risk 25% + demand 20%)	✅
Skill semantic similarity (_skill_semantic_sim: SentenceEmbedder cosine on required_skills)	✅ v0.11
Domain proximity (_domain_proximity: adjacent-industry graph, replaces dead jaccard)	✅ v0.11
LLM confidence gate (transition score gated by _transition_confidence from evaluator)	✅ v0.11
Search vs transition paths kept separate (HR-12)	✅ Phase 8
Anchored search hides misleading at-risk column (weak text scores)	✅ Phase 8 / v0.10
Hot-role KB coverage (CORE_HOT_ROLE_QUERIES + search_aliases)	✅ v0.10
Semantic job search (SentenceEmbedder / MiniLM, title-only vectors)	✅ v0.10
Offline CI search (RadarHashingEmbedder via config.ci.yaml)	✅ HR-1
Bilingual + CJK queries (multilingual embed + lexical tokens)	✅ v0.10
Query title normalization (normalize_search_query, title_aliases)	✅ v0.10
User-facing match tiers (无匹配 / 弱 / 强)	✅ Phase 8
Personalized transition weights (experience level, retrain tolerance)	✅ Phase 8
Field survey: canonical title + experience level (HR-13)	✅ Phase 8
BLS + field-feedback calibration overlay	✅
Prediction market UI	✅
LLM KB expansion for unknown titles	✅ (needs API key)

Two crowd surfaces (do not conflate):

Surface	Purpose	Minimum identity
Field feedback (JobFeedback, Radar survey)	Calibrate displacement risk + empirical transition targets	Canonical job title + experience level (HR-13)
Prediction crowd (contribution, REST/Telegram)	Anti-anchoring probability submissions	contributor_id only; optional expert weighting is out of scope

2.5 Job Query Calibration Agent (services/job_query_agent/, Phase 9)

Continuous retrieval QA loop — discovers search queries, evaluates KB match quality, auto-fixes safe alias gaps, and gates risky changes behind human review.

Capability	Status
Query discovery (CORE hot roles, seed file, JobFeedback titles)	✅
Per-query verdict (ok, p0_regression, weak_core, kb_gap, weak_match)	✅
Simulate-before-apply (alias_patch, title_alias)	✅
Auto-apply safe fixes (jobs_kb.json + config.yaml title map)	✅ gated
CI guard (query-agent audit — P0 + weak-core fail build)	✅
Daily closed loop (query-agent run → commit KB/config)	✅
Trace log (data/query_agent_traces.jsonl, gitignored)	✅
Human review queue (pending/job_calibration/) for kb_profile_new	✅
HF search-log ingest (frequency-weighted gaps)	✅ P2

Operating modes:

Command	When	Behaviour
query-agent audit	CI after pytest	Read-only; raises on P0 regression or weak-core
query-agent once	Manual triage	Audit + queue non-ok proposals to pending/
query-agent run	Daily cron	Multi-round discover → simulate → auto-apply → re-check; commits diffs
query-agent apply	After human review	Apply all pending/job_calibration/*.json (incl. kb_profile_new)
query-agent ingest-logs	HF export merge	Merge external search JSONL into data/radar_search_log.jsonl

2.6 Transition Evaluator (services/transition_evaluator/, Phase 10)

LLM-as-judge system that continuously fills and validates the KB's transition_targets, enabling the knowledge base to self-improve over time without manual curation.

Capability	Status
evaluate_pair(anchor, candidate) → Groq feasibility 0.0–1.0 + reasoning	✅
Persistent cache (data/transition_eval_cache.json, key: anchor→candidate)	✅
missing_pairs(jobs) — only targets jobs with empty transition_targets	✅
run_evaluation_pass(jobs, max_pairs, min_feasibility=0.55) → promote to KB	✅
_promote_to_kb — appends to transition_targets with _source: "llm_eval" + confidence	✅
Wired into daily calibration loop (run_calibration_cycle post-step)	✅
CLI: python run.py query-agent transition-eval [N]	✅

Scoring guide used by LLM judge:

Range	Label	Retraining
0.85–1.0	Natural progression	0–6 months
0.60–0.84	Adjacent move	6–18 months
0.35–0.59	Deliberate pivot	18–36 months
0.10–0.34	Major reskill	substantial
0.00–0.09	Extreme leap	very high effort

Pairs scoring ≥ 0.55 are promoted to KB; retrain_months = max(2, round(24 × (1 − feasibility))).

2.7 Read model / external integration seam (services/read_model.py)

Capability	Status
get_scoreboard()	✅
get_ood_assessment()	✅
search_jobs()	✅
list_open_predictions()	✅
MCP Server wrapper	✅ mcp_server.py
REST /v1/*	✅ api_server.py

---

3. Harness Engineering Invariants (non-negotiable)

ID	Rule	Enforcement
HR-1	Offline-first testability	LLM/embeddings behind Protocol + stubs; pytest tests/ passes with no API key
HR-2	Pure scoring core	Brier, JSD, PCA/GMM math side-effect free; tests assert properties
HR-3	Explicit thresholds	tau_*, alpha, beta, n_bootstrap in config.yaml only
HR-4	Citable cases only	test_case_library_integrity rejects missing sources
HR-5	Strict publishing gate	require_review: true in default config; PRs flipping default rejected
HR-6	Model-agnostic LLM	Single seam: forecast.call_llm() (Groq → Anthropic)
HR-7	Citable job profiles	Every JobProfile.sources non-empty
HR-8	BUSL-1.1 license	Commercial SaaS/API requires separate license
HR-9	Citation integrity	LLM-generated sources URLs pass schema check; arXiv IDs must not reference a future YYMM; example.com / placeholder domains rejected at parse time. Enforced in forecast._sanitize_sources().
HR-10	Resolved-state durability	run.py export serialises status, outcome, brier, resolved_at; ensure_demo_registry reloads them via model_validate_json preserving resolution. Track record must never regress to all-open after a cache eviction.
HR-11	Origin transparency	Every prediction shown in the Track Record tab carries an explicit origin badge (seed = curated benchmark, live = daily LLM cron). run.py export writes live-only rows to predictions_live.jsonl; run.py verify-export fails CI if DB live state diverges from the committed file after a resolve.
HR-12	Retrieval ≠ recommendation	Job search ranks by combined_similarity (semantic + lexical blend). Transition fit ranks by transition_score from compute_transition_paths() only. UI and API must not present search hits as career recommendations without an anchor role + transition pass. Thresholds in config.yaml (HR-3); use tier labels (tier_no_match / tier_weak / tier_strong), not raw cosine intuition (~0.7). Production embedder: sentence_transformers; CI/harness: hashing.
HR-13	Field feedback profile	Radar career survey collects canonical job_title (KB dropdown, not free text) plus experience_level (bucket: junior / mid / senior, or years-in-role bands). Aggregates keyed by title today; stratify by (title, experience_level) when n≥5 per cell. Powers personalized retrain tolerance and transition weight overrides.

---

4. Configuration (config.yaml)

Key	Purpose
database_path	Documented SQLite location (engine in schemas.py)
evolution.n_bootstrap	GMM bootstrap iterations (use 10 in tests)
evolution.scenario	Optional override of CURRENT_AI_SCENARIO
job_radar.*	Hybrid RAG weights, KB path, transition weights
job_radar.search.*	embedder (sentence_transformers | hashing), embed/lex blend, tier thresholds, title_aliases (HR-3)
job_radar.personalization.*	Experience-level weight overrides, max retrain months (Phase 8)
job_query_agent.*	Discover sources, evaluate gates, auto_apply types/limits, traces path (Phase 9)
crowd.*	Gate thresholds (for Phase 2 API)
require_review	Publishing safety gate

---

5. Roadmap (honest status)

Phase 1 — Engineering stabilization ✅ (v0.5)

• Flat-layout import strategy (try/except flat vs package imports)
• CLI (run.py score) operational
• Tests in tests/; CI in .github/workflows/ci.yml
• config.yaml aligned with SQLite
• Evolution prior wired into forecast loop

Phase 2 — Crowd contributor surface ✅

• REST anti-anchoring API
• CrowdGate in loop
• Telegram bot (python -m bots.run_bots telegram)
• Discord bot (python -m bots.run_bots discord)

Phase 3 — Read-only MCP ✅ + REST ✅

• MCP Server (mcp_server.py) — stdio, 4 read tools
• REST API (api_server.py) — /v1/*, OpenAPI /docs
• Shared handlers (services/mcp_handlers.py)
• Optional FORECASTER_API_KEY + rate limiting
• API key rotation / multi-tenant auth

Phase 4 — Dashboard decomposition ✅

• Split dashboard.py into ui/tabs/* (accuracy, benchmarks, guard, radar)
• Dashboard reads via services/ — services/dashboard_data.py, services/dashboard_seed.py, services/config_loader.py; dashboard.py is now a 104-LOC orchestrator only

Phase 5 — Trust & scale ✅ (v0.7, 2026-06-24)

• Brier history public dataset: Resolved predictions downloadable as CSV from the Track Record tab (⬇ Download track_record.csv)
• Non-Western + net-loss evolution cases: 4 new TransitionCase entries added to CASE_LIBRARY (total: 19 cases):
  • china_mobile_payment — Alipay/WeChat Pay & bank clerks, China 2013-2023 (net loss, multiplier 0.75)
  • japan_factory_robots — FANUC/Kawasaki robotic assembly, Japan 1970-2000 (slight net gain)
  • india_bpo_automation — RPA + LLM chatbots & BPO agents, India 2015-2025 (net loss, multiplier 0.80)
  • south_korea_steel_automation — POSCO continuous casting, South Korea 1980-2010 (net loss, multiplier 0.65)
• Novice usability score ≥ 7/10 — UX audit identified 5 defects; 4 fixed:
  • Navigation: st.radio → st.tabs (standard, scannable)
  • Onboarding: session-based welcome banner with 3-step guide
  • Tab names: "Plausibility Guard" → "Scenario Advisor"; "Historical Benchmarks" → "History: How AI Changed Jobs"; "Forecast Accuracy" → "Track Record"
  • Brier score: inline explainer st.expander with plain-language table
  • Radar tab: "How to use" hint dismissable caption

Phase 6 — Integrity & Learning Loop (v0.6, 2026-06-24)

Findings from joint architect / open-source engineer / PM review after first real LLM accumulation run (12 predictions via Groq):

• HR-9 Citation sanitiser: forecast._sanitize_sources() strips hallucinated arXiv IDs and placeholder domains at parse time
• Horizon normalisation: forecast._normalize_horizon() coerces "2027" → "2027-Q4", "2027-H1" → "2027-Q2" before persisting
• Accuracy tab sources: judged_rationale surfaced in resolved-predictions table so users can independently verify
• Groq rate-limit degradation: call_llm detects 429 / RateLimitError, retries once after 60 s, falls back to RuntimeError with structured message rather than silent failure
• User feedback visibility: Radar tab shows "N jobs calibrated from your feedback" badge when field_calibration has data, closing the perception gap
• ROOT_CAUSE contrastive pairs in predictions_seed.json + track_record_summary WRONG entries show 280-char rationale window
• PR-aligned commit-back: predictions_live.jsonl now correctly detects new/changed file via git add before git diff --cached

Phase 7 — Live Track Record Credibility (v0.8, 2026-06-24)

Problem: seed demo data (12 resolved) and daily LLM predictions (12 open) were merged in the UI with no origin label — users could not tell curated benchmark from real agent performance.

• HR-11 Origin split: Track Record tab shows separate panels for Curated benchmark (seed) and Live LLM with independent resolved counts and mean Brier
• Upcoming resolutions: timeline of open live predictions sorted by resolution_date, showing criteria so users see the loop is active
• CSV origin column: public download includes seed|live for replication
• Live-only export: run.py export writes only non-seed predictions to predictions_live.jsonl (seed stays in predictions_seed.json)
• CI guard: run.py verify-export after daily cron; fails if live DB state ≠ committed JSONL (catches silent resolve/export regressions)

Phase 8 — Personalized Job Radar & Search Quality (v0.9, 2026-06-23)

Problem: search retrieval and career recommendation were conflated — e.g. a finance process-improvement query surfacing an unrelated AI trading role. Legacy hashing-only embeddings and description-diluted vectors produced weak bilingual matches; recommendations also ignored user seniority (novice vs senior retrain cost) and field survey lacked tenure.

Design principles (ranking priorities for transition paths):

1. Skill extensibility — skill-vector proximity + shared skills (bridgeable gap)
2. User fit — experience level caps acceptable retrain_months; juniors favour lower-gap targets
3. Forward demand — scenario-driven demand under active diffusion assumptions

Shipped pre-Phase-8 (search hotfix, not yet HR-3 compliant):

• combined_similarity = 0.45·embed + 0.55·lexical, multi-word token penalty, industry-only boost
• Code constants _SIMILARITY_THRESHOLD=0.42, _STRONG_MATCH_THRESHOLD=0.55; sort by combined_similarity
• Tests in tests/test_job_radar.py

Phase 8 deliverables:

• HR-12 Enforce retrieval≠recommendation in Radar UI: search → pick anchor role → rank browse/related by transition_score
• HR-3 Move search thresholds + embed/lex blend to config.yaml → job_radar.search.*
• User-facing tiers on combined_similarity: 无匹配 (< weak) / 弱 / 强 (replace misleading single green banner)
• HR-13 Add JobFeedback.experience_level; survey field + i18n; migration-safe default for existing rows
• Stratified field calibration — get_empirical_metrics() groups by (title, experience_level) when n≥5; fallback to title-only
• Sidebar user profile (session): experience level + max retrain months → override _TRANSITION_WEIGHTS / filter candidates
• Strong-match threshold audit — target ~0.65–0.70 on calibrated combined score (after config move + tier labelling)
• Anchored-search UX — when a strong search anchor is set, hide the at-risk column (weak combined_similarity must not surface unrelated roles, e.g. engineer → Logistics Dispatcher)

Explicitly out of Phase 8: prediction-crowd contributor job-title collection; Indeed/LinkedIn scraping. (Embedding model swap delivered in v0.10 — see RELEASE_v0.10.md; stays behind resolve_embedder() + HR-1 dual config.)

Phase 9 — Job Query Calibration Agent (v0.10, 2026-06-24)

Problem: hot-role search gaps (e.g. «software developer» ≠ Software Engineer) were caught only after user reports. Need a continuous retrieval QA loop that discovers queries, evaluates KB match quality, traces gaps, and auto-resolves safe calibration without waiting for manual bug reports.

Design principles:

1. Retrieval QA only — agent calibrates search anchors / aliases / KB coverage, not transition recommendations (HR-12)
2. Simulate before mutate — every auto-apply runs simulate_alias_patch / simulate_title_alias and must beat sim_before
3. Tiered mutation policy — alias_patch + title_alias may auto-apply when gated; kb_profile_new always queues to pending/ (HR-5)
4. CORE guard in CI — P0 regressions fail run.py query-agent audit

Discover sources (discover_queries):

Source	Config flag	Purpose
CORE_HOT_ROLE_QUERIES	discover.include_core	Regression guardrail for high-traffic titles (EN + zh)
data/query_seed.json	discover.include_seed	Curated long-tail / edge-case queries
JobFeedback.job_title	discover.include_feedback_titles	Real user search intent from field survey

Auto-apply gates (job_query_agent.auto_apply):

• enabled: true (daily cron); CI audit never mutates
• Allowed types: alias_patch (→ jobs_kb.json search_aliases), title_alias (→ job_radar.search.title_aliases)
• min_sim_after ≥ tier_weak (default 0.55); CORE queries must resolve to expected_id
• alias_patch requires strict improvement over sim_before
• max_rounds (default 3) — reload KB between rounds

P0 delivered:

• services/job_query_agent/ — discover, evaluate, propose, simulate, apply, traces, audit, loop
• run.py query-agent audit|once|run|apply|ingest-logs
• config.yaml → job_query_agent.* + job_radar.search.title_aliases; seed file data/query_seed.json
• Hot-role KB entries: tech_product_manager, tech_data_scientist, tech_project_manager; expanded tech_software_eng aliases
• CI step: python run.py query-agent audit after pytest
• Daily cron: query-agent run commits jobs_kb.json / config.yaml when auto-apply makes changes
• Traces: data/query_agent_traces.jsonl (gitignored)
• Tests: tests/test_job_query_agent.py, test_anchored_search_hides_unrelated_at_risk in tests/test_radar_render.py

Phase 9 backlog:

• P1: query-agent run — simulate + auto-apply alias_patch / title_alias (gated); manual pending/ for kb_profile_new
• P1: query-agent apply — merge human-approved proposals from pending/job_calibration/
• P2: HF/Radar search log ingest (data/radar_search_log.jsonl) + frequency-weighted discovery
• P3: kb_profile_new → LLM/cache KB append via apply + gated auto in run

Phase 9b — Semantic job search (v0.10, 2026-06-24)

Production search upgraded from hashing-only to multilingual sentence embeddings while CI remains offline.

• SentenceEmbedder — paraphrase-multilingual-MiniLM-L12-v2; _job_embed_title() (title + aliases only)
• resolve_embedder(search_cfg) — config.yaml → sentence_transformers; config.ci.yaml → hashing
• Lexical fix: preserve 2-char abbreviations (ml, hr, qa, gp)
• KB alias corrections + seed tuning; 114–115/115 queries OK, 0 P0
• Dockerfile pre-downloads MiniLM for HF Space

See RELEASE_v0.10.md for score lift table and commit list.

Phase 10 — Self-Evolving Transition Recommendations (v0.11, 2026-06-24)

Problem: transition_score used a dead _skill_jaccard (always 0) for overlap, and cross-industry leaps (Accounts Receivable → AI Engineer) were scored unrealistically high because demand + risk-reduction signals dominated. KB transition_targets for new roles were empty — no way to bootstrap without manual curation.

Design principles:

1. Skill extensibility — real required_skills cosine similarity (_skill_semantic_sim), not bag-of-words
2. Industry proximity — explicit adjacent-industry graph (_domain_proximity) penalises unrelated domain jumps
3. LLM arbitration — LLM-as-judge provides pair-level confidence scores that gate the final score
4. Incremental self-improvement — nightly pass fills uncached pairs; KB grows automatically

Shipped:

• _skill_semantic_sim(a, b) — SentenceEmbedder cosine on required_skills text; cached by sorted (id_a, id_b); falls back gracefully when no embedder
• _domain_proximity(a, b) — same-industry=1.0, adjacent=0.5 (Finance↔Legal, Tech↔Media, etc.), unrelated=0.0
• Overlap term: 0.65 × skill_semantic_sim + 0.35 × domain_proximity (replaces dead jaccard)
• LLM confidence gate: score *= (0.5 + confidence × 0.5) when _transition_confidence cached on target job
• services/transition_evaluator/evaluate.py — LLM-as-judge (Groq llama-3.3-70b); 5-tier feasibility scale
• services/transition_evaluator/cache.py — persistent data/transition_eval_cache.json; 118+ pairs cached
• run_evaluation_pass() wired into run_calibration_cycle — runs after every query calibration pass
• CLI: python run.py query-agent transition-eval [N]
• KB: lawyer, cybersecurity_analyst, ai_engineer transition_targets now LLM-validated (11 entries promoted)

Transition quality improvements:

Role queried	Before (v0.10)	After (v0.11)
Accounts Receivable Manager	→ AI Engineer (Finance↔Tech leap)	→ AI Financial Compliance Auditor (Finance, domain-matched)
Cybersecurity Analyst	no curated targets	→ AI Agent Fleet Manager (Tech, LLM-validated)
Lawyer	no curated targets	→ Corporate General Counsel (Legal, conf=0.9)

Self-evolution path:

daily: query-agent run
  → query calibration (alias/profile gaps)
  → transition-eval pass (fill empty transition_targets via LLM)
  → new roles auto-bootstrapped on next CI run

---

6. Success metrics

Metric	Target
Offline test pass rate	100%
Mean Brier (resolved n≥30)	< 0.20
OOD fires → confidence downshift	Qualitative audit
P0 bugs open	0
MCP read tools documented	Phase 3 exit
LLM-generated sources with hallucinated arXiv IDs	0 (HR-9)
Resolved predictions surviving cache eviction	100% (HR-10)
predictions_live.jsonl committed per real-LLM cron run	✅ (daily)
Track Record UI shows seed vs live origin	HR-11 (Phase 7)
Live resolved count visible independently of seed	Phase 7 exit
Search strong-match precision (manual audit, n≥20 queries)	≥ 80% relevant anchor role (Phase 8)
Field survey includes experience level	HR-13 (Phase 8)
Transition cards respect user retrain cap when profile set	Phase 8
query-agent audit passes in CI (CORE + seed)	Phase 9
query-agent run daily with zero P0 regressions	Phase 9
Search gaps auto-closed via alias/title_alias (weekly)	Qualitative — trace + commit log
LLM fallback rate on top queries (logged)	Phase 9 P2
Transition recommendations stay within same or adjacent industry	Phase 10
LLM-evaluated transition pairs (transition_eval_cache.json)	≥ 200 pairs (growing daily)
New KB roles bootstrapped within 1 daily cycle	Phase 10 self-evolution

---

7. Out of scope (v0.10)

• Auto-publishing with require_review: false as default
• Financial advice positioning
• Real-money prediction markets
• Commercial API without BUSL commercial license
• Using raw embedding cosine (~0.7) as user-facing "similarity" without combined score + tier labels
• Collecting job title / tenure on prediction crowd submissions (HR-13 applies to field feedback only)
• Indeed/LinkedIn job scraping for query-agent discovery (BLS/O*NET + logs + feedback only)