docs/validation-report.md

Validation Report

Scope

This validation report covers the first evidence-based pivot release for the manufacturing vertical.

Checked in this release

• Typed requirement extraction from pasted JD text
• Typed requirement extraction from LinkedIn job pages
• Requirement-by-requirement evidence mapping from CV text
• Grouped result rendering in the public UI
• Evidence Map rendering
• JD Quality Review rendering
• Manufacturing benchmark scaffolding and starter fixtures
• Product and quality telemetry capture
• New parse endpoints:
  • /api/jd/parse
  • /api/cv/parse
  • /api/jd/lint
  • /api/analytics/summary
• Backward-compatible fields still present in /api/analyze

What the release now proves

• The product no longer depends on flat keyword matching as its primary output model.
• Requirements are grouped and typed.
• Each requirement can carry evidence, support level, and confidence.
• The final score is decomposed and secondary.
• The UI exposes uncertainty and JD quality warnings.

Commands executed successfully

• npm run lint
• npm run build
• npm run benchmark:manufacturing
• POST /api/analyze against a pasted Hebrew factory-engineer JD
• POST /api/scrape against LinkedIn job URL https://www.linkedin.com/jobs/view/4398015632/

Local runtime checks

• GET /health
  • returned product=Pulse CV - Evidence-Based Hiring Intelligence
  • returned scoringMode=evidence-based-deterministic
  • returned vertical=manufacturing
• GET /
  • returned HTTP 200 locally
• POST /api/jd/parse
  • returned typed requirement groups for a manufacturing JD
• POST /api/analyze
  • returned decomposed scoring, evidence map, JD quality warnings, and candidate/recruiter recommendations
• GET /api/analytics/summary
  • returned telemetry summary JSON

Behavior checks confirmed

• Strong manufacturing-fit sample produced:
  • explicit tool matches (AutoCAD, SolidWorks)
  • explicit methods/domain matches (Continuous Improvement, Quality Management, Manufacturing)
  • decomposed score instead of flat ATS-only output
• Partial-fit industrial leader sample stayed meaningfully lower than the strong-fit case
• Maintenance-heavy sample against manufacturing-engineer JD stayed low-fit and exposed real tool/domain gaps
• Seniority extraction no longer emits a flat individual_contributor requirement for non-leadership roles
• Must-have calibration is stricter:
  • role title and years-experience remain mandatory by default
  • tools, education, and language only become must-haves when the JD wording actually marks them as required
  • broad domain/methodology signals are no longer auto-promoted to must-have unless the source line carries explicit requirement context
• AI no longer overrides deterministic recommendations or bullet rewrites
• AI no longer overrides the deterministic profile summary / tailored bio in the main payload

Remaining weaknesses

• Manufacturing pack is curated and useful, but still compact rather than benchmark-complete.
• Recruiter and hiring manager views are beta-level surfaces, not separate polished products yet.
• AI rewrite quality still depends on provider availability.
• Benchmark scaffolding now exists, but a full human-reviewed corpus is still a next-step task.

Benchmark harness

• Script: npm run benchmark:manufacturing
• Fixtures: benchmark/manufacturing/cases.json
• Current benchmark asserts:
  • primary domain detection
  • expected role family where relevant
  • score range guardrails
  • matched requirement expectations
  • missing requirement expectations

Telemetry added

• Product events:
  • analysis.started
  • jd.parse
  • cv.parse
  • feedback.correction
• Quality events:
  • analysis.completed
  • analysis.fallback
  • jd.lint
• Summary endpoint:
  • GET /api/analytics/summary