FINAL_AUDIT.md

Final Engineering Audit

Date: 2026-06-11

Result

Status: Pass with documented model limitations

The project is now focused on dark-pattern NLP. Market simulation code and datasets were removed. The web application, batch API, screenshot OCR path, and Chrome Manifest V3 extension share the same calibrated classifier.

Verified

• Grouped held-out evaluation uses page_id and reports zero train/test group overlap.
• Word TF-IDF (1, 3) and character char_wb TF-IDF (3, 5) are combined.
• Logistic Regression probabilities are calibrated with sigmoid calibration.
• OCR words below the configured confidence threshold are discarded.
• Remote image fetching is disabled, removing the previous SSRF surface.
• API request size, text length, batch size, dataset pagination, and image size are bounded.
• CORS allows the local web app and Chrome extensions, but not unrelated web origins.
• The extension uses activeTab, sends one batch request, and does not read values typed into inputs or textareas.
• Google Chrome 149 successfully validated and packed the unpacked extension into a CRX, confirming that the Manifest V3 package is readable.
• The generated unpacked artifact contains the same seven files and SHA-256 contents as the source extension.
• Six backend tests pass, including grouped metrics, CORS, OCR, batch IDs, and remote-image blocking.
• Frontend production build and extension JavaScript syntax checks pass.

Current Metrics

• Dataset rows used: 2,381
• Test rows: 477
• Test source pages: 256
• Accuracy: 93.71%
• Macro F1: 0.756
• Weighted F1: 0.9326
• Multiclass log loss: 0.236
• Train/test page_id overlap: 0
• Word features: 2,638
• Character features: 12,544

Remaining Risks

High: rare-class evidence

Forced Action has 4 samples, Sneaking has 16, and Obstruction has 27. Their class metrics are not reliable. In particular, Sneaking currently has F1 0.00 on three held-out samples. Real, independently reviewed examples are required; synthetic records should not be used to claim improved performance.

Medium: out-of-domain generalization

The model correctly identifies dataset-style urgency text such as FLASH SALE | LIMITED TIME ONLY Shop Now, but a manual probe using Act now or your account will be locked forever was classified as Not Dark Pattern. Aggregate metrics therefore should not be interpreted as proof of robust real-world coverage. Add independently labeled adversarial and paraphrase test cases before relying on the extension for production enforcement.

Medium: extension browser coverage

Closed shadow roots, cross-origin frame contents, canvas text, and image-only interfaces are not visible to the DOM scanner. Screenshot OCR remains the fallback for these cases.

Medium: compliance interpretation

Compliance descriptions are heuristic screening guidance. They are not legal advice and should not be presented as definitive CFPB findings.

Low: frontend bundle size

Vite reports a JavaScript chunk around 660 kB before gzip. The application works, but route-level lazy loading would improve initial load performance.

Low: presentation artifacts

The existing pitch deck was generated before the grouped evaluation and Chrome extension work. Its performance methodology and removed market features should be refreshed before the next formal presentation.

Release Checklist

• Run npm run test:backend.
• Run npm run check:extension.
• Run npm run build.
• Run npm run package:extension.
• In chrome://extensions, choose Load unpacked and select artifacts\dark-pattern-nlp-auditor-unpacked. Do not select the ZIP or its parent project directory.
• Use Gunicorn or the Docker image for deployment; Flask's development server is only for local work.