FINAL_AUDIT.md

# Final Engineering Audit

Date: 2026-06-11

## Result

Status: **Pass with documented model limitations**

The project is now focused on dark-pattern NLP. Market simulation code and
datasets were removed. The web application, batch API, screenshot OCR path, and
Chrome Manifest V3 extension share the same calibrated classifier.

## Verified

- Grouped held-out evaluation uses `page_id` and reports zero train/test group
  overlap.
- Word TF-IDF `(1, 3)` and character `char_wb` TF-IDF `(3, 5)` are combined.
- Logistic Regression probabilities are calibrated with sigmoid calibration.
- OCR words below the configured confidence threshold are discarded.
- Remote image fetching is disabled, removing the previous SSRF surface.
- API request size, text length, batch size, dataset pagination, and image size
  are bounded.
- CORS allows the local web app and Chrome extensions, but not unrelated web
  origins.
- The extension uses `activeTab`, sends one batch request, and does not read
  values typed into inputs or textareas.
- Google Chrome 149 successfully validated and packed the unpacked extension
  into a CRX, confirming that the Manifest V3 package is readable.
- The generated unpacked artifact contains the same seven files and SHA-256
  contents as the source extension.
- Six backend tests pass, including grouped metrics, CORS, OCR, batch IDs, and
  remote-image blocking.
- Frontend production build and extension JavaScript syntax checks pass.

## Current Metrics

- Dataset rows used: 2,381
- Test rows: 477
- Test source pages: 256
- Accuracy: 93.71%
- Macro F1: 0.756
- Weighted F1: 0.9326
- Multiclass log loss: 0.236
- Train/test `page_id` overlap: 0
- Word features: 2,638
- Character features: 12,544

## Remaining Risks

### High: rare-class evidence

`Forced Action` has 4 samples, `Sneaking` has 16, and `Obstruction` has 27.
Their class metrics are not reliable. In particular, `Sneaking` currently has
F1 `0.00` on three held-out samples. Real, independently reviewed examples are
required; synthetic records should not be used to claim improved performance.

### Medium: out-of-domain generalization

The model correctly identifies dataset-style urgency text such as `FLASH SALE |
LIMITED TIME ONLY Shop Now`, but a manual probe using `Act now or your account
will be locked forever` was classified as `Not Dark Pattern`. Aggregate metrics
therefore should not be interpreted as proof of robust real-world coverage.
Add independently labeled adversarial and paraphrase test cases before relying
on the extension for production enforcement.

### Medium: extension browser coverage

Closed shadow roots, cross-origin frame contents, canvas text, and image-only
interfaces are not visible to the DOM scanner. Screenshot OCR remains the
fallback for these cases.

### Medium: compliance interpretation

Compliance descriptions are heuristic screening guidance. They are not legal
advice and should not be presented as definitive CFPB findings.

### Low: frontend bundle size

Vite reports a JavaScript chunk around 660 kB before gzip. The application
works, but route-level lazy loading would improve initial load performance.

### Low: presentation artifacts

The existing pitch deck was generated before the grouped evaluation and Chrome
extension work. Its performance methodology and removed market features should
be refreshed before the next formal presentation.

## Release Checklist

- Run `npm run test:backend`.
- Run `npm run check:extension`.
- Run `npm run build`.
- Run `npm run package:extension`.
- In `chrome://extensions`, choose **Load unpacked** and select
  `artifacts\dark-pattern-nlp-auditor-unpacked`. Do not select the ZIP or its
  parent project directory.
- Use Gunicorn or the Docker image for deployment; Flask's development server
  is only for local work.