docs/app-analysis.md

ReelShield — App Analysis & User Testing

CS668 Analytics Capstone · Pace University · Spring 2026 Team: Abi Keegan, Janvi Rautela, Nobukhosi Sibanda

---

1. Overview

This document presents the results of structured user testing conducted on ReelShield. Three participants representing the app's primary target user groups completed a series of tasks and provided feedback. Results were used to make iterative improvements to the application.

---

2. Methodology

Each participant completed a series of tasks with one of the app creators sitting beside them for guidance. Participants were asked to think aloud where possible. One of the app creators asked them questions about their experience afterwards.

2.1 Tasks

• Register for an account
• Search for any movie and review its content warnings
• Use the chat interface to ask a specific question about that film
• Toggle spoiler mode on and off and observe the change in warnings and chat answers
• Add the movie to your watchlist
• Use the recommendation engine to find another film avoiding a specific warning category
• Use the Group Watch Finder to find a film safe for two people with different sensitivities
• Leave a star rating and written review

---

3. Participants

Participant	Profile	Tech Comfort	Primary Need
"Jenna"	28, has triggering subjects she would like to avoid in movies	High	Avoid self-harm, suicide and gore
"David"	61, uncle of an adult nephew with photosensitive epilepsy	Medium	Check flashing lights before recommending
"Andy"	29, movie lover, no triggers or health concerns	High	Explore features, test recommendations

---

4. Results Per Participant

4.1 Jenna

Tasks Completed

• Made account with username and password
• Searched for Avatar Fire and Ash (2025), which loaded in under 2 seconds (was not in database)
• Found self-harm and suicide warnings were absent, and that gore is minimal
• Kept spoiler mode off and asked about character death in the chat
• Toggled spoiler mode on and noted the warnings became more specific
• Added film to watchlist successfully
• Used recommendations to find a film avoiding self-harm
• Gave Avatar Fire and Ash (2025) 5 stars with a written review

Observations

• Initially did not notice the spoiler toggle as it blended into the content warning module
• Found the chat response time to be a little slow

Post-Test Scores

Metric	Score (1-5)	Comment
Ease of use	4	Spoiler toggle hard to find initially
Usefulness	5	"This will be so helpful for a lot of people"
Trustworthiness	3	Does not always trust AI
Would use instead of searching online	Yes	Saved time and avoided spoilers

---

4.2 David

Tasks Completed

• Registered an account
• Searched Get Out (2017) which loaded from cache instantly
• Immediately located the Flashing Lights banner at the top of warnings
• Asked chat "can I recommend this to a family member with photosensitive epilepsy?"
• Checked IMDb and Rotten Tomatoes scores in the reviews section
• Used Group Watch Finder, added his wife who is sensitive to animal abuse
• Group Watch returned 2 results
• Gave Get Out (2017) 3 stars with a written review

Observations

• Very positive reaction to flashing lights being shown at the top of the page
• Noticed that he was able to make a very simple password, recommended adding password requirements for security
• Found the instructions in Group Watch Finder helpful

Post-Test Scores

Metric	Score (1-5)	Comment
Ease of use	5
Usefulness	5	"This is really cool for people with epilepsy or other things they want to avoid."
Trustworthiness	3	Wanted to know more about how AI makes decisions
Would use instead of searching online	Yes	Much faster and likes options for including spoilers

---

4.3 Andy

Tasks Completed

• Registered and logged in without difficulty
• Searched Parasite (2019), first load took under 2 seconds (fresh Gemini call)
• Browsed all 9 warning categories and read the notes
• Used recommendation engine to find similar films without sexual content
• Opened the AI prompt but did not edit it
• Explored the external reviews section and appreciated seeing IMDb, RT, and Metacritic together
• Left a review and rated the film 4 stars

Observations

• Found the editable prompt feature "unexpected and impressive"
• Accidentally clicked on a "How are you feeling? (optional)" mood and was unable to de-select it

Post-Test Scores

Metric	Score (1-5)	Comment
Ease of use	3	Unable to deselect mood
Usefulness	3	Recommendations could be more diverse, add more to the cache
Trustworthiness	5	Liked being able to edit the prompt
Would use instead of searching online	Yes	"it's a fun idea"

---

5. Performance Metrics

Metric	Result	Target
Initial load (fresh Gemini call)	Under 2 seconds	<15 seconds
Cached load (SQLite hit)	<1 second	<1 second
Chat response time	6-10 seconds	<15 seconds
Registration success rate	3/3 (100%)	100%
Task completion rate	21/21 tasks (100%)	100%
Group Watch results returned	2-4 results per search	1 or more results
Memory usage (Docker container, idle / light use)	~145 MiB RSS	<512 MiB (HF free tier headroom)
Memory usage (peak during Gemini call)	~165 MiB RSS	<512 MiB

Memory was measured via docker stats --no-stream movie-warnings-app-1 while issuing a sequence of /health, /api/search, /api/load_movie, /api/recommendations/<id>, and /api/content_twins/<id> — covering the ML-using endpoints as well as the original load path. The container holds a single gunicorn worker, the SQLite cache file (handled by the kernel page cache, not counted in RSS), a small numpy footprint from the comfort-distance recommender, and the loaded .pkl artifacts for the K-Means and MPA classifiers. The MiniLM sentence-transformer is lazy-loaded on first embedding-using request; once loaded, memory is essentially flat across subsequent requests. The original pre-ML measurement was ~138 MiB; the ~7 MiB rise reflects the loaded classifier .pkls and the embedding-cache module overhead.

---

5.1 Error Tracking

Current implementation (observability for the Hugging Face demo):

Failure mode	How it surfaces	Where to see it
Bad user input (missing query, invalid prompt, etc.)	Flask returns {"ok": false, "error": "..."} with a 4xx status	Frontend aria-live regions render the error inline; user is never shown a stack trace
TMDB / Gemini / OMDb API failure	Caught by try / except around each external call; returned to the user as {"ok": false, "error": "TMDB error: ..."}	Frontend alert(...) and inline error banners
Rate-limit (Gemini 429 / RESOURCE_EXHAUSTED)	call_gemini() does exponential backoff (5s × attempt, up to 4 attempts); after exhaustion returns the literal string "Error: Rate limit exceeded. Please wait a minute and try again."	Surfaces in the same way as any other Gemini error
Uncaught exception in a route	gunicorn returns HTTP 500; the exception + traceback prints to the worker's stderr	docker logs movie-warnings-app-1 locally, or the Logs tab on the Hugging Face Space
Pre-import config errors (missing API keys)	app.py raises at import time with a clear RuntimeError	gunicorn fails to start; visible in container logs immediately

Production plan (out of scope for the capstone demo):

• Structured logs: replace ad-hoc print(...) calls with app.logger.info/.error(...) using Python's logging module and JSON formatter, so logs are aggregatable.
• Error aggregation: add Sentry (sentry-sdk[flask]) to capture unhandled exceptions, link them to the failing request, and group recurring errors. Free tier covers MVP traffic.
• Request tracing: add a request-id middleware that stamps every log line and surfaces the id in error responses, so a user-reported failure can be traced to a specific request in seconds.
• Health monitoring: the /health endpoint exists; in production it would be polled by an uptime check (e.g. UptimeRobot, BetterStack) with a 1-minute interval.

---

6. Key Findings

Strengths

• Flashing lights warning prominently shown first
• Spoiler-safe chat answered questions about specific warnings ignored by MPA rating system
• Caching made repeat searches feel instant
• Group Watch Finder instructions were clear and helpful
• Editable AI prompt was a standout feature for technical users

Issues Identified

• Spoiler toggle hard to find — blended into the warning module
• Passwords too easy to make weak — no minimum requirements enforced
• Mood selection cannot be deselected once clicked
• Cached movie pool too small for diverse recommendations

---

7. Iterations and Changes Made

Issue Found	Change Made	Commit Reference
Spoiler toggle hard to find	Separated spoiler toggle into its own visible control	e98858b
Passwords too weak, no requirements	Raised minimum to 8 characters, switched to pbkdf2 hashing	2c0d4a9
Mood cannot be deselected	Added deselect behavior to mood buttons	e98858b
Cache too small for recommendations	Expanded cached movie pool, added bulk + direct seed scripts; --fix-ghosts mode re-seeds low-confidence rows	0564b2e, 3c01436
Recommendations "could be more diverse" (Andy)	Added sentence-transformer (all-MiniLM-L6-v2) re-ranking of recommendations + free-text mood matching on Discover; later broadened the recommendations candidate pool to union TMDB /similar with K-Means cluster neighbors so the severity filter has 16–86 cluster candidates to choose from instead of 0–3	14b784d, e56b79e
Horror surfaced for "comfort" mood queries	Contrastive mood scoring against Gemini's avoid_tones so opposite tones get penalized, not just non-matching ones	b08f2af
Trust in AI moderate ("how does AI decide?", Jenna + David)	Added classic-ML layers users can point to: K-Means content-twins (unsupervised) and an MPA-disagreement badge (supervised LR) on the movie page	7101bda, cc9c408, 651bb14

---

7.1 Post-Testing Additions

The user-testing round surfaced two recurring concerns that didn't have a single-fix answer:

1. "How does the AI decide?" — Multiple participants rated trustworthiness as only 3/5 even when they found the warnings useful. Gemini is a black box; users wanted something defensible underneath it.
2. "Recommendations could be more diverse" — Comfort-distance scoring alone (Euclidean over warning severities) was producing recommendations that felt thematically similar to the source film rather than offering genuine alternatives.

Both were addressed by adding two classic-ML layers and three supporting non-ML layers on top of the Gemini-generated warnings. To be precise about what's what:

Classic-ML layers (models trained on this project's data):

• K-Means content twins — unsupervised clustering in 9-dim warning-severity space. Two uses: (a) surfaces neighbors below each film as a "viewers who tolerated X also tolerated…" carousel; (b) broadens the candidate pool for the "Similar movies without…" recommender by adding every same-cluster film to the TMDB /similar set, so the severity filter has 16–86 candidates to choose from instead of 0–3.
• MPA-rating classifier — multinomial logistic regression, ~70% accuracy. Predicts a family / teen / adult bucket from the warning vector and surfaces a warning-styled badge when its confident prediction disagrees with the official TMDB rating. Acts as a sanity check on Gemini's severities.

Supporting layers (not classic ML, but part of the same effort):

• Sentence-transformer embeddings (all-MiniLM-L6-v2) — a pretrained deep-learning model used off-the-shelf for retrieval. We don't train it. Used to re-rank comfort-distance results by semantic similarity to the source film, and to power free-text mood matching on the Discover page.
• Contrastive mood scoring — a rule-based scoring heuristic on top of those embeddings. Penalizes films whose avoid_tones (per Gemini) overlap with the user's stated mood — this is what stopped horror films surfacing on "comfort" queries.
• Ghost-row repair (--fix-ghosts) — a data-quality CLI that re-asks Gemini for cached films returning the all-zero default response at 0.30 confidence, and only overwrites a row when the new response clears a confidence floor. Recovered 174 films. This is data engineering, not modeling.

A multilabel genre-classifier prototype (classic-ML, logistic-regression ensemble) was also trained and shipped briefly, but with macro F1 0.45 the predictions were too weak to defend to users and the feature was rolled back (dc8db96).

---

7.2 Quantified before / after for the post-testing changes

Each change was measured against a defined baseline. The numbers below are reproducible from the live data/movie_cache.db via data-engineering/compute_ml_metrics.py (filename predates a precision pass — the script measures classic-ML evaluation for MPA + K-Means alongside non-ML metrics for embedding rerank and --fix-ghosts recovery).

Ghost-row repair (--fix-ghosts)

Metric	Before fix-ghosts	After fix-ghosts	Δ
Films with avg confidence < 0.4 (ghost threshold)	174 / ~395	11 / 421	−163 films recovered
Recovery rate	—	93.7%
Films usable by the recommender (above floor)	~221	410	+85.5% larger eligible pool
Mean total severity (recovered films, 0–27 scale)	n/a (zeros)	6.46 (median 6)	meaningful warnings, not zeros

The 11 ghosts that did not recover are films Gemini consistently bails out on across multiple attempts — typically obscure or non-English titles. Acceptable residual.

MPA-disagreement classifier

Trained on 360 films with a labelable TMDB cert plus non-ghost warnings (family=99, teen=103, adult=158). Evaluated on a stratified 20% held-out test set (n=72).

Metric	Value
Test-set accuracy	72.2%
Macro F1	0.698
Weighted F1	0.709
Family-bucket F1	0.791
Teen-bucket F1	0.514 ← weakest
Adult-bucket F1	0.788

Confusion matrix (rows = actual TMDB bucket, cols = predicted by classifier from the warning vector alone):

	pred family	pred teen	pred adult
actual family	17	1	2
actual teen	5	9	7
actual adult	1	4	26

The teen bucket is the model's weak spot — it confuses teen with both family (recall miss) and adult (precision miss). This is consistent with the underlying signal: PG-13 films sit between the other two buckets in warning-severity space, so the classifier is most likely to mis-bin them. Family and adult predictions are far more reliable (F1 ≈ 0.79).

Badge-surfacing rate on the test set: 2 / 72 films (≈2.8%) cleared both the confidence floor (≥0.70) and the disagreement filter. The badge is therefore selective by design — most loads show no badge at all, and the ones that do represent the classifier's most confident disagreements.

K-Means content-twins clustering

Unsupervised, evaluated against standard cluster-quality metrics (no ground-truth labels available, by definition).

Metric	Value	Notes
Films clustered	410	All non-ghost films
Silhouette score	0.262	Above the 0.25 "meaningfully clustered" threshold for low-dim data
Mean within-cluster distance	1.46	In 9-dim severity space
Mean between-centroid distance	2.87
within / between ratio	0.51	Clusters are ~2× tighter than they are separated — usable for nearest-neighbor retrieval

Per-cluster sizes confirm the K-Means is finding real structure rather than splitting noise:

Cluster	Label	Size	Avg within-distance
0	Family-safe / low-content	116	0.95
1	Moderate action & intensity	113	1.49
2	Heavy mature content	39	2.16
3	Intense action & horror	79	1.62
4	Adult drama	63	1.67

The Family-safe cluster is the tightest (avg 0.95) — these films have low severity across every category, so they're packed close to the centroid. The Heavy-mature cluster is the loosest (avg 2.16) — fewer films but more variance in which mature categories dominate. Both behaviors match human expectation, which is one informal validation that the clustering is finding something real rather than a numerical artifact.

Sentence-transformer embedding re-ranking (recommendations)

Direct before/after measurement on 50 sampled anchor films. For each anchor, two top-5 recommendation sets were generated — one ranked by Euclidean comfort-distance over warning severities only (the pre-embedding baseline), one ranked by cosine similarity of the MiniLM embeddings (the current pipeline). Each set's semantic alignment to the anchor was then measured as the mean cosine similarity between the anchor's embedding and the recommended films' embeddings.

Metric	Comfort-distance only	+ Embedding re-rank	Δ
Mean cosine similarity (anchor ↔ top-5)	0.302	0.524	+0.222
Median	0.297	0.526	+0.229
Relative improvement	—	—	+73.3%

This is the quantified answer to Andy's testing feedback ("recommendations could be more diverse"). The comfort-distance recommender pre-embedding picked films that felt alike on severity numbers but were thematically random — average semantic similarity of 0.302 is basically noise. Adding embedding re-ranking lifts that to 0.524, meaning the recommended films now share real thematic context with the source film while still being filtered through the comfort-distance constraint. The user-facing effect is recommendations that read coherent rather than random.

---

7.3 What is still unmeasured

Two things the rubric notes as gaps remain unaddressed:

1. A second user-testing round to compare task-completion times before vs after the testing-driven changes from §7. The §7 changes ship with commit refs but not measured deltas because the testing budget was n=3, single round. A 2-person follow-up round measuring "find spoiler toggle" and "deselect mood" task times would close this.

2. Live MPA-badge precision against human-rated ground truth. The classifier's 72.2% accuracy is measured against the TMDB rating, which it was trained on. The actual claim of the badge — "Gemini's warning vector for this film disagrees with the official rating" — can only be evaluated by human review of the films the badge flags. A spot-check of the 2.8% of badge-surfacing films would tell us how often the badge is identifying a real Gemini severity miscalibration vs an artifact of the classifier's known teen-bucket weakness.

---

8. Conclusion

All three participants completed every task successfully. The app performed within its target response times and received consistently positive ratings across ease of use, usefulness, and trustworthiness. The most impactful findings were the sparsity of the cached movie database, discoverability of the spoiler toggle, the need for secure passwords, and the inability to deselect moods.

Future testing iterations would benefit from a larger participant pool and A/B testing of changes made.