AGENTS.md

NewsApp — Agent Summary

Project Structure

newsapp/
├── AGENTS.md               ← this file
├── config.py               ← dataclass with all settings (feeds, weights, interests)
├── main.py                 ← entrypoint (CLI or module)
├── requirements.txt
├── webapp.py               ← Flask web server (serves frontend)
├── templates/
│   └── index.html          ← tabbed UI, dynamically generated from categories
├── static/                 ← (empty, reserved for assets)
└── src/
    ├── __init__.py
    ├── rss_feed_scraper.py ← RSS feed fetcher + article extractor (trafilatura)
    ├── models.py           ← NewsItem, Article, Post, Cluster, Analysis dataclasses
    ├── analyzer.py         ← NLP: summarisation, topic classification, trust, opinion, leaning
    ├── clustering.py       ← similarity-based article clustering + scoring (removed — logic in presenter)
    └── presenter.py        ← output formatting (terminal + web JSON)

Current State

RSS Feeds (42 sources across 9 categories)

• Geopolitical: BBC World, NYT World, NYT Politics, NPR, Al Jazeera, The Guardian
• World Health: BBC Health, NYT Science, Stat News, Science Daily
• Tech: BBC Tech, NYT Tech, TechCrunch, Wired, The Verge, Ars Technica
• Cybersecurity: The Hacker News, Krebs on Security, BleepingComputer, Threatpost, The Record
• Funny/Weird: The Onion, r/nottheonion, Daily Mash, Babylon Bee
• Gaming: IGN, Eurogamer, PC Gamer, Kotaku, Gamespot, Polygon
• Movies: Variety, Hollywood Reporter, Deadline, Screen Rant
• Arab World: Arab News, Middle East Eye, The New Arab, France24 ME
• Tunisia: Tunisia Online News, North Africa Post, Africa News
• posts_per_subreddit = 3 → ~126 posts total per run

Topic Detection (keyword-based, word-boundary regex)

• 21 topics tracked: gaming, movies, tunisia, arab_world, AI, climate, health, economy, space, cybersecurity, politics, science, technology, sports, education, immigration, energy, world, funny, weird, onion
• Source-domain fallback: applies only when keyword matching finds zero topics (avoids false positives for off-topic articles from movie/entertainment sites)
• Priority ordering determines which category wins: onion > funny > weird > cybersecurity > gaming > technology > AI > health > science > tunisia > arab_world > world > politics > immigration > economy > energy > education > climate > space > sports > movies

Categories (9)

Geopolitical | World Health | Tech | Cybersecurity | Funny/Weird | Gaming | Movies | Arab World | Tunisia

Scoring Model

• final_score = 0.20*pop + 0.35*trust + 0.30*coverage + 0.15*recency
• Trust spread 0.05–1.00 based on source reputation, extraction success, article length, factual language, clickbait/opinion penalties
• Clustering: similarity threshold 0.70; articles merge into clusters; cluster gets max(article scores)
• Recency: 1.0 for <1h, decays linearly to 0.0 at 72h

Web UI

• Flask app at port 5050
• Tabbed interface (dynamically generated from CATEGORIES list)
• Cards with image, topic badge, summary, trust/coverage badges, bias/factuality badges, relevance score bar, date badge, sponsor modal
• Articles grouped by date (date headers in red uppercase) and sorted by relevance within each day
• Article extraction via trafilatura with readability fallback
• NYT blocked (403) → falls back to title-only extraction
• PressBook News Dark theme: pure black bg, red accent (#d72924), IBM Plex Serif + Lora fonts
• Alive animations: entrance stagger (60ms steps), card hover glow+scale+image zoom, tab sliding underline, status dot pulse, score bar fill on reveal, button hover brightness+scale, counter count-up, scroll-triggered reveal (IntersectionObserver), loading skeleton shimmer

What We Did

Category Overhaul (prior to this session)

• Split everything into 5 categories: Geopolitical, World Health, Tech, Cybersecurity, Funny/Weird
• Removed crypto topic; merged funny/weird/onion/sports into Funny/Weird
• Fixed category "bleeding" (NYT World → Geopolitical, not Tech)
• Added subject-based keywords + regex for all topics
• Added source-domain fallback for krebs, bleepingcomputer, theonion
• Priority-based _map_category with word-boundary patterns
• Removed fluff tagging; expanded factual/clickbait/opinion signals
• Recency weight reduced from 0.25 to 0.15
• Off-topic detection: penalized AI/space articles in general news feeds
• Rebranded "Health" → "World Health"

Stability & Config

• Moved all settings to Config dataclass
• Reduced feed count, added per-feed error tolerance (continues on failure)
• Removed pointless re-clustering in presenter; now uses clusters from pipeline directly

Gaming + Arab World (previous session)

• Added 3 gaming feeds and 2 Arab world feeds
• Added gaming and arab_world topics with keyword patterns + domain fallback
• Moved arab_world above world in priority
• Added Gaming and Arab World to CATEGORIES
• Lowered posts_per_subreddit to 2

Date Grouping (recent)

• Added published_iso field to Article model for reliable date sorting
• Clusters sorted by date (newest first), then by relevance score within each day
• Date headers rendered as indigo uppercase badges above article groups
• Failed date parsing falls back to ISO format parsing for display

Movies + Tunisia (current session)

• Added 4 movie feeds (Variety, Hollywood Reporter, Deadline, Screen Rant)
• Added 3 Tunisia feeds (Tunisia Online News, North Africa Post, Africa News)
• Added movies and tunisia topics to TOPIC_MAP with keyword patterns
• Added movie/entertainment and Tunisia domains to DOMAIN_TOPICS
• Changed domain fallback to only trigger when keyword matching finds zero topics (reduces off-topic leakage)
• Movies placed last in priority (acts as catch-all for entertainment sites)
• Tunisia placed above arab_world/world for correct categorization of Tunisia articles
• Added Movies and Tunisia to CATEGORIES (now 9 total)
• Raised posts_per_subreddit to 3 for adequate article counts per category

HOW TO RUN:

• python webapp.py

Media Bias / Factuality + Ownership (current session)

• Expanded Analysis model with source_bias/source_factuality/article_leaning fields
• _detect_article_leaning() replaces old _detect_political_leaning() with a 6-point scale: left, left-center, center, right-center, right, satire (keyword-count-based)
• SPONSOR_INFO now stores bias (6-point MBFC-style) and factuality (high/mixed/low/satire) per source — 42+ outlets mapped
• _detect_sponsor() returns category (Ground News 7-type system) instead of old type key
• Sponsor page (/sponsors) shows bias + factuality badges per ownership group
• Article cards show bias + factuality badges next to trust/coverage badges
• Bias/factuality from source assigned to all articles from that source (matching Ground News methodology)
• article_leaning kept as separate signal (keyword-based, article-specific) vs source_bias (MBFC rating of the outlet)
• Removed orphaned _detect_political_leaning() static method
• All new fields exposed in both HTML templates and JSON REST API

PressBook Dark Theme + Animations (latest session)

• Applied PressBook News Dark theme: pure black bg (#000), red accent (#d72924), dark cards (rgba(12,12,12,0.92)), IBM Plex Serif + Lora fonts, red gradient buttons
• Replaced standalone sponsors page with per-article sponsor modal (owner name, parent, category badge, bias/factuality badges, shareholders, Wikipedia link)
• Removed header link to standalone sponsors page
• Changed card display from source_bias to article_leaning (keyword-based), with source bias as muted hint when different
• Fixed political leaning algorithm: expanded LEFT_KEYWORDS to 30+ terms, lowered threshold to diff >= 2
• Added sourcing penalty: detects explicit outlet citations ("According to The Guardian"), penalizes score up to 0.12
• Added all "alive" animations:
  • Entrance stagger: cards fade+slide up with 60ms step delay per card
  • Hover: card glow (#d72924 shadow) + scale 1.02 + image zoom 1.05
  • Tab sliding underline: JS-positioned element slides between tabs (300ms cubic-bezier)
  • Status dot pulse: CSS keyframe pulse (scale + box-shadow oscillation, 2s infinite)
  • Score bar fill: animate width from 0 to target via double rAF
  • Button hover brightness (1.2) + scale (1.04), press (0.97)
  • Counter count-up: step-wise increment from 0 to target
  • Scroll reveal: IntersectionObserver with 100ms threshold, reveals cards on scroll
  • Loading skeleton shimmer: CSS gradient shimmer (1.5s infinite), hidden on JS init
• All animation durations in 150-350ms range (snappy), skeletons/shimmer at 1.5s