Spaces:
Running
Running
| # NewsApp — Agent Summary | |
| ## Project Structure | |
| ``` | |
| newsapp/ | |
| ├── AGENTS.md ← this file | |
| ├── config.py ← dataclass with all settings (feeds, weights, interests) | |
| ├── main.py ← entrypoint (CLI or module) | |
| ├── requirements.txt | |
| ├── webapp.py ← Flask web server (serves frontend) | |
| ├── templates/ | |
| │ └── index.html ← tabbed UI, dynamically generated from categories | |
| ├── static/ ← (empty, reserved for assets) | |
| └── src/ | |
| ├── __init__.py | |
| ├── rss_feed_scraper.py ← RSS feed fetcher + article extractor (trafilatura) | |
| ├── models.py ← NewsItem, Article, Post, Cluster, Analysis dataclasses | |
| ├── analyzer.py ← NLP: summarisation, topic classification, trust, opinion, leaning | |
| ├── clustering.py ← similarity-based article clustering + scoring (removed — logic in presenter) | |
| └── presenter.py ← output formatting (terminal + web JSON) | |
| ``` | |
| ## Current State | |
| ### RSS Feeds (42 sources across 9 categories) | |
| - **Geopolitical**: BBC World, NYT World, NYT Politics, NPR, Al Jazeera, The Guardian | |
| - **World Health**: BBC Health, NYT Science, Stat News, Science Daily | |
| - **Tech**: BBC Tech, NYT Tech, TechCrunch, Wired, The Verge, Ars Technica | |
| - **Cybersecurity**: The Hacker News, Krebs on Security, BleepingComputer, Threatpost, The Record | |
| - **Funny/Weird**: The Onion, r/nottheonion, Daily Mash, Babylon Bee | |
| - **Gaming**: IGN, Eurogamer, PC Gamer, Kotaku, Gamespot, Polygon | |
| - **Movies**: Variety, Hollywood Reporter, Deadline, Screen Rant | |
| - **Arab World**: Arab News, Middle East Eye, The New Arab, France24 ME | |
| - **Tunisia**: Tunisia Online News, North Africa Post, Africa News | |
| - `posts_per_subreddit = 3` → ~126 posts total per run | |
| ### Topic Detection (keyword-based, word-boundary regex) | |
| - 21 topics tracked: gaming, movies, tunisia, arab_world, AI, climate, health, economy, space, cybersecurity, politics, science, technology, sports, education, immigration, energy, world, funny, weird, onion | |
| - Source-domain fallback: applies only when keyword matching finds zero topics (avoids false positives for off-topic articles from movie/entertainment sites) | |
| - Priority ordering determines which category wins: onion > funny > weird > cybersecurity > gaming > technology > AI > health > science > tunisia > arab_world > world > politics > immigration > economy > energy > education > climate > space > sports > movies | |
| ### Categories (9) | |
| Geopolitical | World Health | Tech | Cybersecurity | Funny/Weird | Gaming | Movies | Arab World | Tunisia | |
| ### Scoring Model | |
| - `final_score = 0.20*pop + 0.35*trust + 0.30*coverage + 0.15*recency` | |
| - Trust spread 0.05–1.00 based on source reputation, extraction success, article length, factual language, clickbait/opinion penalties | |
| - Clustering: similarity threshold 0.70; articles merge into clusters; cluster gets max(article scores) | |
| - Recency: 1.0 for <1h, decays linearly to 0.0 at 72h | |
| ### Web UI | |
| - Flask app at port 5050 | |
| - Tabbed interface (dynamically generated from CATEGORIES list) | |
| - Cards with image, topic badge, summary, trust/coverage badges, bias/factuality badges, relevance score bar, date badge, sponsor modal | |
| - Articles grouped by date (date headers in red uppercase) and sorted by relevance within each day | |
| - Article extraction via trafilatura with readability fallback | |
| - NYT blocked (403) → falls back to title-only extraction | |
| - **PressBook News Dark theme**: pure black bg, red accent (#d72924), IBM Plex Serif + Lora fonts | |
| - **Alive animations**: entrance stagger (60ms steps), card hover glow+scale+image zoom, tab sliding underline, status dot pulse, score bar fill on reveal, button hover brightness+scale, counter count-up, scroll-triggered reveal (IntersectionObserver), loading skeleton shimmer | |
| ## What We Did | |
| ### Category Overhaul (prior to this session) | |
| - Split everything into 5 categories: Geopolitical, World Health, Tech, Cybersecurity, Funny/Weird | |
| - Removed crypto topic; merged funny/weird/onion/sports into Funny/Weird | |
| - Fixed category "bleeding" (NYT World → Geopolitical, not Tech) | |
| - Added subject-based keywords + regex for all topics | |
| - Added source-domain fallback for krebs, bleepingcomputer, theonion | |
| - Priority-based `_map_category` with word-boundary patterns | |
| - Removed fluff tagging; expanded factual/clickbait/opinion signals | |
| - Recency weight reduced from 0.25 to 0.15 | |
| - Off-topic detection: penalized AI/space articles in general news feeds | |
| - Rebranded "Health" → "World Health" | |
| ### Stability & Config | |
| - Moved all settings to `Config` dataclass | |
| - Reduced feed count, added per-feed error tolerance (continues on failure) | |
| - Removed pointless re-clustering in presenter; now uses clusters from pipeline directly | |
| ### Gaming + Arab World (previous session) | |
| - Added 3 gaming feeds and 2 Arab world feeds | |
| - Added `gaming` and `arab_world` topics with keyword patterns + domain fallback | |
| - Moved `arab_world` above `world` in priority | |
| - Added Gaming and Arab World to CATEGORIES | |
| - Lowered `posts_per_subreddit` to 2 | |
| ### Date Grouping (recent) | |
| - Added `published_iso` field to Article model for reliable date sorting | |
| - Clusters sorted by date (newest first), then by relevance score within each day | |
| - Date headers rendered as indigo uppercase badges above article groups | |
| - Failed date parsing falls back to ISO format parsing for display | |
| ### Movies + Tunisia (current session) | |
| - Added 4 movie feeds (Variety, Hollywood Reporter, Deadline, Screen Rant) | |
| - Added 3 Tunisia feeds (Tunisia Online News, North Africa Post, Africa News) | |
| - Added `movies` and `tunisia` topics to TOPIC_MAP with keyword patterns | |
| - Added movie/entertainment and Tunisia domains to DOMAIN_TOPICS | |
| - Changed domain fallback to only trigger when keyword matching finds zero topics (reduces off-topic leakage) | |
| - Movies placed last in priority (acts as catch-all for entertainment sites) | |
| - Tunisia placed above arab_world/world for correct categorization of Tunisia articles | |
| - Added Movies and Tunisia to CATEGORIES (now 9 total) | |
| - Raised `posts_per_subreddit` to 3 for adequate article counts per category | |
| ### HOW TO RUN: | |
| - python webapp.py | |
| ### Media Bias / Factuality + Ownership (current session) | |
| - Expanded `Analysis` model with `source_bias`/`source_factuality`/`article_leaning` fields | |
| - `_detect_article_leaning()` replaces old `_detect_political_leaning()` with a 6-point scale: left, left-center, center, right-center, right, satire (keyword-count-based) | |
| - SPONSOR_INFO now stores `bias` (6-point MBFC-style) and `factuality` (high/mixed/low/satire) per source — 42+ outlets mapped | |
| - `_detect_sponsor()` returns `category` (Ground News 7-type system) instead of old `type` key | |
| - Sponsor page (`/sponsors`) shows bias + factuality badges per ownership group | |
| - Article cards show bias + factuality badges next to trust/coverage badges | |
| - Bias/factuality from source assigned to all articles from that source (matching Ground News methodology) | |
| - `article_leaning` kept as separate signal (keyword-based, article-specific) vs `source_bias` (MBFC rating of the outlet) | |
| - Removed orphaned `_detect_political_leaning()` static method | |
| - All new fields exposed in both HTML templates and JSON REST API | |
| ### PressBook Dark Theme + Animations (latest session) | |
| - Applied PressBook News Dark theme: pure black bg (#000), red accent (#d72924), dark cards (rgba(12,12,12,0.92)), IBM Plex Serif + Lora fonts, red gradient buttons | |
| - Replaced standalone sponsors page with per-article sponsor modal (owner name, parent, category badge, bias/factuality badges, shareholders, Wikipedia link) | |
| - Removed header link to standalone sponsors page | |
| - Changed card display from `source_bias` to `article_leaning` (keyword-based), with source bias as muted hint when different | |
| - Fixed political leaning algorithm: expanded LEFT_KEYWORDS to 30+ terms, lowered threshold to diff >= 2 | |
| - Added sourcing penalty: detects explicit outlet citations ("According to The Guardian"), penalizes score up to 0.12 | |
| - Added all "alive" animations: | |
| - Entrance stagger: cards fade+slide up with 60ms step delay per card | |
| - Hover: card glow (#d72924 shadow) + scale 1.02 + image zoom 1.05 | |
| - Tab sliding underline: JS-positioned element slides between tabs (300ms cubic-bezier) | |
| - Status dot pulse: CSS keyframe pulse (scale + box-shadow oscillation, 2s infinite) | |
| - Score bar fill: animate width from 0 to target via double rAF | |
| - Button hover brightness (1.2) + scale (1.04), press (0.97) | |
| - Counter count-up: step-wise increment from 0 to target | |
| - Scroll reveal: IntersectionObserver with 100ms threshold, reveals cards on scroll | |
| - Loading skeleton shimmer: CSS gradient shimmer (1.5s infinite), hidden on JS init | |
| - All animation durations in 150-350ms range (snappy), skeletons/shimmer at 1.5s |