tools/CRON_README.md

Knowledge-base refresh schedule

This project's knowledge base — policy PDFs, premium anchors, insurer reviews, all source URLs — would slowly rot without automation. Insurer sites change filenames, IRDAI replaces annual reports, aggregator ratings move. The schedule below keeps everything fresh without us touching it.

What "cron" means

"Cron" is a unix term for a job that runs on a fixed schedule, unattended. The name comes from chronos (Greek for "time"). The original cron daemon on Linux servers reads a table of min hour day month weekday command rows and fires each command at its scheduled time.

On macOS the modern equivalent is launchd, configured by XML plist files under ~/Library/LaunchAgents/. A LaunchAgent is essentially a cron entry — it tells the OS "wake up at X, run this command, log the output." The macOS scheduler honors these even when the laptop is asleep (it queues them and runs at the next wake), and respects per-user permissions.

This project uses three LaunchAgents + one manual script. They run themselves in the background. You don't have to remember anything.

The four jobs

Frequency	Job	Plist label	What it does	Auto-fix
Daily 03:00	Link-rot	com.rohit.insurancebot.linkrot	HEAD every external URL in the KB	Wayback Machine snapshot; http↔https swap; query-string strip
Weekly Sun 04:00	PDF freshness	com.rohit.insurancebot.pdfetags	ETag/Last-Modified diff for every policy PDF	Re-download + delete-from-Chroma + re-ingest
Monthly 1st 06:00	Premium refresh	com.rohit.insurancebot.premiums	Re-fetch every premium anchor; detect price drift; HEAD aggregator URLs	Wayback if dead; >15% drift goes to MUST_FIX
Quarterly (manual)	Full rebuild	bash tools/quarterly_rebuild.sh	Wipe Chroma, re-ingest everything, re-extract schema, rebuild KB pages, run gold eval	—

Auto-fix philosophy

Every cron job is detect → auto-fix → report:

1. Detect — record the issue with structured logs in ~/Library/Logs/insurance-bot/
2. Auto-fix — try every safe repair strategy applicable to that error class
3. Report — if and only if no fix worked, write to MUST_FIX.md at the project root and post a macOS notification

Things we auto-fix without asking:

• Dead URLs → swap to working Wayback snapshot
• Stale PDFs → re-download + re-ingest
• Cosmetic URL drift (http↔https, query-string ambiguity)

Things we don't auto-fix (logged to MUST_FIX.md instead):

• Premium price drift > 15% — a real rate change needs human eyeballs
• PDF re-ingest crashes — could mask a schema regression
• Hosts that block bots entirely — manually verify via Playwright

MUST_FIX.md is regenerated every run, so it always reflects current state. If it's missing, everything is clean.

What gets logged

~/Library/Logs/insurance-bot/
├── link_rot.log               # JSON-per-URL audit trail (append-only)
├── link_rot.stdout.log        # Summary stdout
├── link_rot.stderr.log        # Any crashes
├── pdf_etags.log              # ETag history per policy URL
├── premium_refresh.log        # Price-drift detail
└── com.rohit.insurancebot.*.{out,err}.log  # launchd's own view

Tail any log with tail -f to watch live.

Operations

# Trigger any job manually
launchctl start com.rohit.insurancebot.linkrot
launchctl start com.rohit.insurancebot.pdfetags
launchctl start com.rohit.insurancebot.premiums

# Run the quarterly full rebuild (interactive)
bash tools/quarterly_rebuild.sh

# Check which agents are loaded + last exit code
launchctl list | grep com.rohit.insurancebot

# Disable a job
launchctl unload ~/Library/LaunchAgents/com.rohit.insurancebot.linkrot.plist

# Re-enable / re-install everything
bash tools/install_crons.sh

Why scripts live in ~/Library/Scripts/ not the project

macOS Transparency, Consent, and Control (TCC) blocks launchd-spawned shells from reading files in ~/Documents/ unless Full Disk Access is granted. The wrapper scripts are installed under ~/Library/Scripts/insurance-bot/ which TCC doesn't gate. Those wrappers cd into the project and invoke the project's venv Python (which itself isn't TCC-restricted because it lives inside a user-owned project directory).

This is the same lesson learned from the AI Thesis project's cache-sweeper incident (May 2026) — LaunchAgent scripts under ~/Documents/ silently fail with exit code 126 ("Operation not permitted") and never run.

Why the venv Python, not the system Python

The system python3 at /Library/Developer/CommandLineTools/usr/bin/python3 is itself TCC-restricted when invoked from launchd context. The project's .venv/bin/python3 is not (it's a binary inside the user's project, owned by the user, not under any system path). Plus the venv has httpx already installed — system Python doesn't.

First-time install

cd "Insurance Sales Bot"
bash tools/install_crons.sh

The installer:

1. Writes wrapper shell scripts to ~/Library/Scripts/insurance-bot/
2. Writes LaunchAgent plists to ~/Library/LaunchAgents/
3. Loads each agent with launchctl load
4. Runs a TCC self-test by firing the link-rot job and checking stderr
5. If TCC is blocking, opens the relevant System Settings pane and exits 2