bank-statement-api / docs /OVERWATCH.md
3v324v23's picture
Merge Conflict solved | Build error solved
205f25f
|
Raw History Blame Contribute Delete
5.28 kB

BeFree Overwatch Observability

This document describes the design, schema, and API operations of Overwatch β€” the structured log observability overlay running alongside the BeFree Bank Statement Processor.


1. System Overview

Overwatch is a non-blocking structured event logger designed to track statement data as it passes through the extraction and classification stages. Unlike normal text logging formats, Overwatch stores events as strongly typed objects in a structured database, allowing operators to monitor:

  • System latency per pipeline stage.
  • Processing success/failure rates.
  • Accuracy statistics and classification confidence.
  • Document-level processing history and health metrics.

2. Models & Stages

2.1 The OverwatchEvent Pydantic Model

Defined in overwatch/models.py, it models the API and in-memory representation:

  • event_id (str): Unique UUIDv4 identifier.
  • document_id (str): Statement ID associated with the uploaded file.
  • document_name (str): Raw filename of the statement (e.g. May_Statement.pdf).
  • timestamp (str): ISO-8601 UTC timestamp of log creation.
  • log_level (LogLevel: DEBUG | INFO | WARNING | ERROR)
  • stage (Stage): Active pipeline stage.
  • message (str): Human-readable event description (truncated to 500 characters max).
  • metadata (dict): Arbitrary dictionary containing key-value parameters (e.g. prompt tokens, classification confidence).

2.2 Valid Stage Values

The Stage field tracks specific processing phases:

  1. Extraction Phases:
    • "upload" β€” PDF statement uploaded and written to disk.
    • "layout_detection" β€” Coordinate detection of column boundaries.
    • "coordinate_extraction" β€” Cell and coordinate reading.
    • "normalisation" β€” Field type casting and value normalizations.
    • "audit" β€” Ledger balance mathematical audits.
  2. Classification Phases:
    • "classification_started" β€” Invoking classification pipeline.
    • "transaction_classified" β€” Logging a single classified transaction.
    • "classification_completed" β€” Finalizing run.
    • "rule_learned" β€” Rule suggestions auto-discovered from LLM fallbacks.

3. Database LogStore Layer

The LogStore class (overwatch/store.py) manages persistence. It is thread-safe and initializes a module-level singleton using double-checked locking.

3.1 Database Target Prioritization:

  • OVERWATCH_DB_URL (optional): Dedicated environment variable to specify an independent database for log tracking.
  • DATABASE_URL: If set to a PostgreSQL connection, Overwatch shares tables directly with the main authentication system in PostgreSQL.
  • SQLite Fallback: If no PostgreSQL parameters are configured, it writes to a local SQLite database at data/logs/overwatch.db.
    • WAL Mode: For SQLite targets, it configures journal mode to Write-Ahead Logging (PRAGMA journal_mode=WAL) to allow concurrent reads and high-speed write loops without thread locks.

3.2 90-Day Retention Scheduler:

LogStore schedules a daemon worker thread named overwatch-retention that runs daily to purge expired logs:

DELETE FROM overwatch_events WHERE timestamp < :cutoff

The default retention window is 90 days but can be modified by the environment variable OVERWATCH_RETENTION_DAYS.


4. The EventLog Context Manager

Pipeline modules write logs using the EventLog class (overwatch/event_log.py).

Key Features:

  • Exception Safety: The emit() function wraps all SQLite/PostgreSQL write transactions in a try-except block. Standard python subclasses of Exception are caught and logged as warnings; they never propagate. This prevents Overwatch database issues from crashing the main document extraction and classification threads. BaseException (like SystemExit and KeyboardInterrupt) are allowed to propagate.
  • Flushing on Exit: The context manager accumulates logs locally and flushes them when exiting:
    with EventLog(document_id=statement_id, document_name=filename) as el:
        el.emit_info("upload", "Processing started", page_count=3)
    

5. Endpoints & UI

5.1 REST API Routes:

  • GET /api/v1/overwatch/events β€” Query raw events with filters (log level, stage, document, time).
  • GET /api/v1/overwatch/documents β€” Lists document health summaries.
  • GET /api/v1/overwatch/documents/{id}/summary β€” Detailed timeline of stages and average confidence per classification engine for a document.

5.2 Frontend Components:

  • befree-frontend/features/overwatch/OverwatchFeature.tsx β€” Shows log charts, event level distributions, and real-time processing timelines.
  • befree-frontend/features/overwatch/DocumentDetailFeature.tsx β€” Displays page-level extraction stages and layout metrics.

6. Backfilling History

The script scripts/backfill_overwatch.py migrates legacy audit logs and rule learning logs into overwatch_events.

  • Maps legacy actions (e.g. upload_statement action maps to the upload stage).
  • Decodes transaction audit details.
  • Reads learned_rule_logs and translates rule suggestions into rule_learned stage entries.
  • To execute backfill:
    python scripts/backfill_overwatch.py