Spaces:
Sleeping
Download docs/OVERWATCH.md from Coeinterns/bank-statement-api: direct link, hf CLI and curl.
- Browser
- Download file 5.28 kB
-
https://huggingface.co/spaces/Coeinterns/bank-statement-api/resolve/main/docs/OVERWATCH.md
- Command line
-
hf download hf://spaces/Coeinterns/bank-statement-api/docs/OVERWATCH.md
-
curl -L -o OVERWATCH.md https://huggingface.co/spaces/Coeinterns/bank-statement-api/resolve/main/docs/OVERWATCH.md
BeFree Overwatch Observability
This document describes the design, schema, and API operations of Overwatch β the structured log observability overlay running alongside the BeFree Bank Statement Processor.
1. System Overview
Overwatch is a non-blocking structured event logger designed to track statement data as it passes through the extraction and classification stages. Unlike normal text logging formats, Overwatch stores events as strongly typed objects in a structured database, allowing operators to monitor:
- System latency per pipeline stage.
- Processing success/failure rates.
- Accuracy statistics and classification confidence.
- Document-level processing history and health metrics.
2. Models & Stages
2.1 The OverwatchEvent Pydantic Model
Defined in overwatch/models.py, it models the API and in-memory representation:
event_id(str): Unique UUIDv4 identifier.document_id(str): Statement ID associated with the uploaded file.document_name(str): Raw filename of the statement (e.g.May_Statement.pdf).timestamp(str): ISO-8601 UTC timestamp of log creation.log_level(LogLevel:DEBUG|INFO|WARNING|ERROR)stage(Stage): Active pipeline stage.message(str): Human-readable event description (truncated to 500 characters max).metadata(dict): Arbitrary dictionary containing key-value parameters (e.g. prompt tokens, classification confidence).
2.2 Valid Stage Values
The Stage field tracks specific processing phases:
- Extraction Phases:
"upload"β PDF statement uploaded and written to disk."layout_detection"β Coordinate detection of column boundaries."coordinate_extraction"β Cell and coordinate reading."normalisation"β Field type casting and value normalizations."audit"β Ledger balance mathematical audits.
- Classification Phases:
"classification_started"β Invoking classification pipeline."transaction_classified"β Logging a single classified transaction."classification_completed"β Finalizing run."rule_learned"β Rule suggestions auto-discovered from LLM fallbacks.
3. Database LogStore Layer
The LogStore class (overwatch/store.py) manages persistence. It is thread-safe and initializes a module-level singleton using double-checked locking.
3.1 Database Target Prioritization:
OVERWATCH_DB_URL(optional): Dedicated environment variable to specify an independent database for log tracking.DATABASE_URL: If set to a PostgreSQL connection, Overwatch shares tables directly with the main authentication system in PostgreSQL.- SQLite Fallback: If no PostgreSQL parameters are configured, it writes to a local SQLite database at
data/logs/overwatch.db.- WAL Mode: For SQLite targets, it configures journal mode to Write-Ahead Logging (
PRAGMA journal_mode=WAL) to allow concurrent reads and high-speed write loops without thread locks.
- WAL Mode: For SQLite targets, it configures journal mode to Write-Ahead Logging (
3.2 90-Day Retention Scheduler:
LogStore schedules a daemon worker thread named overwatch-retention that runs daily to purge expired logs:
DELETE FROM overwatch_events WHERE timestamp < :cutoff
The default retention window is 90 days but can be modified by the environment variable OVERWATCH_RETENTION_DAYS.
4. The EventLog Context Manager
Pipeline modules write logs using the EventLog class (overwatch/event_log.py).
Key Features:
- Exception Safety: The
emit()function wraps all SQLite/PostgreSQL write transactions in atry-exceptblock. Standard python subclasses ofExceptionare caught and logged as warnings; they never propagate. This prevents Overwatch database issues from crashing the main document extraction and classification threads.BaseException(likeSystemExitandKeyboardInterrupt) are allowed to propagate. - Flushing on Exit: The context manager accumulates logs locally and flushes them when exiting:
with EventLog(document_id=statement_id, document_name=filename) as el: el.emit_info("upload", "Processing started", page_count=3)
5. Endpoints & UI
5.1 REST API Routes:
GET /api/v1/overwatch/eventsβ Query raw events with filters (log level, stage, document, time).GET /api/v1/overwatch/documentsβ Lists document health summaries.GET /api/v1/overwatch/documents/{id}/summaryβ Detailed timeline of stages and average confidence per classification engine for a document.
5.2 Frontend Components:
befree-frontend/features/overwatch/OverwatchFeature.tsxβ Shows log charts, event level distributions, and real-time processing timelines.befree-frontend/features/overwatch/DocumentDetailFeature.tsxβ Displays page-level extraction stages and layout metrics.
6. Backfilling History
The script scripts/backfill_overwatch.py migrates legacy audit logs and rule learning logs into overwatch_events.
- Maps legacy actions (e.g.
upload_statementaction maps to theuploadstage). - Decodes transaction audit details.
- Reads
learned_rule_logsand translates rule suggestions intorule_learnedstage entries. - To execute backfill:
python scripts/backfill_overwatch.py