docs/explanation/architecture.md

Architecture Explanation

1. Executive summary

classifier-general is a single FastAPI service that classifies text and files with local extraction/preprocessing, a local Hugging Face zero-shot NLI model, and local language detection.

Evidence:

• app/main.py
• app/routers/classification.py
• app/services/classifier_service.py
• app/services/language_service.py

2. Purpose and scope

What exists

• Contract-compatible endpoints for classify/language/transform flows.
• Pipeline split into preprocess, extraction, classification orchestration.
• Configurable runtime through environment variables.

Evidence:

• app/routers/classification.py
• app/pipelines/classification_pipeline.py
• app/core/config.py

How it works

• Router accepts JSON or multipart requests.
• Files are written to disk (static/uploads).
• Extraction service parses document/image/text into plain text.
• Text preprocessing enforces minimum quality.
• Pipeline calls language and classifier services.

Evidence:

• app/services/file_storage_service.py
• app/services/extraction_service.py
• app/pipelines/text_pipeline.py
• app/pipelines/classification_pipeline.py

Why designed this way (inferred)

• Maintain old API contract while introducing modular services and safer config handling.

3. C4-style views

Context view

Actors/systems:

• API client sending text/files.
• Hugging Face model hub (model download/auth when needed).
• Local filesystem for uploaded files.

Evidence:

• app/core/config.py
• app/services/classifier_service.py
• app/services/language_service.py

Container view

• One container/service (classifier-api) with FastAPI + OCR binary.

Evidence:

• docker-compose.yml
• Dockerfile

Component view

• API routing: app/routers/*
• Orchestration pipelines: app/pipelines/*
• Integration services: app/services/classifier_service.py, app/services/language_service.py
• Extraction + storage services: app/services/extraction_service.py, app/services/file_storage_service.py
• Config/exceptions/schemas: app/core/*, app/schemas/*

Code-level workflow: file classification

1. POST /classify receives file.
2. File saved to upload directory.
3. Text extracted by extension-specific handlers.
   • For /classify, PDF extraction is first-page only.
4. Text preprocessed (regex cleanup + min words).
5. Local language detector called.
6. Zero-shot NLI classifier scores runtime labels and selects top label.
7. Response returns {label, language} plus type=not english when applicable.

Evidence:

• app/routers/classification.py
• app/services/file_storage_service.py
• app/services/extraction_service.py
• app/pipelines/text_pipeline.py
• app/pipelines/classification_pipeline.py

4. Cross-cutting concerns

Validation and error mapping

• Input schemas use strict extra=forbid.
• Error mapping explicitly separates validation/extraction (400) from classifier/language inference failures (502).

Evidence:

• app/schemas/classification.py
• app/routers/classification.py

Configuration and secrets

• Runtime config sourced from env.
• HF token optional and no hardcoded secret in current service code.

Evidence:

• app/core/config.py
• app/services/classifier_service.py

Concurrency and mutable state

• Labels guarded by thread lock (LabelConfig._lock).
• State is still process-local; multi-instance deployments can diverge.

Evidence:

• app/models/label_config.py
• app/services/label_service.py

Testing strategy

• Route contract tests monkeypatch pipeline methods for deterministic tests.
• Tests validate response shape and key endpoint behavior, not remote network calls.

Evidence:

• tests/test_routes.py

5. Risks, gaps, and technical debt

• Local model initialization can fail if model/token/resources are invalid.
• No upload retention/cleanup process.
• Readiness check does not probe external AI services, only local label readiness.
• No authentication/authorization layer on API endpoints.

Evidence:

• app/services/language_service.py
• app/services/classifier_service.py
• app/routers/health.py
• app/routers/classification.py

6. Unknown or inferred

• Unknown: expected SLA and acceptable latency.
• Unknown: intended persistence/retention policy for uploaded files.
• Inferred: service is optimized for local/dev contract compatibility and integration testing.