docs/ARCHITECTURE_REFACTOR.md

OMyFish — Architecture Refactor Plan

Current State Analysis

What exists

File	Concerns mixed inside
app/api.py	Routes + raw SQL + GeoJSON build + EXIF extraction + predictor loading
app/main.py (Streamlit)	UI + direct DB writes (raw SQL) + map rendering + EXIF extraction
app/database.py	Engine + DDL + SQLite/PostGIS dual-path
app/gis.py	Single function: extract_exif_gps()
app/clip_predictor.py	CLIP zero-shot AI predictor
src/	Training pipeline (model, dataset, train, evaluate, predict, transforms)

Key problems

1. No service layer — routes call predictors and hit the DB directly
2. No repository layer — raw SQL strings live inside routes and Streamlit pages
3. GIS is vestigial — one function; GeoJSON built inline in routes
4. Two predictors are disconnected — FishPredictor and CLIPFishPredictor have the same predict() interface but no shared contract
5. Config is hardcoded — DB URL, thresholds, checkpoint paths scattered across files
6. PostGIS already partially wired — geom GEOGRAPHY(POINT,4326) is in init_db() but created through raw DDL, no migrations

---

1. Target Folder Structure

omyfish-python/

├── apps/
│   ├── omyfish-api/                  # FastAPI backend
│   │   ├── main.py                   # App factory + middleware
│   │   ├── routes/
│   │   │   ├── observations.py
│   │   │   ├── species.py
│   │   │   ├── gis.py
│   │   │   └── health.py
│   │   ├── repositories/
│   │   │   ├── observation_repository.py
│   │   │   └── species_repository.py
│   │   └── db/
│   │       ├── engine.py             # SQLAlchemy engine setup
│   │       └── migrations/           # Alembic migrations
│   │
│   ├── omyfish-web/                  # Streamlit frontend
│   │   ├── main.py
│   │   └── pages/
│   │       ├── identify.py
│   │       └── map.py
│   │
│   └── omyfish-admin/               # Placeholder — future admin dashboard
│       └── __init__.py
│
├── services/
│   ├── fish-ai/                      # All ML logic
│   │   ├── service.py                # FishAIService (unified interface)
│   │   ├── predictors/
│   │   │   ├── base.py               # BaseFishPredictor ABC
│   │   │   ├── efficientnet.py       # FishPredictor (trained model)
│   │   │   └── clip.py               # CLIPFishPredictor (zero-shot)
│   │   ├── training/                 # move src/train.py, dataset.py, transforms.py here
│   │   │   ├── train.py
│   │   │   ├── dataset.py
│   │   │   ├── evaluate.py
│   │   │   └── transforms.py
│   │   ├── model/                    # move src/model.py here
│   │   │   └── classifier.py
│   │   └── tests/
│   │
│   ├── gis-service/                  # All geospatial logic
│   │   ├── service.py                # GISService
│   │   ├── exif.py                   # extract_exif_gps (move from app/gis.py)
│   │   ├── geojson.py                # GeoJSON builders
│   │   ├── spatial_queries.py        # PostGIS spatial queries
│   │   └── tests/
│   │
│   ├── ingestion-service/            # Placeholder — external data sources
│   │   ├── interfaces.py             # ObservationSource ABC
│   │   └── __init__.py
│   │
│   └── analytics-service/            # Placeholder — heatmaps, distributions
│       ├── interfaces.py             # AnalyticsService ABC
│       └── __init__.py
│
├── shared/
│   ├── models/
│   │   ├── observation.py            # SQLAlchemy ORM + Pydantic schemas
│   │   ├── species.py
│   │   ├── user.py
│   │   └── location.py
│   ├── schemas/
│   │   ├── observation.py            # Request/response Pydantic models
│   │   ├── prediction.py
│   │   └── geojson.py
│   ├── constants/
│   │   └── thresholds.py             # UNCERTAIN_THRESHOLD etc.
│   └── utils/
│       └── ids.py                    # new_id(), UUID helpers
│
├── data/
│   ├── raw/
│   ├── processed/
│   ├── training/
│   ├── metadata/
│   └── exports/
│
├── infrastructure/
│   ├── docker/
│   │   ├── docker-compose.yml        # move from root
│   │   ├── docker-compose.prod.yml
│   │   └── Dockerfile                # move from root
│   └── k8s/                          # Future Kubernetes manifests
│
├── configs/
│   ├── development.yaml
│   ├── production.yaml
│   └── training.yaml                 # move config.yaml here
│
├── docs/
│   └── ARCHITECTURE_REFACTOR.md     # this file
│
└── research/                         # notebooks/, scripts/
    ├── notebooks/
    └── scripts/

---

2. Architecture Diagram

┌─────────────────────────────────────────────────────────┐
│                     CLIENT LAYER                        │
│                                                         │
│   omyfish-web (Streamlit)    omyfish-admin (future)     │
└────────────────┬────────────────────────────────────────┘
                 │ HTTP
┌────────────────▼────────────────────────────────────────┐
│                    omyfish-api (FastAPI)                 │
│                                                         │
│  routes/           repositories/                        │
│  ├── observations  ├── ObservationRepository ──────────┐│
│  ├── species       └── SpeciesRepository               ││
│  ├── gis                                               ││
│  └── health                                            ││
└──────┬────────────────────────────────────────────────┬─┘
       │ calls                                          │ ORM
┌──────▼──────────────────┐          ┌─────────────────▼──┐
│    services/fish-ai     │          │   PostgreSQL        │
│                         │          │   + PostGIS         │
│  FishAIService          │          │                     │
│  ├── EfficientNet mode  │          │  observations       │
│  └── CLIP zero-shot     │          │  (geom GEOGRAPHY)   │
└─────────────────────────┘          └────────────────────┘
┌──────────────────────────┐
│   services/gis-service   │
│                          │
│  GISService              │
│  ├── EXIF extraction     │
│  ├── GeoJSON building    │
│  └── spatial queries ───────────────────────────────────┐
└──────────────────────────┘                              │
                                                          │
                                              PostGIS spatial
                                              functions (ST_*)

---

3. Domain Model Definitions

Observation (core entity)

# shared/models/observation.py

from uuid import UUID, uuid4
from datetime import datetime
from typing import Optional
from sqlalchemy import Column, String, Float, DateTime, Text
from sqlalchemy.dialects.postgresql import UUID as PG_UUID
from geoalchemy2 import Geography
from shared.db import Base

class Observation(Base):
    __tablename__ = "observations"

    id: UUID               = Column(PG_UUID(as_uuid=True), primary_key=True, default=uuid4)
    species_name: str      = Column(String, nullable=False)
    scientific_name: str   = Column(String)
    confidence: float      = Column(Float, nullable=False)
    timestamp: datetime    = Column(DateTime(timezone=True), default=datetime.utcnow)
    latitude: float        = Column(Float)
    longitude: float       = Column(Float)
    geom                   = Column(Geography("POINT", srid=4326))  # PostGIS
    image_url: str         = Column(Text)
    user_id: str           = Column(String)
    source: str            = Column(String, default="upload")

Species (knowledge entity)

# shared/models/species.py

from pydantic import BaseModel
from typing import Optional

class Species(BaseModel):
    name: str
    scientific_name: Optional[str]
    habitat: Optional[str]
    diet: Optional[str]
    max_size_cm: Optional[float]
    conservation_status: Optional[str]
    description: Optional[str]
    fun_fact: Optional[str]

Location (value object)

# shared/models/location.py

from pydantic import BaseModel
from typing import Optional

class Coordinates(BaseModel):
    latitude: float
    longitude: float

class GeoPoint(BaseModel):
    coordinates: Coordinates
    srid: int = 4326
    source: str  # "manual" | "exif" | "gps"

Prediction (value object — never persisted directly)

# shared/schemas/prediction.py

from pydantic import BaseModel
from typing import Optional, List
from shared.models.species import Species

class PredictionResult(BaseModel):
    species: str
    confidence: float
    metadata: Optional[Species]

class PredictionResponse(BaseModel):
    predictions: List[PredictionResult]
    uncertain: bool
    message: Optional[str]
    top_species: str         # convenience: predictions[0].species
    top_confidence: float    # convenience: predictions[0].confidence

Events (domain events)

# shared/events.py

from pydantic import BaseModel
from uuid import UUID
from datetime import datetime

class DomainEvent(BaseModel):
    event_id: UUID
    occurred_at: datetime

class ObservationCreated(DomainEvent):
    observation_id: UUID
    species_name: str
    latitude: float
    longitude: float
    source: str

class SpeciesPredicted(DomainEvent):
    species_name: str
    confidence: float
    uncertain: bool

class ObservationValidated(DomainEvent):
    observation_id: UUID
    validated_by: str
    is_correct: bool

---

4. Service Interfaces

Fish AI Service

# services/fish-ai/service.py

from abc import ABC, abstractmethod
from typing import List
from PIL import Image
from shared.schemas.prediction import PredictionResponse, PredictionResult
from shared.models.species import Species

class BaseFishPredictor(ABC):
    @abstractmethod
    def predict(self, image: Image.Image, top_k: int = 3) -> PredictionResponse:
        ...

class FishAIService:
    """
    Unified entry point for AI inference.
    Delegates to EfficientNet if checkpoint exists, CLIP otherwise.
    """

    def __init__(self, predictor: BaseFishPredictor):
        self._predictor = predictor

    def predict_species(self, image: Image.Image) -> PredictionResponse:
        return self._predictor.predict(image, top_k=1)

    def get_top_predictions(self, image: Image.Image, top_k: int = 3) -> PredictionResponse:
        return self._predictor.predict(image, top_k=top_k)

    def get_species_info(self, species_name: str) -> Optional[Species]:
        # Looks up the knowledge base — no model inference needed
        ...

GIS Service

# services/gis-service/service.py

from abc import ABC
from typing import Optional, List
from PIL import Image
from shared.models.location import Coordinates, GeoPoint
from shared.models.observation import Observation

class GISService:

    def create_observation_point(self, lat: float, lon: float, source: str) -> GeoPoint:
        ...

    def extract_gps_from_image(self, image: Image.Image) -> Optional[Coordinates]:
        ...

    def export_geojson(self, observations: List[Observation]) -> dict:
        ...

    def observations_within_radius(
        self,
        center: Coordinates,
        radius_m: float,
    ) -> List[Observation]:
        # Delegates to spatial_queries.py (PostGIS ST_DWithin)
        ...

    def nearest_observation(self, point: Coordinates) -> Optional[Observation]:
        ...

Ingestion Service (placeholder)

# services/ingestion-service/interfaces.py

from abc import ABC, abstractmethod
from typing import List, Iterator
from shared.models.observation import Observation

class ObservationSource(ABC):
    """Interface for external observation data providers."""

    @abstractmethod
    def fetch_observations(self) -> Iterator[Observation]:
        ...

    @abstractmethod
    def source_name(self) -> str:
        ...

# Future implementations:
# class INaturalistSource(ObservationSource): ...
# class GBIFSource(ObservationSource): ...
# class GovernmentDatasetSource(ObservationSource): ...

Analytics Service (placeholder)

# services/analytics-service/interfaces.py

from abc import ABC, abstractmethod
from typing import List
from shared.models.observation import Observation

class AnalyticsService(ABC):

    @abstractmethod
    def generate_heatmap(self, observations: List[Observation]) -> dict:
        ...

    @abstractmethod
    def species_distribution(self, species_name: str) -> dict:
        ...

    @abstractmethod
    def density_estimation(self, bbox: tuple) -> dict:
        ...

---

5. Repository Interfaces

# apps/omyfish-api/repositories/observation_repository.py

from abc import ABC, abstractmethod
from uuid import UUID
from typing import Optional, List
from shared.models.observation import Observation
from shared.models.location import Coordinates

class ObservationRepository(ABC):

    @abstractmethod
    def create(self, obs: ObservationCreate) -> Observation:
        ...

    @abstractmethod
    def get_by_id(self, id: UUID) -> Optional[Observation]:
        ...

    @abstractmethod
    def list(self, limit: int = 100) -> List[Observation]:
        ...

    @abstractmethod
    def list_within_radius(
        self, center: Coordinates, radius_m: float
    ) -> List[Observation]:
        ...

    @abstractmethod
    def list_as_geojson(self, limit: int = 1000) -> dict:
        ...


class SQLObservationRepository(ObservationRepository):
    """PostgreSQL/PostGIS implementation."""

    def __init__(self, session):
        self._session = session

    def create(self, obs: ObservationCreate) -> Observation:
        # Uses SQLAlchemy ORM; PostGIS geom set via ST_MakePoint
        ...

    def list_within_radius(self, center: Coordinates, radius_m: float) -> List[Observation]:
        # ST_DWithin(geom, ST_MakePoint(:lon,:lat)::geography, :radius_m)
        ...

# apps/omyfish-api/repositories/species_repository.py

class SpeciesRepository(ABC):

    @abstractmethod
    def get_by_name(self, name: str) -> Optional[Species]:
        ...

    @abstractmethod
    def list_all(self) -> List[Species]:
        ...


class JSONSpeciesRepository(SpeciesRepository):
    """File-based implementation backed by data/metadata/fish_info.json."""
    ...

---

6. PostGIS Migration Strategy

Current state

• init_db() in app/database.py creates the schema via raw DDL on startup
• PostGIS geom column already present when DATABASE_URL is PostgreSQL
• No migration history — schema is recreated on every init_db() call

Target state

Step 1: Add Alembic

pip install alembic geoalchemy2
alembic init apps/omyfish-api/db/migrations

Step 2: Initial migration — captures the current schema

# alembic/versions/001_initial_schema.py
def upgrade():
    op.execute("CREATE EXTENSION IF NOT EXISTS postgis")
    op.create_table(
        "observations",
        sa.Column("id", PG_UUID(as_uuid=True), primary_key=True, server_default=sa.text("gen_random_uuid()")),
        sa.Column("species_name", sa.Text()),
        sa.Column("scientific_name", sa.Text()),
        sa.Column("confidence", sa.Float()),
        sa.Column("timestamp", sa.DateTime(timezone=True), server_default=sa.text("now()")),
        sa.Column("latitude", sa.Float()),
        sa.Column("longitude", sa.Float()),
        sa.Column("geom", Geography("POINT", srid=4326)),
        sa.Column("image_url", sa.Text()),
        sa.Column("user_id", sa.Text()),
        sa.Column("source", sa.Text(), server_default="upload"),
    )
    op.create_index("observations_geom_idx", "observations", ["geom"], postgresql_using="gist")

Step 3: Drop init_db() DDL — replace with alembic upgrade head on startup

Step 4: Future spatial migrations are versioned

# alembic/versions/002_add_species_table.py
# alembic/versions/003_add_validated_flag.py

SQLite fallback

Keep SQLite for local dev/HF Spaces with --dev flag. Alembic handles only the PostgreSQL path; SQLite uses a lightweight DDL shim (no migration history needed there).

---

7. Docker Architecture

# infrastructure/docker/docker-compose.yml

services:

  postgis:
    image: postgis/postgis:16-3.4
    environment:
      POSTGRES_DB: omyfish
      POSTGRES_USER: omyfish
      POSTGRES_PASSWORD: omyfish
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U omyfish"]

  api:
    build:
      context: ../../
      dockerfile: infrastructure/docker/Dockerfile.api
    ports:
      - "8000:8000"
    environment:
      DATABASE_URL: postgresql://omyfish:omyfish@postgis:5432/omyfish
      ENV: development
    depends_on:
      postgis:
        condition: service_healthy
    command: uvicorn apps.omyfish-api.main:app --host 0.0.0.0 --port 8000 --reload

  web:
    build:
      context: ../../
      dockerfile: infrastructure/docker/Dockerfile.web
    ports:
      - "8501:8501"
    environment:
      API_URL: http://api:8000
      DATABASE_URL: postgresql://omyfish:omyfish@postgis:5432/omyfish
    depends_on:
      - api
    command: streamlit run apps/omyfish-web/main.py --server.port=8501

volumes:
  pgdata:

Notes:

• fish-ai is NOT a separate container for now — it runs in-process inside api. Extract to its own container only when GPU inference is needed at scale.
• web can be optionally pointed at API_URL instead of direct DB — this is the cleaner architecture for production.

---

8. Dependency Flow

omyfish-web  ──────────────────────────────►  omyfish-api
                                                  │
                                    ┌─────────────┼─────────────┐
                                    ▼             ▼             ▼
                              routes/      repositories/  services/
                                               │
                                    ┌──────────┴──────────┐
                                    ▼                     ▼
                              PostgreSQL             shared/models
                              + PostGIS                    │
                                                    ┌──────┴──────┐
                                                    ▼             ▼
                                             fish-ai/        gis-service/
                                                    │
                                             ┌──────┴──────┐
                                             ▼             ▼
                                     EfficientNet      CLIP zero-shot
                                     (checkpoints/)    (HuggingFace Hub)

Dependency rules:
  shared/      ← no dependencies on apps/ or services/
  services/    ← depends on shared/ only
  apps/        ← depends on services/ and shared/
  routes/      ← depends on repositories/ and services/ (never on DB directly)
  repositories/← depends on shared/models/ and DB session

---

9. API Redesign

Route layout

GET  /health

POST /species/predict              # image → PredictionResponse
GET  /species/{name}               # species metadata lookup

POST /observations                 # create from manual input
GET  /observations                 # list, with ?limit=
GET  /observations/{id}            # single observation
GET  /observations/geojson         # FeatureCollection for map

GET  /observations/nearby          # ?lat=&lon=&radius_m=

Request/response contracts

# POST /species/predict
# Request: multipart/form-data — file=<image>, top_k=3
# Response:
{
  "predictions": [
    {"species": "Atlantic Salmon", "confidence": 0.87, "metadata": {...}},
    ...
  ],
  "uncertain": false,
  "message": null,
  "top_species": "Atlantic Salmon",
  "top_confidence": 0.87
}

# POST /observations
# Request body:
{
  "species_name": "Atlantic Salmon",
  "scientific_name": "Salmo salar",
  "confidence": 0.87,
  "latitude": 47.5,
  "longitude": -52.8,
  "source": "manual"
}
# Response:
{
  "id": "uuid",
  "status": "created"
}

# GET /observations/geojson
# Response: GeoJSON FeatureCollection (RFC 7946)

Design rules for routes

• Routes must only call services and repositories — never the DB directly
• All SQL lives in repositories
• All AI logic lives in FishAIService
• All GIS logic lives in GISService

---

10. Incremental Refactoring Plan

Priority: never break the running app between phases.

Phase 1 — Folder restructure (no logic changes)

Risk: Low. Pure file moves + import updates.

1. Create new directory skeleton
2. Move files to new locations:
   • app/clip_predictor.py → services/fish-ai/predictors/clip.py
   • src/predict.py → services/fish-ai/predictors/efficientnet.py
   • src/model.py → services/fish-ai/model/classifier.py
   • src/train.py, dataset.py, transforms.py, evaluate.py → services/fish-ai/training/
   • app/gis.py → services/gis-service/exif.py
   • app/api.py → apps/omyfish-api/main.py (minimal, still monolithic for now)
   • streamlit_app.py → apps/omyfish-web/main.py
   • app/database.py → apps/omyfish-api/db/engine.py
   • configs/config.yaml → configs/training.yaml
3. Update all import paths
4. Update Makefile and docker-compose.yml paths
5. Verify: make api and make app still work

Phase 2 — Service layer for AI

Risk: Low. Wraps existing code, no SQL changes.

1. Create services/fish-ai/predictors/base.py with BaseFishPredictor ABC
2. Make CLIPFishPredictor and FishPredictor implement it
3. Create services/fish-ai/service.py with FishAIService wrapping both
4. Add a factory function: build_predictor(checkpoint_path) -> BaseFishPredictor
5. Update apps/omyfish-api/main.py and apps/omyfish-web/main.py to use FishAIService
6. Verify: predictions still work in both CLIP and trained modes

Phase 3 — Repository layer

Risk: Medium. Touches all DB access.

1. Create shared/models/observation.py — SQLAlchemy ORM model
2. Create apps/omyfish-api/repositories/observation_repository.py
3. Create apps/omyfish-api/repositories/species_repository.py (JSON-backed)
4. Replace _insert_observation() in main.py (api) with ObservationRepository.create()
5. Replace raw SQL queries in list/geojson routes with repository calls
6. Replace raw SQL in Streamlit app with repository calls
7. Verify: saving and listing observations works

Phase 4 — GIS service + shared schemas

Risk: Low.

1. Create services/gis-service/service.py with GISService
2. Move EXIF extraction, GeoJSON building into it
3. Create shared/schemas/prediction.py, shared/schemas/observation.py
4. Replace inline ObservationIn Pydantic model in routes with shared schema
5. Verify: /identify-fish and /observations/geojson endpoints still work

Phase 5 — API routes split

Risk: Low.

1. Split apps/omyfish-api/main.py routes into routes/observations.py, routes/species.py, routes/gis.py, routes/health.py
2. Wire them up in main.py via app.include_router(...)
3. Verify: all endpoints still respond

Phase 6 — Config management

Risk: Low.

1. Add pydantic-settings dependency
2. Create configs/development.yaml and configs/production.yaml
3. Replace scattered hardcoded values (DB URL, thresholds, checkpoint paths) with config-loaded values
4. Support: local (yaml), docker (env vars), cloud (env vars)
5. Verify: app starts cleanly in both modes

Phase 7 — Placeholder services + events

Risk: None (new files only).

1. Create services/ingestion-service/interfaces.py
2. Create services/analytics-service/interfaces.py
3. Create shared/events.py with ObservationCreated, SpeciesPredicted, ObservationValidated
4. Emit ObservationCreated from ObservationRepository.create() (logged only, no queue yet)

Phase 8 — Alembic migrations

Risk: Medium (DB schema management change).

1. Install alembic and geoalchemy2
2. alembic init apps/omyfish-api/db/migrations
3. Generate initial migration from current schema
4. Replace init_db() call with alembic upgrade head in API startup
5. Verify: fresh docker-compose up creates schema from migrations

Phase 9 — Docker refactor

Risk: Low.

1. Move Dockerfile to infrastructure/docker/
2. Split into Dockerfile.api and Dockerfile.web (they're the same image today, which wastes space)
3. Update docker-compose.yml to infrastructure/docker/docker-compose.yml
4. Update Makefile paths
5. Verify: docker-compose up brings up all services

---

Implementation Order Summary

Phase	Work	Risk	Breaks MVP?
1	Folder restructure	Low	No
2	Fish AI service layer	Low	No
3	Repository layer	Medium	No
4	GIS service + shared schemas	Low	No
5	API routes split	Low	No
6	Config management	Low	No
7	Placeholder services + events	None	No
8	Alembic migrations	Medium	No
9	Docker refactor	Low	No

Total: 9 phases, each independently verifiable, zero downtime.

---

Generated: 2026-06-06 | Status: Ready for implementation