Spaces:

apoorvrajdev
/

image-captioning-api

Configuration error

App Files Files Community

image-captioning-api / README.md

apoorvrajdev

docs(readme): remove HF frontmatter rendered as table on GitHub

befac80 2 days ago

preview code

raw

history blame contribute delete

59.8 kB

Image Captioning System

CNN + Transformer image-to-language pipeline, lifted from an IEEE-published research notebook into a typed, tested, full-stack production codebase.

A deliberately scoped multimodal-AI showcase that takes a published research notebook and turns it into the kind of codebase a serving team would actually maintain — typed configuration, a structured FastAPI inference service, a polished React SPA, a parity-audit gate against the original notebook, and an honest roadmap that names what is shipped and what is not.

Status

✅ Deployed. Phase 2C (public deployment) is complete. The research → modular conversion (Phase 1) and the full inference stack (Phase 2A backend + 2B frontend) ship as a live, publicly reachable system: a React 19 / Vite 8 SPA at image-captioning-system.vercel.app posts multipart uploads to POST /v1/captions against a Dockerised FastAPI service running on a HuggingFace Space at apoorvrajdev-image-captioning-api.hf.space, which pulls its versioned weights from apoorvrajdev/captioning-inceptionv3-transformer on the Hub at lifespan startup via snapshot_download. The lifespan-managed CaptionPredictor is reused across every request with a warm graph and no per-call TF rebuilds. The IEEE notebook is preserved verbatim and protected by a SHA-256 freeze check, and a four-stage parity audit (scripts/notebook_module_audit.py) re-implements caption preprocessing, tokenizer vocabulary + encoding, image preprocessing, and the decoder forward pass inline and asserts the modular path is byte-identical (or tf.allclose-identical) to the notebook. Phase 1b (training stabilization) shipped beam search, the full corpus metric suite (BLEU-1..4 / CIDEr / METEOR / ROUGE-L), a benchmark runner that emits one machine-readable artefact set per evaluation, and a stabilized training config that gates label smoothing / cosine LR / warmup / dropout-free validation behind ablatable flags. Phase 2C shipped a hardened backend test suite (12 route tests covering the full 200 / 400 / 413 / 415 / 422 / 503 contract via a duck-typed fake predictor, full slice runs in 0.3 s), a multi-stage Dockerfile, Hub-versioned weight loading with an injectable downloader for offline testing, explicit production CORS wired through Space variables, a four-job GitHub Actions CI pipeline (ruff + mypy, pytest matrix on 3.10/3.11/3.12, notebook SHA-256 freeze, frontend lint + build) plus a chained deploy-backend.yml that pushes main to the Space remote only after CI is green, and a full deployment runbook at docs/PHASE_2C_DEPLOYMENT_RUNBOOK.md. Next up: Phase 3 (multimodal baselines) — see Roadmap.

📊 Trained checkpoint shipped. The stabilized training config (configs/train/stabilized.yaml) was trained on COCO 2017 (95,918 train captions, 24,082 val captions, 10 epochs, Kaggle T4 ×2, cosine LR with 500-step warmup, label smoothing 0.1). Results on a 500-sample val2017 slice:

Decode strategy BLEU-1 BLEU-4 ROUGE-L METEOR CIDEr

Greedy 42.20 10.57 37.57 15.45 0.789

Beam (w=4, lp=0.7, rp=1.2) 41.93 10.39 36.84 15.56 0.826

Full artefacts: results/stabilized-greedy/ and results/stabilized-beam-w4-lp07-rp12/. The trained weights are hosted on the Hub at apoorvrajdev/captioning-inceptionv3-transformer and loaded by the backend at startup — the live demo now produces real captions.

Decode strategy	BLEU-1	BLEU-4	ROUGE-L	METEOR	CIDEr
Greedy	42.20	10.57	37.57	15.45	0.789
Beam (w=4, lp=0.7, rp=1.2)	41.93	10.39	36.84	15.56	0.826

🌐 Live Demo

Component	URL	What you can do
Frontend SPA	https://image-captioning-system.vercel.app	Drag-and-drop an image, hit Generate caption, see the typed `CaptionResponse` rendered with model version, decode strategy, and latency
Backend API	https://apoorvrajdev-image-captioning-api.hf.space	Interactive Swagger at `/docs`; liveness + readiness at `/healthz`; inference at `POST /v1/captions`
Weights (HF Hub)	https://huggingface.co/apoorvrajdev/captioning-inceptionv3-transformer	Pinned to tag `v1.0.0`; the backend pulls these at lifespan startup via `snapshot_download` so the Space's git tree never contains the `.h5`

Deployment topology: GitHub main → CI on every push → on green, deploy-backend.yml pushes to a HuggingFace Space (Docker SDK, cpu-basic, port 7860, single uvicorn worker); Vercel's Git integration builds and promotes the SPA in parallel. Production CORS is wired through the Space's CAPTIONING__SERVE__CORS_ALLOWED_ORIGINS variable, not a hardcoded config. Full topology + rollback procedure: docs/PHASE_2C_DEPLOYMENT_RUNBOOK.md. CI/CD workflows: docs/CI.md.

💡 The live demo produces real captions from a COCO-trained checkpoint (CIDEr 0.83). Example: "a bathroom with a toilet and a sink", "a man riding skis down a snow covered slope". See results/stabilized-beam-w4-lp07-rp12/qualitative.jsonl for 30 sample predictions vs. ground-truth references.

📌 What Is This Project?

Image Captioning System is a research-to-production conversion of the IEEE paper "AI Narratives: Bridging Visual Content and Linguistic Expression". The original work — a Kaggle notebook training an InceptionV3-encoder + multi-head Transformer-decoder on MS COCO — is preserved verbatim as the canonical research artefact. Around it sits a typed Python package, a FastAPI inference service, and a React SPA that together turn the published model into something a serving team could actually run, version, and reason about.

It is not a hosted product (yet — Phase 2C is shipping that), and it is not a thin Streamlit wrapper around model.predict. What this project is is a deliberate engineering showcase aimed at hiring teams evaluating ML, multimodal-AI, and backend skills, and at anyone who has ever wondered what it actually takes to lift a research notebook into a codebase the rest of an engineering org can build on. Every architectural decision in this repository is one I can defend in an interview.

🎯 Why It Matters

Research notebooks and production ML systems are different artefacts with different audiences. A notebook proves an idea works. A production system has to survive being maintained — by people who did not write it, on schedules nobody planned, against inputs the original author never anticipated. The hardest part of an ML career is not getting a model to converge once; it is making the resulting pipeline legible, typed, testable, deployable, and replaceable without losing the behaviour the paper claimed.

This project demonstrates that conversion end-to-end at a scale one engineer can build and reason about:

Parity-gated refactor — the notebook stays byte-stable and a four-stage audit script asserts the modular package reproduces the notebook's behaviour at every behavioural seam.
Strict typed configuration — Pydantic v2 with extra="forbid" so a typo in a hyperparameter is a load-time error, not a silent training run that produces wrong numbers.
Lifespan-managed inference — one warm CaptionPredictor shared across every HTTP request, not a graph rebuilt per call.
Train/serve shared preprocessing — the same preprocess_image_tensor runs in tf.data pipelines and at inference, so the bytes that enter the model in training are byte-identical to the bytes that enter it at serve time.
Stabilized training experiments behind ablatable flags — every quality intervention is opt-in, so any delta between two runs is attributable to one named change rather than a tangled rewrite.
Reproducible benchmarking — every evaluation writes a machine-readable metrics.json + diagnostics.jsonl set, so two checkpoints (or one checkpoint with two decoders) can be diffed without bespoke parsers.

💡 What This Project Demonstrates

Lifting a research notebook into an installable, typed Python package (src/ layout) without breaking the published architecture.
A production-style FastAPI inference service with lifespan-managed model loading, structured logging, request-ID propagation, and a typed Pydantic schema for every payload.
A polished React 19 + Vite 8 + Tailwind v4 SPA with drag-and-drop upload, client-side validation, AbortController timeouts, typed ApiError classification, and a polled health badge.
Pydantic v2 strict configuration with YAML + env-var overrides and extra="forbid" to eliminate the silent-defaults failure mode.
Custom multi-head Transformer decoder with masked sparse-categorical cross-entropy, masked accuracy, learned (not sinusoidal) positional embeddings, and the IEEE paper's exact dropout / head configuration.
Beam search decoder with length normalisation and n-gram repetition suppression alongside greedy, selectable per inference call and per evaluation run.
Corpus-level metric suite — BLEU-1..4 (sacrebleu), CIDEr, METEOR, ROUGE-L — emitted as one typed artefact per run.
Notebook freeze + parity audit — SHA-256 lock on the IEEE notebook plus a four-stage inline re-implementation that fails CI if the modular path drifts.
Pre-commit governance — Ruff, mypy (strict), nbstripout, gitleaks, line-ending and TOML/YAML hygiene, all enforced before commits land.
Clean Git workflow with Conventional Commits and small, reviewable changesets (CLAUDE.md codifies the contribution rules).

🏗️ Architecture

                       ┌───────────────────────────────────────┐
                       │     React 19 + Vite 8 SPA             │
                       │   Tailwind v4 · AbortController · ApiError │
                       └──────────────────┬────────────────────┘
                                          │ multipart/form-data
                       ┌──────────────────▼────────────────────┐
                       │      FastAPI 0.111 (Pydantic v2)      │
                       │  RequestContextMiddleware · /healthz · /v1/captions  │
                       └──────────────────┬────────────────────┘
                                          │
                       ┌──────────────────▼────────────────────┐
                       │       PredictorService (anyio thread) │
                       │   bytes → tensor → predict → caption  │
                       └──────────────────┬────────────────────┘
                                          │ singleton, warmed in lifespan
                       ┌──────────────────▼────────────────────┐
                       │       CaptionPredictor (TensorFlow)   │
                       │   InceptionV3 → TF encoder → TF decoder → tokenizer │
                       └──────────────────┬────────────────────┘
                                          │
                       ┌──────────────────▼────────────────────┐
                       │       models/vX.Y.Z/ artefacts        │
                       │   model.h5 · vocab.json (versioned)   │
                       └───────────────────────────────────────┘

                 ┌───────────────────────────────────────────────┐
                 │  configs/*.yaml (Pydantic v2, extra="forbid") │
                 │  drives training, evaluation, AND serving     │
                 └───────────────────────────────────────────────┘

Model topology

┌──────────────┐   ┌──────────────────┐   ┌──────────────────┐   ┌──────────────────┐   ┌────────────┐
│  Input image │──▶│  InceptionV3     │──▶│  Transformer     │──▶│  Transformer     │──▶│  Caption   │
│  299×299×3   │   │  encoder         │   │  encoder         │   │  decoder         │   │  string    │
└──────────────┘   │  (ImageNet,      │   │  (1 layer,       │   │  (2 layers,      │   └────────────┘
                   │   frozen)        │   │   1 head)        │   │   8 heads)       │
                   └──────────────────┘   └──────────────────┘   └──────────────────┘
                          ▼                       ▼                       ▼
                    [B, 64, 2048]          [B, 64, 512]            [B, T, vocab=15000]

Components

CNN encoder — models/encoder_cnn.py. Pretrained InceptionV3 with the classification head removed; output reshaped to 64 spatial positions × 2048 channels. Weights frozen during training.
Transformer encoder — models/transformer_encoder.py. Single layer, one attention head. Projects InceptionV3 features into the decoder's embedding dimension.
Embeddings — models/embeddings.py. Sum of token + learned positional embeddings, preserved verbatim from the published architecture.
Transformer decoder — models/transformer_decoder.py. Causal self-attention over partial captions, cross-attention over image features, feed-forward sub-block. 8 heads, embedding_dim=512, dropouts (0.1 / 0.3 / 0.5) preserved from the IEEE configuration.
Captioning model — models/captioning_model.py. Custom train_step / test_step with masked sparse-categorical cross-entropy and masked accuracy.
Tokenizer — preprocessing/tokenizer.py. CaptionTokenizer wraps tf.keras.layers.TextVectorization; persists vocabulary as both pickle (notebook-compatible) and JSON sidecar.
Inference — inference/predictor.py. CaptionPredictor.from_artifacts(weights, vocab, config) loads everything once at boot, exposes predict_path(...) and predict_tensor(...) for stateless calls, and warmup() to amortise first-request latency.
Configuration — config/schema.py. Pydantic v2 (AppConfig / ModelConfig / TrainConfig / DataConfig / ServeConfig); strict so typos in YAML or env vars become load-time errors.

Why a monolith on a single process? Splitting training, evaluation, and serving across services would burn the project's budget on Kubernetes manifests instead of the things a reviewer can actually click. A layered package + one FastAPI app captures the same separation-of-concerns thinking with a tenth of the operational surface area, and the seams are placed so pulling serving into its own container (Phase 2C) is a deployment change, not a refactor.

Why TensorFlow 2.15 specifically? TF 2.16 ships Keras 3 by default and silently breaks TextVectorization save/load — the project's tensorflow-cpu==2.15.0 pin is deliberate. Documented in requirements.txt and in the engineering-decisions section below.

🖼️ Sample outputs

Image	Generated caption
	a man is standing on a beach with a surfboard
	a man riding a motorcycle on a street

Outputs above are from the IEEE notebook; the modular pipeline reproduces these via the parity audit (scripts/notebook_module_audit.py). Live captions from the current bootstrap weights will not match — see Current model quality status.

📚 Research backing

The model architecture and the BLEU-4 ~24 baseline below come from the IEEE paper and its accompanying notebook:

Paper: AI Narratives: Bridging Visual Content and Linguistic Expression (IEEE)
Original notebook: Kaggle — image-captioning-using-dl
Frozen artefact in this repo: notebooks/01_ieee_inceptionv3_transformer.ipynb — byte-stable; pre-commit + CI enforce its SHA-256.

The notebook is preserved verbatim as the canonical research artefact. Improvements happen in the modular package; the notebook does not.

📊 Performance

Metric	Value	Source
BLEU-4 (IEEE baseline)	~24	Reported in the IEEE paper / Kaggle notebook
Vocabulary size	15,000 tokens	`TextVectorization` adapt over preprocessed COCO captions
Training set	~120k captions sampled from COCO 2017	`data.sample_size` in `configs/base.yaml`
Image resolution	299 × 299 (InceptionV3)	`preprocessing/image.py`
Max caption length	40 tokens	`model.max_length` in `configs/base.yaml`
Backend test suite	12 tests · 0.3 s · no TF loaded	`backend/app/tests/`
Full suite	90 tests passing	`pytest` (unit + backend + parity)

Re-training on the modular pipeline is a Phase 1b deliverable; once a fresh checkpoint exists, this table will publish corpus BLEU-1..4, CIDEr, METEOR, and ROUGE-L (the harnesses already exist under evaluation/).

📊 Model quality — stabilized training results

The stabilized training config (configs/train/stabilized.yaml) converged on COCO 2017 in 10 epochs on Kaggle T4 ×2. Training loss dropped monotonically from 4.69 (epoch 1) to 3.33 (epoch 10); validation accuracy climbed from 0.43 to 0.48. No overfitting was observed — val_acc was still rising at epoch 10.

Corpus-level metrics (500-sample val2017 slice)

Metric	Greedy	Beam (w=4, lp=0.7, rp=1.2)
BLEU-1	42.20	41.93
BLEU-2	26.09	25.41
BLEU-3	16.52	16.01
BLEU-4	10.57	10.39
ROUGE-L	37.57	36.84
METEOR	15.45	15.56
CIDEr	0.789	0.826

Beam search trades a marginal n-gram overlap regression for a +5% CIDEr lift — CIDEr down-weights generic phrases and rewards image-specific vocabulary, making it the better quality signal for captioning. Full artefact sets (metrics, predictions, diagnostics, qualitative samples) are committed under results/.

Qualitative highlights

The model produces fluent, semantically grounded captions with correct object identification across diverse scenes. Sample predictions vs. COCO references (beam decode):

Image	Predicted	Reference	BLEU-4
000000129379	a woman sitting on a bench talking on a cell phone	a woman sitting on a cement wall talking on a cell phone	64.1
000000360371	a white toilet sitting in a bathroom next to a sink	a toilet sitting in a bathroom next to a scale	69.9
000000402020	a sandwich on a plate on a table	a sandwich on a plate and full wine glass are under blurry lights	74.2
000000082881	a man riding skis down a snow covered slope	two people ski over a snow covered slope	29.8
000000252596	a person riding a skateboard down a street	a person skateboards down a street that has greenery on either side	15.7

Known failure modes: colour attribute errors (red vs. yellow), count mismatches (one vs. two), generic fallback on unusual compositions. These are expected limitations of a frozen-InceptionV3 encoder and addressable in Phase 3 with modern vision backbones.

Training configuration

Parameter	Value
Encoder	InceptionV3 (frozen, ImageNet weights)
Decoder	Multi-head Transformer (4 heads, 512-dim)
Data	COCO 2017, 95,918 train / 24,082 val captions
Epochs	10 (no early stopping triggered)
Batch size	64
LR schedule	Cosine decay, peak 0.001, 500-step warmup
Label smoothing	0.1
Platform	Kaggle T4 ×2, TF 2.19, tf-keras 2.19 (legacy Keras 2 shim)
Wall-clock	~3.3 hours

🛠️ Tech Stack

Layer	Technologies
Core ML	Python 3.10–3.12, TensorFlow-CPU 2.15.0 (pinned), NumPy, Pillow
Model	InceptionV3 encoder (frozen) + custom multi-head Transformer decoder
Backend	FastAPI 0.111, Pydantic v2, `pydantic-settings` 2.x, structlog 24, anyio 4
Frontend	React 19, Vite 8, Tailwind v4, ESLint flat config
Evaluation	sacrebleu, custom CIDEr / METEOR / ROUGE-L implementations
Tooling	Ruff (lint + format), mypy (strict), pytest 8, pre-commit, nbstripout, gitleaks
Infra (planned, Phase 2C)	HuggingFace Hub (weights), HuggingFace Spaces (backend), Vercel (frontend), GitHub Actions (CI/CD)

📁 Repository Structure

image-captioning-system/
├── notebooks/
│   ├── 01_ieee_inceptionv3_transformer.ipynb   # FROZEN — IEEE research artefact
│   └── README.md                                # Frozen-notebook policy
│
├── src/captioning/                              # Installable package
│   ├── config/         schema.py · loader.py
│   ├── preprocessing/  caption.py · image.py · tokenizer.py · augmentation.py
│   ├── data/           coco.py · splits.py · pipeline.py
│   ├── models/         encoder_cnn.py · transformer_encoder.py · embeddings.py
│   │                   transformer_decoder.py · captioning_model.py · factory.py
│   ├── training/       losses.py · callbacks.py · trainer.py
│   ├── inference/      image_loader.py · greedy.py · beam.py · predictor.py
│   ├── evaluation/     bleu.py · cider.py · meteor.py · rouge.py
│   │                   runner.py · benchmark.py · inspection.py · tokenization.py
│   └── utils/          logging.py · seed.py · hashing.py
│
├── backend/                                     # Phase 2A — FastAPI inference service
│   └── app/
│       ├── main.py                              # App factory + lifespan-managed predictor singleton
│       ├── api/routes.py                        # Thin HTTP — /healthz, /v1/captions
│       ├── core/                                # BackendSettings, structlog setup, RequestContextMiddleware
│       ├── schemas/                             # Pydantic request/response models
│       ├── services/predictor_service.py        # bytes → caption + latency (anyio thread offload)
│       ├── utils/image.py                       # Content-type allow-list + ImageDecodeError
│       └── tests/                               # Phase 2C WS-D — 12 route tests, no TF loaded
│
├── frontend/                                    # Phase 2B — React 19 + Vite 8 + Tailwind v4 SPA
│   ├── vite.config.js · eslint.config.js · package.json · .env.example
│   └── src/
│       ├── main.jsx · App.jsx · index.css
│       ├── services/api.js                      # checkHealth / captionImage — AbortController + typed ApiError
│       └── components/
│           ├── Header.jsx · StatusBadge.jsx     # Sticky brand bar + 10s health poller
│           ├── UploadZone.jsx · ImagePreview.jsx
│           ├── CaptionResult.jsx · ErrorBanner.jsx · Spinner.jsx
│
├── configs/
│   ├── base.yaml                                # IEEE hyperparameters (notebook cell 6 mirror)
│   └── train/
│       ├── debug.yaml                           # CI smoke override (1 epoch, 64 captions)
│       └── stabilized.yaml                      # Phase 1b stability experiment (4 ablatable flags)
│
├── scripts/
│   ├── train.py · evaluate.py · predict.py
│   ├── inspect_predictions.py                   # Per-sample diagnostics + diagnostics.jsonl
│   ├── bootstrap_dev_artifacts.py               # Smoke-test artefacts so the API can boot pre-training
│   └── notebook_module_audit.py                 # 4-stage parity gate vs. notebook
│
├── tests/unit/                                  # 78 unit tests (parity, tokenizer, eval, splits, …)
├── docs/                                        # restructure-plan · PHASE_0_NOTES · PHASE_1_NOTES · STABILIZED_TRAINING_RUNBOOK
├── pyproject.toml · requirements*.txt · Makefile
├── .pre-commit-config.yaml · .python-version · .env.example
├── .paper-notebook.sha256                       # Locked notebook hash for the freeze check
├── CLAUDE.md                                    # Contribution + commit governance
└── README.md

🚀 Quick Start

Prerequisites

Python 3.10 – 3.12 (TensorFlow 2.15 has no 3.13 wheels)
Node 20+
Git

Backend

# PowerShell (Windows)
py -3.10 -m venv .venv
.venv\Scripts\activate
pip install -r requirements-dev.txt -r requirements-eval.txt
pip install -e ".[hf,mlflow]"
pre-commit install

# bash (Linux / macOS)
python3.10 -m venv .venv
source .venv/bin/activate
pip install -r requirements-dev.txt -r requirements-eval.txt
pip install -e ".[hf,mlflow]"
pre-commit install

Boot the API:

uvicorn --app-dir backend app.main:app --host 0.0.0.0 --port 8000

Interactive Swagger UI is live at http://localhost:8000/docs; raw OpenAPI 3.1 at http://localhost:8000/openapi.json.

Frontend

cd frontend
npm install
npm run dev

The SPA is live at http://localhost:5173 (Vite picks the next free port if 5173 is busy). VITE_API_BASE (see frontend/.env.example) points it at any backend origin; absent the env var, it falls back to http://127.0.0.1:8000.

Tests

pytest -q                          # All 90 tests (unit + backend + parity)
pytest backend/app/tests/ -v       # Backend route tests only (0.3 s, no TF loaded)
make freeze-paper-notebook         # Asserts the IEEE notebook SHA-256 has not changed

One-shot caption (CLI)

python -m scripts.predict \
    --config configs/base.yaml \
    --weights models/v1.0.0/model.h5 \
    --tokenizer-dir models/v1.0.0 \
    --image samples/photo.jpg

One-shot caption (HTTP)

curl -X POST http://localhost:8000/v1/captions -F "image=@samples/photo.jpg"

Reproduce training

python -m scripts.train --config configs/base.yaml
# Or with the stabilization experiment flags enabled:
python -m scripts.train --config configs/base.yaml --override configs/train/stabilized.yaml
# Or a 64-caption CI smoke run:
python -m scripts.train --config configs/base.yaml --override configs/train/debug.yaml

Outputs (weights.h5, vocab.pkl + vocab.json sidecar, history.json, training_log.csv) land under outputs/runs/latest/ by default.

make help lists every available command (lint, format, type-check, test, train, serve, evaluate, predict, Docker, freeze-paper-notebook, …).

🌐 FastAPI backend (Phase 2A)

Phase 2A delivers a production-style inference service rather than a thin demo wrapper:

App factory + lifespan — backend/app/main.py. create_app() builds the FastAPI instance; the lifespan loads the YAML AppConfig, instantiates a CaptionPredictor, calls warmup(), and stashes a PredictorService singleton on app.state so every request reuses one warm model.
Routes — backend/app/api/routes.py. Intentionally thin: validate inputs, delegate, shape the response. No TF imports leak into the HTTP layer.
Service layer — backend/app/services/predictor_service.py. Wraps the predictor, decodes uploaded bytes off the event loop via anyio.to_thread.run_sync, measures per-request latency, returns (caption, latency_ms).
Schemas — backend/app/schemas/caption.py. Pydantic v2 (CaptionResponse, HealthResponse, ErrorResponse); every payload that crosses the wire is typed and OpenAPI-documented.
Backend settings — backend/app/core/config.py. Separate BackendSettings (env-overridable: weights path, tokenizer dir, model version, warmup toggle) layered on top of the research-side AppConfig. Research hyperparameters and serving knobs change on different cadences and live in different settings objects.
Structured logging + request IDs — backend/app/core/logging.py. RequestContextMiddleware stamps each request with a UUID; structlog carries it through every log line so a single failed caption can be traced end-to-end.
Image safety — backend/app/utils/image.py. Content-type allow-list (JPEG / PNG / WebP / BMP), explicit ImageDecodeError so malformed bytes produce a clean 422 rather than a 500.

Method	Path	Purpose
`GET`	`/healthz`	Liveness + readiness — reports `model_loaded`, `model_version`, `api_version`. Always 200; readiness is conveyed in the body.
`POST`	`/v1/captions`	Multipart image upload → generated caption + decode strategy + latency + request ID.
`GET`	`/docs`	Interactive Swagger UI, auto-generated from the Pydantic schemas.
`GET`	`/openapi.json`	Raw OpenAPI 3.1 spec for client codegen.

POST /v1/captions enforces input validation at the boundary: 415 on disallowed content types, 413 on oversized uploads (serve.max_upload_bytes), 422 on undecodable image bytes, 400 on empty uploads, 503 while the predictor is still loading during a rolling restart. All six status codes are covered by the backend/app/tests/ suite added in Phase 2C WS-D.

🎨 Frontend UI (Phase 2B)

Phase 2B ships a single-page inference UI under frontend/ — not a styled demo. The split mirrors the backend's separation between transport, service, and presentation:

Application shell — frontend/src/App.jsx. Owns the request lifecycle (selected file → preview → generate → result). The preview URL.createObjectURL is useMemo-derived and revoked through an effect cleanup so previews never leak across uploads. Four useState slots (file, result, error, loading) cover every UI state — no Redux, no React Query, no context.
API service layer — frontend/src/services/api.js. Single boundary for every backend call. Reads import.meta.env.VITE_API_BASE once at module load (falls back to http://127.0.0.1:8000), wraps fetch with AbortController-driven timeouts (3 s for /healthz, 60 s for /v1/captions), and classifies failures into timeout / network / http / unknown kinds on a typed ApiError.
Upload zone — frontend/src/components/UploadZone.jsx. Drag/drop + click-to-browse + keyboard activation. Validates content-type (JPEG / PNG / WebP) and size (10 MB) before the file ever touches the network — invalid uploads are rejected client-side with the same wording the backend would have returned.
Status badge — frontend/src/components/StatusBadge.jsx. Polls /healthz every 10 seconds and on window focus, runs a three-state machine (checking / online / offline), recovers automatically when the backend comes back.
Error banner — frontend/src/components/ErrorBanner.jsx. Single surface for every failure class. Reads ApiError.message so the user sees "Cannot reach backend" or "Request timed out" instead of a raw browser error.
Caption result — frontend/src/components/CaptionResult.jsx. Consumes the backend's typed CaptionResponse directly: caption text plus model version, decode strategy, latency, and the request ID echoed from the x-request-id header.

┌──────────────┐  drag/drop   ┌─────────────┐  validate   ┌──────────────┐
│ UploadZone   │ ───────────▶ │  App state  │ ──────────▶ │ ImagePreview │
└──────────────┘              └─────────────┘             └──────────────┘
                                     │ click "Generate"
                                     ▼
                            ┌─────────────────┐  multipart   POST /v1/captions
                            │ services/api.js │ ───────────▶ FastAPI backend
                            └─────────────────┘
                                     │   typed CaptionResponse / ApiError
                                     ▼
                         ┌──────────────────────┐
                         │ CaptionResult /      │
                         │ ErrorBanner          │
                         └──────────────────────┘

Frontend and backend are deployed independently. The SPA only knows the backend's origin via VITE_API_BASE; the backend only trusts SPAs whose origin appears in serve.cors_allowed_origins. Dev origins are pre-allowed in configs/base.yaml; production origins join the same list at deploy time (Phase 2C WS-F). No shared build, no shared runtime — only the typed Pydantic schemas in backend/app/schemas/caption.py cross the wire.

⚙️ Configuration system

Hyperparameters are not globals. They live in YAML validated by Pydantic v2:

# configs/base.yaml — mirrors the IEEE notebook cell 6 verbatim
model:
  embedding_dim: 512
  units: 512
  max_length: 40
  vocabulary_size: 15000
  decoder_num_heads: 8
  decoder_dropout_inner: 0.3
  decoder_dropout_outer: 0.5
  decoder_attention_dropout: 0.1
train:
  epochs: 10
  batch_size: 64
  early_stopping_patience: 3
  seed: 42
data:
  sample_size: 120000
  train_val_split: 0.8

Three load-time guarantees:

Type validation. batch_size: "64" (string instead of int) raises a ValidationError pointing at the field, not a downstream tensor-shape error.
No silent typos. extra="forbid" rejects unknown keys — typos in ML hyperparameters silently using defaults is the worst failure mode, and extra="forbid" eliminates it.
Env overrides. CAPTIONING__TRAIN__BATCH_SIZE=32 overrides at any nesting depth — useful for CI smoke tests, ablations, and serve-time tuning without rebuilding images.

Schema in src/captioning/config/schema.py; loader in src/captioning/config/loader.py.

🧪 Testing & code quality

make test            # pytest — 90/90 (unit + backend route tests + parity)
make lint            # Ruff lint + format check
make typecheck       # mypy strict on src/captioning + scripts
make pre-commit      # All hooks across all files
make freeze-paper-notebook   # Asserts notebook SHA-256 unchanged

Layer	Tool	Status
Lint + format	Ruff (replaces black + isort + flake8)	✅ clean
Type-check	mypy with `pandas-stubs`, `types-PyYAML`, `types-requests`	✅ 0 errors
Tests	pytest + pytest-cov + pytest-asyncio	✅ 90 passing
Notebook hygiene	`nbstripout` (pre-commit)	✅ outputs stripped on commit
Secret scanning	`gitleaks` (pre-commit)	✅ enabled
Notebook integrity	SHA-256 freeze via `make freeze-paper-notebook`	✅ locked
Parity audit	`scripts/notebook_module_audit.py` — 4 stages	✅ all passing

The parity audit re-implements four notebook stages inline (caption preprocessing, tokenizer vocabulary + encoding, image preprocessing, decoder forward pass) and asserts the modular path produces byte-identical (or tf.allclose-identical) output. It is the contract that gates any behavioural improvement.

The backend test suite (backend/app/tests/) introduced in Phase 2C WS-D uses a duck-typed FakePredictorService to exercise every status code in the /v1/captions contract — 200 / 400 / 413 / 415 / 422 / 503 — plus the /healthz readiness flip and x-request-id propagation, all without loading TensorFlow. The full backend slice runs in 0.3 seconds.

🗺️ Roadmap

Phase 0 — Bootstrap ✅

0A — Repo scaffolding, pyproject.toml, Makefile, Conventional Commits
0B — Pre-commit hooks (Ruff, mypy, nbstripout, gitleaks, line-ending + TOML/YAML hygiene)
0C — Notebook freeze policy + .paper-notebook.sha256 SHA-256 lock
0D — Pinned dependency surface (requirements*.txt + pyproject.toml extras: hf, eval, mlflow, dev)

Phase 1 — Modularisation ✅

1A — Notebook → installable captioning package (src/ layout)
1B — Pydantic v2 strict config (AppConfig / ModelConfig / TrainConfig / DataConfig / ServeConfig) with YAML loader + env-var overrides
1C — Preprocessing modules (caption.py, image.py, tokenizer.py, augmentation.py) — shared train/serve preprocessing
1D — Data pipeline (coco.py, splits.py, pipeline.py) with seeded sampling
1E — Model factory (encoder_cnn.py, transformer_encoder.py, embeddings.py, transformer_decoder.py, captioning_model.py, factory.py)
1F — Training loop (losses.py, callbacks.py, trainer.py) with structured logging + history serialisation
1G — Greedy inference (predictor.py, image_loader.py, greedy.py) with lifespan-friendly from_artifacts(...) + warmup()
1H — Notebook parity audit (scripts/notebook_module_audit.py) — 4 stages, byte/tensor-identical
1I — Unit test suite (parity, tokenizer, evaluation, splits, hashing, image preprocessing, caption preprocessing)

Phase 1b — Training stabilization ✅

1b-A — Beam-search decoder (inference/beam.py) with length normalisation + n-gram repetition suppression, selectable per call/run
1b-B — CIDEr implementation (evaluation/cider.py)
1b-C — METEOR implementation (evaluation/meteor.py)
1b-D — ROUGE-L implementation (evaluation/rouge.py)
1b-E — Benchmark runner (evaluation/benchmark.py) emitting one metrics.json + diagnostics.jsonl per run
1b-F — Per-sample inspection tool (scripts/inspect_predictions.py) — sentence-level BLEU/ROUGE, length, longest repeated-token run, failure flags
1b-G — Stabilization config (configs/train/stabilized.yaml) — label smoothing, cosine LR, warmup, dropout-free validation, all ablatable
1b-H — Stabilized training runbook (docs/STABILIZED_TRAINING_RUNBOOK.md)
1b-I — Fresh stabilized COCO-trained checkpoint uploaded to HF Hub apoorvrajdev/captioning-inceptionv3-transformer (tag v2.0.0)
1b-J — Headline numbers (BLEU-1..4, CIDEr, METEOR, ROUGE-L) published in Model quality and committed under results/

Phase 2A — FastAPI inference service ✅

2A-1 — App factory + lifespan-managed CaptionPredictor singleton with warmup() on boot
2A-2 — Thin /healthz and POST /v1/captions routes with full status-code contract (200 / 400 / 413 / 415 / 422 / 503)
2A-3 — Pydantic v2 schemas (CaptionResponse, HealthResponse, ErrorResponse) with auto-generated Swagger + OpenAPI 3.1
2A-4 — PredictorService with anyio.to_thread.run_sync offload so TF inference never blocks the event loop
2A-5 — Structured logging (structlog) + RequestContextMiddleware propagating x-request-id across log lines
2A-6 — BackendSettings separated from research AppConfig (different change cadences, different env prefixes)
2A-7 — Bootstrap dev artefacts script so the API boots before training has produced real weights

Phase 2B — Frontend SPA ✅

2B-1 — React 19 + Vite 8 + Tailwind v4 scaffolding, flat ESLint config with eslint-plugin-react-hooks + eslint-plugin-react-refresh
2B-2 — Drag/drop + click-to-browse upload zone with keyboard activation and client-side content-type + size validation
2B-3 — services/api.js boundary: VITE_API_BASE env, AbortController timeouts (3 s health / 60 s caption), typed ApiError classification
2B-4 — Polled /healthz status badge with three-state machine, window-focus refetch, and automatic recovery
2B-5 — Typed CaptionResponse rendering — caption, model version, decode strategy, latency, request ID — with copy-to-clipboard
2B-6 — Single ErrorBanner surface mapping every ApiError.kind to actionable copy
2B-7 — CORS allow-list wired through backend YAML (serve.cors_allowed_origins), dev origins pre-allowed

Phase 2C — Public deployment ✅ (complete)

WS-A — Backend containerisation: Dockerfile (python:3.11-slim, non-root UID 1000, EXPOSE 7860, HEALTHCHECK on /healthz) + .dockerignore + corrected .env.example schema
WS-A4 — Lifespan integration with HuggingFace Hub: extended BackendSettings with weights_hub_repo / weights_hub_revision / weights_hub_filename / weights_cache_dir; new app.services.weights_loader.resolve_weights calls huggingface_hub.snapshot_download when configured, falls back to local paths otherwise (4 new unit tests, downloader injected for offline testing)
WS-B — Uploaded dev-scaffold weights + tokenizer to apoorvrajdev/captioning-inceptionv3-transformer on HuggingFace Hub, tagged v1.0.0, verified via snapshot_download (SHA-256 hashes match local artefacts byte-for-byte)
WS-C — First manual deploy to apoorvrajdev/image-captioning-api on HuggingFace Spaces (Docker SDK, cpu-basic, port 7860, single worker) — Space variables wire BACKEND_WEIGHTS_HUB_REPO / _REVISION / _FILENAME + BACKEND_WARMUP=true; lifespan pulls weights from the Hub on cold start; /healthz returns model_loaded: true and /v1/captions verified end-to-end via Swagger UI
WS-D — Backend test suite (backend/app/tests/): 12 route tests covering the full /healthz + /v1/captions contract (200 / 400 / 413 / 415 / 422 / 503) with a duck-typed FakePredictorService — no TF loaded, full slice runs in 0.3 s
WS-E — Frontend deploy to Vercel: frontend/ imported as a Vite project, VITE_API_BASE env var baked at build time, production alias image-captioning-system.vercel.app auto-redeployed on every push to main via Vercel's GitHub integration
WS-F — Production CORS: deployed Vercel origin added to serve.cors_allowed_origins via the Space's CAPTIONING__SERVE__CORS_ALLOWED_ORIGINS variable (JSON array, pydantic-settings parsed), so the policy is explicit in app config rather than relying on the HF reverse-proxy default
WS-G — GitHub Actions CI/CD:
- ci.yml — Python quality (ruff lint + format check, mypy), pytest matrix on 3.10/3.11/3.12, notebook SHA-256 freeze check, frontend lint + build, concurrency cancel-in-progress, pip + npm caching
- deploy-backend.yml — chained via workflow_run after CI, pushes HEAD:main to the HF Space remote using the HF_TOKEN repo secret; also supports workflow_dispatch for manual redeploys
- deploy-frontend.yml (skipped — Vercel-native GitHub integration deploys on every push, no separate workflow needed)
WS-H — "Live Demo" section above + docs/PHASE_2C_DEPLOYMENT_RUNBOOK.md (full topology, prerequisites, weights upload, Space setup, Vercel setup, CORS, CI/CD, smoke tests, known quirks, rollback) + docs/CI.md (workflow reference)

Phase 3 — Multimodal baselines ⏳ (planned)

3A — Side-by-side comparison harness: original CNN + Transformer vs. BLIP-base vs. ViT-GPT2 vs. GIT-base-coco
3B — Per-model BLEU / CIDEr / METEOR / ROUGE-L on a shared COCO slice with deterministic tokenisation
3C — Per-model latency benchmarking (single-image, batch, CPU vs. GPU)
3D — Comparison-result dashboard exposed through the existing SPA

Phase 4 — Observability ⏳ (planned)

4A — Sentry error tracking on backend + frontend
4B — Prometheus metrics (per-route latency histograms, predictor cache hits, lifespan boot duration)
4C — DagsHub-hosted MLflow tracking link surfaced in the README
4D — Architecture Decision Records (docs/adr/) — every non-trivial choice (TF version pin, anyio offload, env-var prefix separation, etc.) gets a one-page ADR

Detailed phase notes live under docs/: restructure plan · Phase 0 notes · Phase 1 notes · Stabilized training runbook.

🎯 Engineering Decisions

Why preserve the notebook verbatim instead of refactoring it in place? The notebook is the published research artefact and the only thing that can credibly produce the BLEU-4 ~24 baseline the IEEE paper claims. Editing it would silently destroy that reproducibility. The freeze + parity-audit pattern keeps the published result anchored while the modular package evolves; if the audit ever fails, the modular path has drifted from the paper and the diff is exactly where to start debugging.

Why pin tensorflow-cpu==2.15.0? TF 2.16 ships Keras 3 as the default backend, and Keras 3 silently breaks TextVectorization save/load — the tokenizer round-trip the entire serving stack depends on. The pin is documented in requirements.txt and protected by the env setup commands above. Phase 3's foundation-model baselines will live in optional dependency groups so they can install on a newer TF without unpinning the research pipeline.

Why two separate settings objects (AppConfig + BackendSettings)? Research hyperparameters (model.*, train.*, data.*) and serving knobs (weights path, model version, warmup toggle, request-id header) change on different cadences and have different audiences. Folding them into one object would mean every backend env var lived in a research YAML, and every research-side schema change risked breaking a deploy. Two objects with two prefixes (CAPTIONING__* vs BACKEND_*) gives each surface its own change schedule.

Why anyio.to_thread.run_sync for inference instead of async def predict? TensorFlow's predict call is synchronous and CPU-bound. Calling it directly from an async route handler would block the event loop and starve every other request. Offloading via anyio.to_thread.run_sync lets the event loop keep serving health checks and concurrent uploads while the model runs.

Why is the bootstrap-weights script committed? The serving stack (lifespan, predictor wiring, multipart upload, frontend integration) has to be verifiable before a real COCO-trained checkpoint exists. The bootstrap script makes the entire path runnable from a fresh clone, which is what lets reviewers actually evaluate the architectural work independently of the model-quality work. The captions are gibberish — by design — and the README states that prominently to keep expectations honest.

Why extra="forbid" on every config schema? ML projects fail catastrophically when a typo in a hyperparameter silently uses a default. vocabularsy_size: 30000 should be a load-time error, not a quiet retraining run on the wrong vocabulary size. Strict configs are the cheapest possible insurance against the most expensive class of bug in this domain.

Why ship the metric suite and beam search before publishing new numbers? Without deterministic tokenisation + a corpus-level runner + a non-greedy decoder, any "improved" number is unfalsifiable — it could be a real gain, a decoding artefact, or a tokenisation difference. The harness is the prerequisite to making the next training run mean something. Publishing the bar before the harness exists is how research projects accumulate numbers nobody can reproduce.

🔬 Experimental evaluation pipeline

The repository is evolving from a "research notebook reproduction" into a reproducible experimentation platform. Evaluation is no longer a single BLEU number printed at the end of training — it is a structured set of artefacts any future run, including the Phase 3 multimodal baselines, can be diffed against.

scripts/evaluate.py — single entrypoint for full corpus evaluation. Loads a checkpoint + tokenizer, runs decoding (greedy or beam) over the COCO validation slice, computes BLEU-1..4 / CIDEr / METEOR / ROUGE-L, and writes a versioned artefact set under results/<run_id>/.
scripts/inspect_predictions.py — per-sample diagnostic view. Prints N random predictions vs. references with sentence-level BLEU-4 / ROUGE-L, prediction length, longest repeated-token run, and failure flags (empty / very_short / repetitive / under_length). Used when the aggregate metric moves but the qualitative behaviour does not.
evaluation/benchmark.py — RunMeta and write_run_artifacts(...), the contract every evaluation run honours. Phase 3 cross-model comparison code joins multiple results/<run_id>/ directories without bespoke parsers per model.
Greedy vs. beam evaluation support — the same evaluator accepts --decode-strategy greedy|beam plus beam-search controls (--beam-width, --length-penalty, --no-repeat-ngram-size), so a single command-line difference produces directly comparable artefact sets for the same checkpoint.

⚖️ Limitations

The model produces generic captions on cluttered or rare-object scenes — a known limitation of the IEEE-era architecture, addressed in Phase 3 by adding modern foundation-model baselines for side-by-side comparison.
BLEU-4 (10.57 greedy / 10.39 beam) is below the IEEE notebook's reported ~24. The gap is attributable to frozen encoder features and a 10-epoch budget; fine-tuning the encoder or training longer would close it. See Model quality for the full metric table.
Colour attribute errors (red vs. yellow), count mismatches (one vs. two), and generic fallback on unusual compositions are the dominant failure modes — visible in results/stabilized-beam-w4-lp07-rp12/qualitative.jsonl.
Validation pipeline includes a leftover shuffle() from the notebook (functionally harmless, removed in Phase 1b).

These are explicitly tracked rather than hidden; full list in docs/PHASE_1_NOTES.md § Technical debt.

🧭 What I'd Build Next

Clear extension paths beyond the current scope, ordered by how much I'd learn building them:

Foundation-model fine-tuning — fine-tune BLIP-2 or LLaVA on COCO and benchmark per-token cost vs. caption quality against the InceptionV3 + Transformer baseline.
Streaming generation — server-sent events from /v1/captions so the SPA renders tokens as the decoder produces them, instead of waiting for the full sequence.
Batch inference endpoint — a second route that accepts an array of images, runs them through one TF graph call, and amortises the per-request Python overhead — useful for any downstream pipeline that needs to caption a folder.
Visual Question Answering — extend the same encoder + decoder pattern to POST /v1/vqa taking image + question, sharing the warmed CNN encoder.
VLM-backed comparison endpoint — an opt-in route that runs the same image through Anthropic Claude vision or OpenAI Vision behind a feature flag, returns both captions, and surfaces a side-by-side card in the SPA. The framing is "here's what a 2024 VLM does for the same input", not a replacement for the local model.
Online evaluation — a background job that periodically scores the latest checkpoint against a held-out COCO slice and pushes BLEU / CIDEr / latency to a Grafana dashboard, so model regressions surface without a human running scripts/evaluate.py.
Active-learning loop — surface low-confidence captions in the SPA, capture user corrections, and route them into a labelled corpus for the next training run.

📚 Lessons Being Learned

The hardest engineering skill on a research → production conversion is not the code — it is the discipline of not improving the model while you fix the codebase around it. Every quality intervention you fold in mid-refactor makes the parity audit ambiguous: when the numbers change, you cannot tell whether the new metric harness, the new tokenisation, the new decoder, or the new training schedule was responsible. The four ablatable flags in configs/train/stabilized.yaml exist specifically so each change can be diffed in isolation.

Pydantic with extra="forbid" has caught more real bugs in this codebase than every other tool combined. A typo in a YAML key that silently uses a default is the single most expensive class of bug in ML, and the fix is one config option.

The split between research config (AppConfig) and serving config (BackendSettings) felt over-engineered the day it was introduced and has paid for itself every week since. The two surfaces change on different cadences, ship on different schedules, and need different env-var prefixes for the deploy story to make sense. Conflating them would have meant every backend-only env var lived in a research YAML.

Notebook freezing is the smallest possible piece of engineering that earns the largest amount of trust. A SHA-256 file plus a pre-commit hook plus one CI step is enough to guarantee the published research is exactly what reviewers think it is, three years from now.

📝 License & Contact

This project is released under the MIT License.

Built by apoorvrajdev — reach me at apoorvrajmgr@gmail.com.

Contribution + commit governance for this repo is codified in CLAUDE.md.

Built as a flagship portfolio project for ML and multimodal-AI engineering roles.