feat: add nfcorpus quality eval for fp32 vs INT8Running
Aditya Kulkarni
c745c50
Measure cosine drift, rank agreement, and nDCG/recall so INT8 is judged on embedding quality, not speed alone.
chore: add pyright CI and use get_running_loop
Aditya Kulkarni
14b0898
Replace deprecated get_event_loop, drop the tutorial Settings docstring, and
fail CI if pyright reports type errors.
feat: landing server-status panel and embed ttft/ttfr
Aditya Kulkarni
5e96f99
Render GET /health on the landing as server status so the App tab does
not jump to raw JSON. POST /embed includes ttft_ms (enqueue to encode
start) and ttfr_ms (enqueue to embeddings ready). Keep GET /metrics as
an unused always-200 JSON alias.
feat: add HTML landing page for the Spaces App tab
Aditya Kulkarni
c4fe3e6
Serve GET / as a static page that lists the public endpoints and links
to Swagger. Restore the live-demo README framing now that the Space
responds.
feat: add /metrics snapshot and demote dead live-demo badge
Aditya Kulkarni
84098e5
Reuse the /health body on GET /metrics and always return 200. Drop the
"metrics not exposed" limitation. The HF Space timed out on /health and
/embed, so the README no longer frames it as a live demo.
chore: keep local SUMMARY.md off the repo
Aditya Kulkarni
c75969d
Public README and benchmark writeups already have the nfcorpus tables
and INT8-negative result. SUMMARY.md is private resume notes.
feat: land nfcorpus benches, encode batch_size, and INT8-negative docs
Aditya Kulkarni
dfdc75f
Pass max_batch_size through model.encode so 128/256/512 batches are one
forward pass, not silently chunked at 32. Add file-sourced bench harness,
prepare_dataset.py, and the 24-run latency/throughput matrix including
dynamic INT8.
Document the two-regime result and the INT8 slowdown. Keep nfcorpus text
pools gitignored; they regenerate with a fixed seed.
feat: add INT8 ONNX quantization with configurable backend file
Aditya Kulkarni
275b8e5
Add scripts/quantize_onnx.py for data-free dynamic INT8 quantization of the
exported ONNX model (90.4 -> 22.8 MB, 4x smaller). Run shape-inference
preprocessing inside a temp dir (best-effort, with a direct-quant fallback) so
it doesn't litter the working tree.
Thread a configurable ONNX_FILE_NAME (app/config.py) through
load_model(..., onnx_file_name=...) so the server can serve either the fp32 O3
export or the quantized model without code changes.
Benchmark subset + fp32-vs-int8 quality delta (STS Spearman) pending a real
test dataset.
feat: add readiness /health and structured request logging
Aditya Kulkarni
3e47cef
- /health is now a readiness probe (200/503) reporting model_loaded,
worker_alive, queue_depth, and inflight; previously a static "ok" that
inspected nothing
- DynamicBatcher exposes is_running()/queue_depth()/inflight() with an
in-flight counter tracked in submit()
- JSON logging (app/logging_config.py) with a per-request X-Request-ID
correlation id via middleware; structured access logs (method, path,
status, duration_ms); inference errors logged server-side with a
sanitized 500 returned to the client
- drop the unused request param from /embed; add LOG_LEVEL setting
- tests for readiness-503 and request-id generate/echo; README /health
+ logging docs
feat: add input validation, backpressure, and request timeouts
Aditya Kulkarni
3c0d20e
Harden the /embed serving contract:
- Validate request body (non-empty texts, max texts/request, max chars/text) -> 422
- Bound the request queue; reject with 503 when full instead of growing unbounded
- Enforce a per-request deadline; return 504 on timeout
- Map batcher errors to clean HTTP statuses; sanitize unexpected errors to 500
- Guard future resolution so a timed-out request can't crash the worker
Limits are env-configurable (MAX_TEXTS_PER_REQUEST, MAX_CHARS_PER_TEXT,
MAX_QUEUE_SIZE, REQUEST_TIMEOUT_S). Add tests for queue-full, timeout, and the
422/503/504 endpoint paths; update README failure-handling and API docs.
test: add pytest suite and CI
Aditya Kulkarni
299567b
- DynamicBatcher tests: size/time flush triggers, request->response mapping across
mixed batch sizes, exception propagation, and shutdown queue-drain, using a
deterministic fake model (no MiniLM download, CPU-only)
- endpoint tests for /health and /embed via FastAPI TestClient with load_model stubbed
- pytest + pytest-asyncio + ruff as a dev dependency group; pytest/ruff config
- GitHub Actions workflow: uv sync, ruff check, pytest
- Dockerfile: uv sync --no-dev so dev tools and the plot extra stay out of the image
- add CI badge to README
test: add benchmarking suite and publish results
Aditya Kulkarni
72c62ed
- run_matrix.py drives the full 16-config matrix (one server per config)
- matrix spans cpu/mps, batch sizes 1-32, and pytorch vs onnx backends
- naive_bench + utils add model-only and serving-overhead baselines
- bench.py records the device in its output metadata
- plot_results.py renders the batching sweep chart
- results in benchmarks/README.md, methodology in benchmarks/ANALYSIS.md
- surface the results in the project README
docs: overhaul project README
Aditya Kulkarni
df526fb
feat: add ONNX backend support
Aditya Kulkarni
f864c14
fix: interpret batch wait timeout in milliseconds
Aditya Kulkarni
57e38d0
cleanup: renamed api endpoint /predict to /embed
Aditya Kulkarni
d690983
Merge pull request #3 from BeardedAmbivert/feat/dynamic-batching
