(Right! Luxury!) Lakehouse

"Luxury! We used to dream of serverless!"

Open-source soccer analytics platform built on Databricks Lakebase — replacing a 6-service traditional AWS pipeline with a unified lakehouse architecture that scales to zero. The Hugging Face Hub serves as the public distribution layer for models, datasets, and interactive demos.

Try it now: Full Dashboard — 12-page Streamlit app with live data from 380+ matches across 5 providers. Or explore the Gradio Demo for a quick look.

---

Platform Scale & Data Engineering

The infrastructure uses a Medallion architecture (Bronze → Silver → Gold) provisioned entirely via Terraform IaC, unifying multi-vendor event and tracking data into a single analytical layer.

• 38M+ tracking frames ingested from three optical tracking providers (25fps and 10fps)
• 5 distinct data sources unified: StatsBomb, Wyscout, Metrica Sports, IDSSE (Bundesliga), and SkillCorner (A-League)
• 12 Streamlit dashboard pages deployed on HuggingFace Spaces (Docker SDK), querying Lakebase PostgreSQL via OAuth
• 19 synced tables with Zero-ETL continuous sync from Gold Delta Lake to Lakebase PostgreSQL 17
• 38 PostgreSQL indexes (34 btree + 4 HNSW vector indexes) for sub-10ms OLTP queries
• Pipeline reliability enforced through 807 unit tests (819+ with gensim) and 381 dbt data tests

The Hugging Face Footprint

All public artifacts are hosted entirely within the HF ecosystem.

Models

All model serialization uses JSON envelopes — zero pickle files (banned by project security policy).

Datasets

Interactive Spaces

Compute & Bidirectional Sync

While Databricks handles core data engineering, we use HF Jobs for workloads where a serverless Python environment is the right tool.

Examples:

• Expected Threat grids run as a CPU-based HF Jobs pipeline — downloads SPADL data from an HF Dataset, computes Markov chain value iteration, and publishes xT grids back to the Hub.
• xG v2 neural model trains on an A10G GPU via HF Jobs — a Deep Sets architecture with MC dropout, processing 131K shots with 360 freeze-frame context, exporting pure-NumPy weights for serverless inference.
• Space Creation computes per-player counterfactual pitch control surfaces on A10G via JAX double-vmap — 875K player-frame values across 40K frames in under 6 minutes.

All HF Jobs scripts use PEP 723 inline script metadata for zero-setup reproducibility.

Model weights published to HF Hub are synced back to Databricks UC Volumes for inference in the production Streamlit app. This creates a bidirectional flow: Databricks produces training data → HF Hub hosts artifacts → Databricks consumes model weights for scoring.

Academic Foundations

Every analytics module is grounded in peer-reviewed research, cited directly in the platform UI:

Engineering Quality

The platform maintains professional-grade engineering standards:

• Security: OAuth M2M everywhere, HTTPS-only, zero secrets in code, input validation on all identifiers, SSL verification enforced, JSON-only model serialization
• Type safety: Pyright basic mode, Pydantic models for configuration
• Testing: 807 pytest unit tests (819+ with gensim, including performance benchmarks), 381 dbt data quality tests
• CI/CD: GitHub Actions with OIDC federation (zero-secret CI), ruff linting, pre-commit hooks
• UX discipline: 71 of 78 findings resolved across two cognitive interface audits (CHI-AUDIT-180, CHI-AUDIT-190), grounded in 15 HCI frameworks (Norman, Sweller, Gergle, Kahneman, Cleveland & McGill, and others) — every metric has a help tooltip, every page has academic citations, every analytics term is defined in a context-sensitive glossary (Streamlit and HF Space)

Links

• License: Apache 2.0

_{Named after Monty Python's Four Yorkshiremen sketch, where each comedian one-ups the others about how deprived their childhood was. In data engineering, moving from hand-managed EC2 instances and 5-hop Reverse ETL pipelines to serverless Lakebase truly is... right luxury.}