π€ AutoMLOps
[](https://www.python.org/)
[](https://flask.palletsprojects.com/)
[](https://airflow.apache.org/)
[](https://mlflow.org/)
[](https://www.docker.com/)
[](https://huggingface.co/mnoorchenar/spaces)
[](#)
**π€ AutoMLOps** β A full-stack ML experiment tracking and pipeline orchestration platform. Train 50+ algorithms, run visual DAG pipelines powered by Apache Airflow, and manage models β all in one Docker container deployed to HuggingFace Spaces.
---
## Table of Contents
- [Features](#-features)
- [Architecture](#οΈ-architecture)
- [Getting Started](#-getting-started)
- [Docker Deployment](#-docker-deployment)
- [Pages](#-pages)
- [Pipelines](#-pipelines)
- [ML Algorithms](#-ml-algorithms)
- [Project Structure](#-project-structure)
- [Author](#-author)
- [Contributing](#-contributing)
- [Disclaimer](#disclaimer)
- [License](#-license)
---
## β¨ Features
| π¨ Pipeline Studio |
Interactive full-screen DAG canvas with clickable nodes, slide-in config panel, and live execution terminal |
| βοΈ Real Apache Airflow |
Pipelines execute as genuine Airflow DAGs with XCom, TaskInstance tracking, and DagRun polling |
| π€ AutoML Engine |
Automated hyperparameter search across all algorithm categories for classification and regression tasks |
| π MLflow Tracking |
Every training run logs parameters, metrics, and model artifacts to a persistent SQLite-backed MLflow store |
| π¦ Model Registry |
Register, version, and transition models through Staging β Production β Archived lifecycle stages |
| π Theme Toggle |
Dark and light mode with instant CSS variable switching and localStorage persistence |
| π³ Single-Container Deployment |
Flask + Airflow Scheduler + SQLite in one Docker image β no external services required |
---
## ποΈ Architecture
```
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β AutoMLOps β
β β
β βββββββββββββββ ββββββββββββββββββββ βββββββββββββββββββββ β
β β Datasets βββββΆβ MLOps Engine βββββΆβ Flask API β β
β β (sklearn + β β (sklearn / β β Backend β β
β β custom) β β XGBoost / β ββββββββββ¬βββββββββββ β
β βββββββββββββββ β LightGBM / β β β
β β MLP) β ββββββββββΌβββββββββββ β
β ββββββββββββββββββββ β Pipeline Studio β β
β β AutoML Page β β
β βββββββββββββββ ββββββββββββββββββββ β Model Registry β β
β β MLflow DB ββββββ Airflow β βββββββββββββββββββββ β
β β (SQLite) β β Scheduler β β
β βββββββββββββββ β (DAGs / XCom / β β
β β TaskInstance) β β
β ββββββββββββββββββββ β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
```
---
## π Getting Started
### Prerequisites
- Python 3.11+
- Docker (for containerised deployment)
- Git
### Local Installation
```bash
# 1. Clone the repository
git clone https://github.com/mnoorchenar/AutoMLOps.git
cd AutoMLOps
# 2. Create a virtual environment
python -m venv venv
source venv/bin/activate # Windows: venv\Scripts\activate
# 3. Install app dependencies
pip install -r requirements.txt
# 4. Install Apache Airflow with official constraints
AIRFLOW_VERSION=2.10.4
pip install "apache-airflow==${AIRFLOW_VERSION}" \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-3.11.txt"
# 5. Initialise Airflow metadata DB
export AIRFLOW_HOME=$(pwd)/airflow_home
export AIRFLOW__CORE__DAGS_FOLDER=$(pwd)/dags
airflow db migrate
# 6. Start the Airflow scheduler (background)
airflow scheduler &
# 7. Run the Flask application
python app.py
```
Open your browser at `http://localhost:7860` π
---
## π³ Docker Deployment
```bash
# Build and run
docker build -t automlops .
docker run -p 7860:7860 automlops
# Or deploy directly to HuggingFace Spaces
# Push to a Space with sdk: docker in README frontmatter
```
The Dockerfile builds a single image containing:
- Flask application (served via Gunicorn on port 7860)
- Apache Airflow Scheduler (started by `start.sh`)
- All Python dependencies with Airflow constraint-file pinning
- Pre-initialised Airflow SQLite metadata DB (`airflow db migrate`)
---
## π Pages
| Page | Route | Description | Status |
|------|-------|-------------|--------|
| π¨ Pipeline Studio | `/` | Interactive DAG canvas β click nodes to configure and execute pipelines | β
Live |
| π€ AutoML | `/automl` | Automated algorithm search across 50+ models for any dataset | β
Live |
| π¦ Model Registry | `/models` | Browse registered models, versions, and lifecycle stages | β
Live |
---
## π Pipelines
Three production-quality pipelines are pre-built and immediately executable from the Pipeline Studio:
**Training Pipeline** (`training_pipeline`)
```
Load Data β Validate β Preprocess β Feature Engineering β Train Model β Evaluate β Report β Register β Deploy to Staging
```
**Retraining Pipeline** (`retraining_pipeline`)
```
Drift Detection β Fetch New Data β Merge Datasets β Retrain Champion β A/B Test β Promote to Production
```
**Data Processing Pipeline** (`data_pipeline`)
```
Ingest Raw Data β Clean Data β Encode Features β Scale Features β Save to Feature Store
```
Each pipeline node is clickable in the UI β configurable nodes (dataset picker, algorithm picker) show a purple indicator dot. Execution logs stream live in the built-in terminal panel.
---
## π§ ML Algorithms
```python
# AutoMLOps Algorithm Registry β 50+ algorithms across 2 tasks
ALGORITHMS = {
"classification": {
"Linear Models": ["Logistic Regression", "Logistic Regression (L1)", "Ridge Classifier",
"SGD Classifier", "Passive Aggressive", "Linear Discriminant Analysis"],
"Tree-Based": ["Decision Tree", "Random Forest", "Extra Trees",
"Quadratic Discriminant Analysis"],
"Ensemble / Boosting": ["Gradient Boosting", "AdaBoost", "Bagging Classifier",
"XGBoost", "LightGBM"],
"Support Vector Machines":["SVC (RBF Kernel)", "SVC (Polynomial)", "SVC (Linear)", "LinearSVC"],
"Probabilistic": ["Gaussian Naive Bayes", "Bernoulli Naive Bayes", "Complement Naive Bayes"],
"Instance-Based (KNN)": ["KNN (k=3)", "KNN (k=5)", "KNN (k=9)"],
"Neural Networks": ["MLP (Small)", "MLP (Medium)", "MLP (Deep)"],
},
"regression": {
"Linear Models": ["Linear Regression", "Ridge Regression", "Lasso",
"ElasticNet", "Bayesian Ridge", "Huber Regressor"],
"Tree-Based": ["Decision Tree Regressor", "Random Forest Regressor",
"Extra Trees Regressor"],
"Ensemble / Boosting": ["Gradient Boosting Regressor", "AdaBoost Regressor",
"Bagging Regressor", "XGBoost Regressor", "LightGBM Regressor"],
"Support Vector Machines":["SVR (RBF)", "SVR (Linear)"],
"Instance-Based (KNN)": ["KNN Regressor (k=3)", "KNN Regressor (k=5)"],
"Neural Networks": ["MLP Regressor (Small)", "MLP Regressor (Medium)"],
}
}
```
**Built-in Datasets:**
| Dataset | Task | Samples | Features |
|---------|------|---------|---------|
| Iris Flowers | Classification | 150 | 4 |
| Wine Quality | Classification | 178 | 13 |
| Breast Cancer | Classification | 569 | 30 |
| Diabetes Progression | Regression | 442 | 10 |
| California Housing | Regression | 20,640 | 8 |
---
## π Project Structure
```
AutoMLOps/
β
βββ π mlops/
β βββ algorithms.py # 50+ algorithm registry (classification + regression)
β βββ datasets.py # Dataset loaders (sklearn built-ins + California Housing)
β βββ trainer.py # Training & AutoML job management
β βββ airflow_runner.py # Apache Airflow DAG trigger & watcher
β
βββ π pipelines/
β βββ dag_engine.py # Built-in DAG execution engine (fallback)
β βββ pipeline_defs.py # Training / Retraining / Data pipeline definitions
β
βββ π dags/ # Apache Airflow DAG files (parsed by scheduler)
β
βββ π templates/
β βββ base.html # Base layout: sidebar + topnav + theme toggle
β βββ pipeline.html # Pipeline Studio (home page β interactive DAG canvas)
β βββ automl.html # AutoML experiment launcher
β βββ models.html # Model Registry browser
β
βββ π static/
β βββ css/style.css # Global styles + dark/light theme CSS variables
β βββ js/app.js # Shared JS (toasts, theme switching)
β
βββ π app.py # Flask application entry point + all API routes
βββ π Dockerfile # Single-container image (Flask + Airflow)
βββ π start.sh # Startup: Airflow scheduler β Gunicorn Flask
βββ π requirements.txt # Python dependencies
βββ π README.md
```
---
## π¨βπ» Author
