| # CODIN Feature Guide |
|
|
| ## What This Project Is |
|
|
| CODIN is an end-to-end AutoML studio built with: |
|
|
| - `FastAPI` for APIs and backend orchestration |
| - `Streamlit` for the multi-page workspace UI |
| - `SQLAlchemy + SQLite` for datasets, jobs, experiment history, notes, registry labels, and drift records |
| - `scikit-learn`, `LightGBM`, `Optuna`, and custom pipeline logic for training and evaluation |
|
|
| The product is designed as a guided ML workspace rather than a single training script. A user can ingest data, profile it, train models, inspect results, simulate predictions, detect drift, generate synthetic data, compare runs, and export reports from the same app. |
|
|
| ## Core Product Flow |
|
|
| 1. A dataset is uploaded or imported. |
| 2. The backend profiles the dataset and stores metadata in `datasets`. |
| 3. A training job is created and the pipeline runs validation, cleaning, feature engineering, model selection, training, and evaluation. |
| 4. Results are saved to run artifacts and experiment history. |
| 5. The user explores explainability, contracts, what-if tools, drift, augmentation, and reports. |
|
|
| ## Main User-Facing Workspaces |
|
|
| ### 1. Home / Upload And Configure |
|
|
| Primary purpose: |
|
|
| - ingest new data |
| - restore exported bundles |
| - import from external connectors |
| - merge datasets |
| - configure AutoML training |
|
|
| Key features: |
|
|
| - File upload for tabular, document, image, PDF, SQLite, and export bundle formats |
| - Connector-based import for SQL sources |
| - Merge Studio for previewing joins before materializing merged datasets |
| - Auto problem detection to infer likely target/task hints |
| - Training configuration for mode, metric, imbalance handling, cleaning, CV, and feature selection |
|
|
| Logic: |
|
|
| - Uploads are normalized through `core/file_loader.py` |
| - Dataset profile information is generated and saved in `profile_json` |
| - Dataset lineage is tracked with `parent_dataset_id` |
| - Training configuration is stored on the job and reused in later pages |
|
|
| ### 2. Dataset DNA |
|
|
| Primary purpose: |
|
|
| - inspect dataset structure and quality before training |
|
|
| Key features: |
|
|
| - dataset profile overview |
| - numeric/categorical breakdown |
| - per-column stats table |
| - auto-detect details |
| - leakage and quality analysis |
| - repair preview and repair apply |
| - dataset lineage timeline and graph |
|
|
| Logic: |
|
|
| - Profiling computes row counts, column types, missingness, target hints, and column stats |
| - Leakage checks look for suspicious target correlation, ID-like fields, constants, duplicates, and temporal leakage hints |
| - Repair flows preview transformations first, then create a cleaned derived dataset |
|
|
| ### 3. Training Lab |
|
|
| Primary purpose: |
|
|
| - run AutoML training with pipeline-backed execution |
|
|
| Key features: |
|
|
| - job launch |
| - progress/status monitoring |
| - reasoning stream from the pipeline |
| - deep analysis handoff to Results Console |
|
|
| Logic: |
|
|
| - Training uses component-based pipeline execution from `core/pipeline_engine.py` |
| - Each component updates status and appends reasoning so the UI can show not just what happened, but why |
|
|
| ### 4. Results Console |
|
|
| Primary purpose: |
|
|
| - inspect the trained model and all downstream analysis |
|
|
| Key features: |
|
|
| - leaderboard and winner summary |
| - execution profile and pipeline metrics |
| - tested model breakdown |
| - SHAP global importance |
| - calibration report |
| - threshold tuner |
| - feature lineage |
| - recommendations |
| - trust heatmap |
| - drift timeline |
| - prediction sandbox |
| - feature contract checker |
| - counterfactual generation |
| - scenario sweep |
| - synthetic augmentation controls |
| - lightweight model chat |
|
|
| Logic: |
|
|
| - Results are normalized through `infra/result_contract.py` |
| - UI panels call dedicated APIs for each analysis instead of overloading one giant endpoint |
| - Mixed display tables are sanitized before rendering to keep Streamlit Arrow serialization stable |
|
|
| ### 5. Experiment Tracker |
|
|
| Primary purpose: |
|
|
| - compare historical runs and manage model promotion workflow |
|
|
| Key features: |
|
|
| - experiment archive |
| - global leaderboard |
| - run history viewer |
| - reasoning stream viewer |
| - registry labels: champion, challenger, candidate, archived |
| - team notes |
| - side-by-side run comparison |
| - run diff engine |
| - battle arena charts |
|
|
| Logic: |
|
|
| - Each completed job can be persisted as an `ExperimentRun` |
| - Registry and note models allow lightweight model governance without needing an external MLOps platform |
|
|
| ### 6. Drift Monitor |
|
|
| Primary purpose: |
|
|
| - detect dataset drift against the saved baseline and operationalize retraining |
|
|
| Key features: |
|
|
| - upload new batch for drift check |
| - per-feature drift metrics |
| - drift severity summary |
| - cadence scheduling |
| - drift history |
| - one-click retrain on drifted data |
|
|
| Logic: |
|
|
| - Baseline distributions are fit during training |
| - Later uploads are compared with PSI/KS-style checks |
| - Drift checks are stored so timeline views and cadence workflows can be built on top |
|
|
| ### 7. Smart AI Hub |
|
|
| Primary purpose: |
|
|
| - provide higher-order utilities around the trained models and datasets |
|
|
| Key features: |
|
|
| - ensemble builder |
| - what-if simulation |
| - synthetic data generation |
| - natural language ML helpers |
|
|
| Logic: |
|
|
| - This page reuses completed jobs and current workspace datasets rather than creating isolated tooling |
| - It is meant to sit on top of the main AutoML lifecycle |
|
|
| ## Training Pipeline Components |
|
|
| ### Data Validation |
|
|
| Implemented in `backend/services/training/components.py`. |
|
|
| What it does: |
|
|
| - loads the dataset |
| - checks target existence |
| - trims to selected features if requested |
| - drops columns with extreme missingness |
| - attempts numeric coercion on object columns |
| - runs leakage detection |
| - saves the data contract |
| - fits drift baseline |
|
|
| Why: |
|
|
| - it prevents garbage-in training runs |
| - it creates the baseline metadata needed by later features such as drift monitoring and contract checks |
|
|
| ### Feature Engineering |
|
|
| What it does: |
|
|
| - optional auto-cleaning |
| - task-type inference |
| - managed feature generation for smaller datasets |
| - invalid target cleanup |
| - label encoding for classification |
| - train/test split |
| - summary statistics for later reporting |
|
|
| Why: |
|
|
| - it balances automation with safety |
| - feature synthesis is only used when dataset size and width are still manageable |
|
|
| ### Model Selection |
|
|
| What it does: |
|
|
| - builds a profile of the problem |
| - asks the selector/meta-learner for a candidate pool |
| - chooses light or full preprocessing based on mode |
| - enables dimensionality guidance for wide datasets |
|
|
| Why: |
|
|
| - it makes the model search adaptive instead of static |
|
|
| ### Training |
|
|
| What it does: |
|
|
| - handles imbalance when configured |
| - executes sweep/tuning behavior |
| - trains the final pipeline |
| - evaluates on holdout data |
| - computes leaderboard, metrics, and explainability artifacts |
| - stores model metadata and run artifacts |
|
|
| Why: |
|
|
| - it separates fast exploration from deeper optimization |
| - it preserves enough metadata for later explainability and operational tools |
|
|
| ## Explainability Features |
|
|
| ### SHAP Global Importance |
|
|
| - ranks feature impact across the trained model |
| - used in the Results Console and reports |
|
|
| ### Local Explanation |
|
|
| - explains a single prediction using per-feature contributions |
| - used to rank candidate features for counterfactual search |
|
|
| ### Counterfactual Generator |
|
|
| Endpoint: |
|
|
| - `POST /api/counterfactual/{job_id}` |
|
|
| Logic: |
|
|
| - only enabled for classification jobs |
| - loads the saved model and training dataset context |
| - validates that all expected feature inputs are present |
| - scores the original row |
| - gets local feature contributions |
| - ranks the most influential features |
| - tries one-feature changes using numeric quantiles or common categorical alternatives |
| - returns the smallest single-field changes that flip the prediction |
|
|
| Current behavior: |
|
|
| - verified against job `3c29d593-31b3-4116-bf5e-a1b3d48d130b` |
| - returned a valid one-feature flip suggestion on `Age` |
|
|
| ### Feature Lineage |
|
|
| - inspects the saved preprocessor and maps transformed feature names back to raw groups |
|
|
| ### Calibration And Threshold Tuning |
|
|
| - computes classification calibration bins |
| - sweeps thresholds and compares precision/recall/F1 |
|
|
| ### Trust Heatmap |
|
|
| - checks how often features appear across recent runs for the same dataset |
| - compares historical importance stability |
| - marks features as stable, noisy, drift-prone, or leakage-risky |
|
|
| ## Prediction And Simulation Features |
|
|
| ### Live Prediction |
|
|
| Endpoint: |
|
|
| - `POST /api/predict/{job_id}` |
|
|
| Logic: |
|
|
| - validates exact feature contract |
| - builds a one-row inference frame in expected column order |
| - returns prediction and confidence when probabilities exist |
|
|
| ### Scenario Sweep / What-If Simulation |
|
|
| Endpoint: |
|
|
| - `POST /api/future` |
|
|
| Logic: |
|
|
| - takes a base feature vector |
| - varies one selected feature over multiple values |
| - scores each generated row independently |
| - returns prediction and optional confidence for each point |
|
|
| Current behavior: |
|
|
| - verified against job `3c29d593-31b3-4116-bf5e-a1b3d48d130b` |
| - returned valid prediction points for a `Salary` sweep with no point-level errors |
|
|
| ### Feature Contract Checker |
|
|
| - compares uploaded inference data against the training feature schema |
| - reports missing features, extra columns, dtype mismatches, and alignment status |
|
|
| ## Data Management Features |
|
|
| ### Dataset Catalog |
|
|
| - lists known datasets with profile summaries |
| - supports derived datasets and lineage-aware workflows |
|
|
| ### Merge Studio |
|
|
| - previews join quality before creating merged datasets |
| - reports overlap, duplicate keys, row multiplier, and sample merged records |
|
|
| ### Repair Preview / Apply |
|
|
| - shows proposed cleaning effect before materializing a new dataset version |
|
|
| ### Synthetic Data Generator |
|
|
| - creates synthetic rows from the current dataset |
| - stores derived dataset with lineage back to parent |
|
|
| ### Synthetic Data Judge |
|
|
| - compares synthetic output against the parent dataset |
| - checks numeric distribution drift and categorical mix overlap |
| - returns realism score, verdict, and notes |
|
|
| ## Experiment And Governance Features |
|
|
| ### Experiment Tracking |
|
|
| - stores run-level summary information in `experiment_runs` |
| - powers run archive and comparison UI |
|
|
| ### Registry Labels |
|
|
| - lightweight promotion workflow for production candidates |
|
|
| ### Team Notes |
|
|
| - lets users annotate runs without leaving the product |
|
|
| ### Run Diff Engine |
|
|
| - compares config and output changes between two runs |
|
|
| ## Reporting And Export |
|
|
| ### Report Generator |
|
|
| - creates PDF summaries with dataset overview, metrics, and SHAP importance |
|
|
| ### Export Bundle |
|
|
| - packages model, metadata, and training context for later restore |
|
|
| ## Natural Language And AI Helpers |
|
|
| ### Narrative Generator |
|
|
| - builds a human-readable summary of the experiment |
| - uses stored job story when available, otherwise composes a fallback narrative |
|
|
| ### Natural Language ML Helpers |
|
|
| - parse or support ML-oriented natural language workflows from the Smart AI Hub |
|
|
| ## Storage And State Model |
|
|
| ### Database Models |
|
|
| - `DatasetModel`: uploaded/imported/derived datasets |
| - `JobModel`: training jobs and result payloads |
| - `ExperimentRun`: completed-run archive |
| - `DriftCheck`: drift history |
| - `DriftSchedule`: cadence configuration |
| - `ModelRegistryEntry` and `TeamNote`: governance helpers |
|
|
| ### Run Artifacts |
|
|
| Per-run directories store: |
|
|
| - model pickle |
| - metrics JSON |
| - schema/data contract |
| - drift baseline |
| - model metadata |
| - exports and reports |
|
|
| ## Important Implementation Notes |
|
|
| ### Result Contract Normalization |
|
|
| - all result payloads are sanitized to be JSON-safe |
| - prevents NaN/Inf and shape drift from breaking the UI |
|
|
| ### Session Safety |
|
|
| - the DB session now uses `expire_on_commit=False` |
| - this avoids detached-instance crashes when recently loaded attributes are used after commit |
|
|
| ### Known Residual Noise |
|
|
| - LightGBM/scikit-learn still emits `X does not have valid feature names` warnings during some prediction paths |
| - these warnings did not block counterfactual or scenario sweep in runtime verification, but they are worth cleaning up later for quieter logs |
|
|
| ## Verified Status From This Pass |
|
|
| - Counterfactual: working on the verified completed job |
| - Scenario sweep: working on the verified completed job |
| - Trust heatmap: no longer failing from the detached-session regression in the verified direct check |
| - Dataset list helper: no longer failing from the detached-session regression in the verified direct check |
|
|
