Initial Release: Polymer Aging With ML [Standalone Appliance]

.dockerignore

@@ -0,0 +1,49 @@
+# Exclude Git history (huge security risk if leaked into an image)
+.git/
+.gitignore
+
+# Exclude secrets, credentials and local environment configs
+.env
+.env.*
+*.pem
+*.key
+*.cert
+*.p12
+credentials.json
+
+# Exclude local Python environments and cache
+__pycache__/
+*.py[cod]
+*$py.class
+.venv/
+venv/
+env/
+ENV/
+
+# Exclude local Node modules and frontend builds (they are rebuilt in Docker)
+node_modules/
+frontend/node_modules/
+frontend/build/
+frontend/dist/
+.eslintcache
+
+# Exclude logs, local data, and local test outputs
+*.log
+tmp/
+backend/tests/
+*.sqlite3
+*.db
+
+# Exclude IDE and OS files (prevents dev machine details leakage)
+.vscode/
+.idea/
+.DS_Store
+Thumbs.db
+*.swp
+
+# Optional: Exclude documentation to reduce build context size
+*.md
+
+# Exclude Dockerfile itself
+Dockerfile
+.dockerignore

.gitattributes

@@ -0,0 +1,37 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+backend/models/weights/*.pth filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1,59 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+
+# Logs
+*.log
+tmp/logs/
+backend/tests/test_output/
+
+# Node/Frontend
+node_modules/
+frontend/node_modules/
+frontend/build/
+.eslintcache
+
+# OS-specific
+.DS_Store
+Thumbs.db
+
+# Large Model Weights (Safety catch - even though we use LFS)
+# backend/models/weights/*.pth # Commented out because we DO want to track them via LFS
+
+# Security & Secrets
+.env*
+*.pem
+*.key
+*.cert
+*.p12
+*secret*
+credentials.json
+
+# Local Databases
+*.sqlite3
+*.db
+
+# IDE & Editor Settings (Can leak local paths)
+.vscode/
+.idea/
+*.swp
+*.swo
+
+# Coverage and Reports
+.coverage
+htmlcov/
+coverage.xml
+
+Makefile
+tmp/
+/tmp
+*.log
+log

ARCHITECTURE.md

@@ -0,0 +1,504 @@
+---
+title: Polymer Aging ML (React + FastAPI)
+emoji: 🔬
+colorFrom: indigo
+colorTo: yellow
+sdk: docker
+app_port: 7860
+pinned: false
+license: apache-2.0
+---
+
+
+AI-Driven Polymer Aging Prediction and Classification (v0.2)
+============================================================
+
+[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
+![React](https://img.shields.io/badge/React-18.2-brightgreen)
+![FastAPI](https://img.shields.io/badge/FastAPI-0.116%2B-blue)
+![Docker](https://img.shields.io/badge/docker-ready-blue)
+![HF Space](https://img.shields.io/badge/HF%20Space-Live-blueviolet)
+
+This web application classifies the degradation state of polymers using **Raman and FTIR spectroscopy** and deep learning.
+It features a modern **React frontend** with a **FastAPI backend** for production-ready deployment with full scientific provenance tracking.
+This document serves as a Good Reference Note (GRN) baseline for the `ml-polymer-recycling` project. It provides a comprehensive snapshot of the system architecture, component interactions, and data flow as of this version.
+
+---
+
+## Current Scope
+
+## 1. Project Overview
+
+- **Modalities**: Raman & FTIR spectroscopy
+- **Input Formats**: `.txt`, `.csv`, `.json` (with auto-detection)
+- **Models**: Figure2CNN (baseline), ResNet1D, ResNet18Vision, Custom CNNs (Enhanced, Efficient, Hybrid)
+- **Task**: Binary classification — Stable vs Weathered polymers
+- **Features**:
+  - Single-spectrum + Batch Spectrum Analysis
+  - Multi-model comparison
+  - Performance tracking dashboard
+  - **Full provenance tracking** (QC checks, preprocessing parameters, model metadata)
+  - **RESTful API** for programmatic access
+- **Architecture**: React + FastAPI + PyTorch
+  This project is a full-stack web application that uses Artificial Intelligence to analyze chemical data from plastics and polymers. Users can upload their own scientific data (a spectrum from Raman or FTIR spectroscopy) and the AI will predict whether the material is **stable** (unweathered) or has been **weathered** (degraded).
+
+The platform also provides advanced features:
+
+- **Model Comparison:** Allows users to run an analysis using multiple different AI models and compare their predictions.
+- **AI Explainability:** Provides a deep dive into _why_ the AI made a specific decision, highlighting the spectral regions that most influenced the outcome.
+- **Performance Tracking:** Offers a dashboard to monitor the health of the backend system and the performance metrics of the loaded AI models.
+
+---
+
+## Architecture
+
+## 2. Table of Contents
+
+**Modern single-container deployment with complete separation of concerns:**
+
+- **Frontend**: React TypeScript application with Material Design-inspired UI
+- **Backend**: FastAPI with comprehensive ML inference pipeline
+- **API Contract**: Strict Pydantic models for all request/response validation
+- **Scientific Fidelity**: Exact preservation of original Streamlit ML logic
+- **Full Provenance**: Every prediction includes QC checks, preprocessing details, and model metadata
+- **Single Port**: Both frontend and API served on the same port for Hugging Face Spaces compatibility
+- 1. Project Overview
+- 2. Table of Contents
+- 3. Technology Stack
+- 4. Project Structure
+- 5. Setup and Installation
+- 6. Architectural Deep Dive
+  - 6.1. React Component Architecture
+  - 6.2. API Endpoint Layer
+  - 6.3. API Data Contracts
+  - 6.4. ML Inference Service
+  - 6.5. Transparent AI & Explainability
+  - 6.6. Model Registry
+- 7. Component & API Mapping
+  - 7.1. Frontend Component Guide
+- 8. Data Flow Example: Standard Analysis
+- 9. Developer Workflow Guide
+
+---
+
+## How to Use
+
+## 3. Technology Stack
+
+The application provides three main analysis modes:
+
+### 1. **Standard Analysis**
+
+- Upload a single spectrum file (`.txt`, `.csv`, `.json`)
+- Choose a model and spectroscopy modality
+- Get detailed predictions with full provenance metadata
+- View spectrum processing visualization
+
+### 2. **Model Comparison**
+
+- Upload a single spectrum file
+- Compare predictions from all available models
+- View consensus predictions and model agreement metrics
+- Analyze confidence variance across models
+
+### 3. **Performance Tracking**
+
+- Monitor system health and model performance
+- View detailed model specifications and availability
+- Track memory usage and processing capabilities
+
+### Supported Input
+
+- Plaintext `.txt`, `.csv`, or `.json` files
+- Data can be space-, comma-, or tab-separated
+- Comment lines (`#`, `%`) are ignored
+- Automatic format detection and resampling to standard length
+
+| Area     | Technology             | Description                                                               |
+| :------- | :--------------------- | :------------------------------------------------------------------------ |
+| Backend  | **Python 3.10+**       | Core programming language.                                                |
+|          | **FastAPI**            | High-performance web framework for building the API.                      |
+|          | **Pydantic**           | Data validation and settings management (defines API Data Contracts).     |
+|          | **PyTorch**            | The deep learning framework used to build and run the AI models.          |
+|          | **NumPy** & **SciPy**  | Libraries for numerical operations and scientific data processing.        |
+|          | **Uvicorn**            | ASGI server that runs the FastAPI application.                            |
+| Frontend | **TypeScript**         | Superset of JavaScript that adds static typing for robustness.            |
+|          | **React**              | A component-based library for building the user interface.                |
+|          | **Recharts**           | A composable charting library for visualizing spectrum data.              |
+|          | **openapi-typescript** | Tool to auto-generate TypeScript types from the backend's OpenAPI schema. |
+
+---
+
+## API Documentation
+
+The API is documented using the OpenAPI standard. When the application is running, you can access the live, interactive documentation (Swagger UI) to explore and test all available endpoints. This is the single source of truth for the API.
+
+- **URL**: [`http://localhost:8000/api/v1/docs`](http://localhost:8000/api/v1/docs)
+
+## 4. Project Structure
+
+The project is a monorepo containing both the `backend` and `frontend` in a single directory.
+
+```text
+ml-polymer-recycling/
+├── backend/
+│   ├── main.py             # FastAPI app: API endpoints, middleware, static file serving.
+│   ├── service.py          # Core MLInferenceService: handles all analysis logic.
+│   ├── models.py           # Pydantic models: defines the API data contracts.
+│   ├── utils/
+│   │   ├── preprocessing.py  # Spectrum cleaning and standardization functions.
+│   │   └── enhanced_ml_service.py # Logic for the explainability features.
+│   └── models/
+│       ├── registry.py       # The Model Registry for managing different AI models.
+│       └── figure2_cnn.py    # Example implementation of a CNN model.
+├── frontend/
+│   ├── src/
+│   │   ├── App.tsx           # Main React component, manages layout and tabs.
+│   │   ├── apiClient.ts      # Central API client for all frontend-backend communication.
+│   │   ├── components/       # Directory for all reusable React components.
+│   │   │   ├── Sidebar.tsx
+│   │   │   ├── StandardAnalysis.tsx
+│   │   │   ├── ResultsDisplay.tsx
+│   │   │   └── ... (other components)
+│   │   └── types/
+│   │       └── api.ts        # Auto-generated TypeScript types from the backend API.
+│   ├── public/
+│   └── package.json          # Frontend dependencies and scripts.
+├── main.py                   # Main entry point to start the application server.
+└── openapi-schema.json       # The OpenAPI specification generated by FastAPI.
+```
+
+## **API Documentation**: Available at `/api/docs` when running
+
+## 5. Setup and Installation
+
+### Prerequisites
+
+- **Python 3.10+** and `pip`
+- **Node.js 16+** and `npm`
+
+### 1. Backend Setup
+
+From the project root directory, install the required Python packages:
+
+```bash
+pip install -r backend/requirements.txt
+```
+
+### 2. Frontend Setup
+
+Navigate to the frontend directory and install the Node.js dependencies:
+
+```bash
+cd frontend
+npm install
+cd ..
+```
+
+### 3. Running the Application
+
+The application is designed for a single-container deployment. The `main.py` script at the root will automatically build the frontend (if needed) and start the backend server, which also serves the frontend files.
+
+From the project root directory, run:
+
+```bash
+python main.py
+```
+
+You can now access the application at `http://localhost:8000`.
+
+---
+
+## 🔧 Development & Deployment
+
+## 6. Architectural Deep Dive
+
+### Local Development
+
+This section provides a high-level overview of the core architectural concepts that power this application.
+
+For the frontend to connect to the backend, you'll need to set an environment variable.
+
+### 6.1. React Component Architecture
+
+1.  **Configure API URL**:
+    Create a file named `.env.development` in the `frontend` directory with the following content:
+    `   REACT_APP_API_BASE_URL=http://localhost:8000`
+    The frontend user interface is built using React. The core idea is to build everything from small, independent, and reusable pieces called **components**.
+
+2.  **Run Backend**:
+    ```bash
+    # Backend development
+    python main.py
+    ```
+
+- **Independent:** A component manages its own logic and appearance (e.g., `SpectrumChart`).
+- **Reusable:** A component can be used in many different places (e.g., a `Button` or `Spinner`).
+- **Composable:** Simple components are combined to build more complex structures.
+
+3.  **Run Frontend**:
+    `bash
+    # Frontend development (separate terminal)
+    cd frontend
+    npm start
+    `The main file,`frontend/src/App.tsx`, assembles the page layout by composing high-level components like `<Header />`, `<Sidebar />`, and the main tab components (`<StandardAnalysis />`, `<ModelComparison />`, etc.). It uses a **state** variable (`activeTab`) to manage which view is currently visible, making the UI interactive.
+
+**Build for Production**:
+
+```bash
+# Build for production
+cd frontend
+npm run build
+python main.py  # Serves both frontend and API
+```
+
+### 6.2. API Endpoint Layer
+
+### Docker Deployment
+
+The frontend (in the browser) cannot run the AI models directly. It communicates with the backend via an **API (Application Programming Interface)**. The API defines a "menu" of services the backend offers at specific URLs, called **endpoints**.
+
+```bash
+# Single-container deployment
+docker build -t polymer-aging-ml .
+docker run -p 8000:8000 polymer-aging-ml
+```
+
+- `GET /api/v1/models`: Asks the backend for the list of available models.
+- `POST /api/v1/analyze`: Sends a spectrum to the backend to be analyzed.
+
+### Hugging Face Spaces
+
+The backend uses the **FastAPI** framework to create these endpoints. An endpoint's job is to manage communication, not to do the core work. It receives a request, delegates the task to a service (like the `MLInferenceService`), and returns the final response.
+
+The application is designed for single-container deployment on Hugging Face Spaces:
+
+### 6.3. API Data Contracts
+
+- Uses Docker SDK with port 8000
+- Serves React frontend and FastAPI backend on the same port
+- Optimized build process with multi-stage Docker builds
+  To prevent miscommunication, the frontend and backend agree on a strict **Data Contract**—a blueprint that defines the exact structure of the data they exchange.
+
+---
+
+- **Backend (Pydantic):** In `backend/models.py`, we use Pydantic models to define the expected structure of requests and responses. FastAPI uses these models to automatically validate all incoming data.
+- **Frontend (TypeScript):** In `frontend/src/types/api.ts`, we have TypeScript interfaces that are auto-generated from the backend's OpenAPI schema. This ensures the frontend always creates data that conforms to the contract.
+
+## Contributors
+
+This dual-blueprint system guarantees that the "data door" built by the frontend will always fit the "data doorframe" expected by the backend.
+
+- Dr. Sanmukh Kuppannagari (Mentor)
+- Dr. Metin Karailyan (Mentor)
+- Jaser Hasan (Author/Developer)
+
+### 6.4. ML Inference Service
+
+## Model Credit
+
+The `MLInferenceService` (in `backend/service.py`) is the core scientific engine of the application. It encapsulates the entire analysis workflow:
+
+Baseline model inspired by:
+
+1.  **Receive** the raw spectrum data.
+2.  **Preprocess** the data (baseline correction, resampling, normalization).
+3.  **Load** the correct AI model from the Model Registry.
+4.  **Run Inference** to get a prediction from the model.
+5.  **Package the Result** with rich **provenance metadata**, documenting every step of the process for scientific validity and reproducibility.
+
+Neo, E.R.K., Low, J.S.C., Goodship, V., Debattista, K. (2023).
+_Deep learning for chemometric analysis of plastic spectral data from infrared and Raman databases._
+_Resources, Conservation & Recycling_, **188**, 106718.
+https://doi.org/10.1016/j.resconrec.2022.106718
+This service separates the complex scientific logic from the web-serving logic of the API endpoints.
+
+---
+
+### 6.5. Transparent AI & Explainability
+
+## 🔗 Links
+
+To avoid being a "black box," the system includes an explainability engine (`enhanced_ml_service.py`). When a user requests an explanation, this service goes beyond a simple prediction and provides deeper insights:
+
+- **Live App**: Hugging Face Space
+- **GitHub Repo**: ml-polymer-recycling
+- **Feature Importance:** It identifies which parts of the spectrum were most influential in the model's decision.
+- **Uncertainty Quantification:** It can estimate how confident the model is and why.
+
+---
+
+This turns the AI from a simple calculator into an expert consultant, making the results more trustworthy and useful for scientific research.
+
+Technical Architecture
+
+### 6.6. Model Registry
+
+**Production-ready architecture with complete separation of concerns:**
+To manage a growing collection of different AI models, the project uses a **Model Registry** (`models/registry.py`). This acts as a central "toolbox" or catalog for all available model architectures.
+
+### Frontend (React TypeScript)
+
+Instead of hard-coding a specific model, the `MLInferenceService` can ask the registry for a model by its simple name (e.g., `"figure2"` or `"resnet"`). The registry handles the details of finding and constructing the correct model object. This makes the system highly flexible and easy to extend with new models.
+
+- **UI Framework**: React 18 with TypeScript for type safety
+- **Styling**: Custom CSS matching original Streamlit design language
+- **Charts**: Recharts for spectrum visualization
+- **File Upload**: React Dropzone for drag-and-drop functionality
+- **API Client**: Axios with comprehensive error handling
+- **Responsive Design**: Mobile-friendly interface
+
+---
+
+### Backend (FastAPI)
+
+## 7. Component & API Mapping
+
+- **API Framework**: FastAPI with automatic OpenAPI documentation
+- **ML Pipeline**: Preserved original PyTorch inference logic
+- **Validation**: Pydantic models for strict request/response contracts
+- **Provenance**: Complete metadata tracking for scientific reproducibility
+- **Performance**: Async processing with memory management
+- **Models**: Registry pattern for dynamic model loading
+
+### 7.1. Frontend Component Guide
+
+### Scientific Fidelity
+
+| Component (`/src/components/`) | Description                                                                                                                                | Key Interactions                                                                                              |
+| :----------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------ |
+| **`App.tsx`**                  | The main application shell. Manages the overall layout and which tab is currently active. Holds the global `spectrumData` state.           | Renders `Sidebar`, `Header`, and the active tab component. Lifts state up from `StandardAnalysis`.            |
+| `Header.tsx`                   | A simple, static component that displays the application title and subtitle.                                                               | None.                                                                                                         |
+| `Sidebar.tsx`                  | Allows the user to select the AI model and spectroscopy modality (`raman`/`ftir`). Displays detailed information about the selected model. | Fetches model list via `apiClient.getModels()`. Updates `selectedModel` and `modality` state in `App.tsx`.    |
+| `StandardAnalysis.tsx`         | The primary analysis tab. Manages file upload, analysis execution, and displaying results for a single spectrum.                           | Uses `react-dropzone` for uploads. Calls `apiClient.uploadSpectrum()` and `apiClient.analyzeSpectrum()`.      |
+| `ModelComparison.tsx`          | UI for selecting multiple models to compare against a single spectrum. Fetches available models and allows user selection.                 | Calls `apiClient.compareModels()`. **Note:** The UI for displaying comparison results is not yet implemented. |
+| `PerformanceTracking.tsx`      | A dashboard that displays system health and detailed performance metrics for all available models.                                         | Fetches data by calling `apiClient.getSystemInfo()`.                                                          |
+| `ExplainabilityPanel.tsx`      | The "AI Explainability" tab. Allows the user to run an analysis that generates feature importance and other insights.                      | Uses the shared `spectrumData` from `App.tsx`. Calls `apiClient.explainSpectrum()`.                           |
+| `ResultsDisplay.tsx`           | A detailed component that takes a `PredictionResult` object and renders all its information in a user-friendly format.                     | Renders class probabilities, performance metrics, and metadata. Uses `SpectrumChart` for visualization.       |
+| `SpectrumChart.tsx`            | A reusable chart component that visualizes spectrum data using the `recharts` library. Can display original and processed spectra.         | Receives `SpectrumData` as props and renders a `LineChart`.                                                   |
+
+- **Exact Preservation**: All ML inference logic preserved from original Streamlit app
+- **Deterministic Outputs**: Identical results to original application
+- **Quality Control**: Comprehensive spectrum validation and QC checks
+- **Preprocessing**: Complete preprocessing pipeline with metadata tracking
+- **Model Metadata**: Full model provenance including training metrics
+
+### 7.2. Backend API Endpoint Guide
+
+### Deployment
+
+| API Endpoint (`/api/v1/...`)  | Method | Description                                                                                               | Service Function Called (backend)                |
+| :---------------------------- | :----- | :-------------------------------------------------------------------------------------------------------- | :----------------------------------------------- |
+| `/api/v1/system`              | `GET`  | Retrieves comprehensive system information, including loaded models, performance, and health metrics.     | `ml_service.get_system_info()`                   |
+| `/api/v1/models`              | `GET`  | Returns a list of all available models and their metadata.                                                | `ml_service.get_available_models()`              |
+| `/api/v1/upload`              | `POST` | Uploads a spectrum file (`.txt`, `.csv`, `.json`) and parses it into the standard `SpectrumData` format.  | `utils.parse_spectrum_data()` (indirectly)       |
+| `/api/v1/analyze`             | `POST` | Analyzes a single spectrum using a specified model and returns a detailed `PredictionResult`.             | `ml_service.run_inference()`                     |
+| `/api/v1/compare`             | `POST` | Compares multiple models on a single spectrum and returns a `ComparisonResult`.                           | `ml_service.run_inference()` (in a loop)         |
+| `/api/v1/explain`             | `POST` | Analyzes a spectrum and returns an enriched result with feature importance and other explainability data. | `enhanced_ml_service.py: predict_with_explanation()` |
+
+- **Single Container**: React frontend and FastAPI backend in one Docker image
+- **Multi-stage Build**: Optimized Docker builds for production
+- **Static Serving**: FastAPI serves React static files efficiently
+- **Health Checks**: Comprehensive monitoring and health endpoints
+- **Port Unification**: Both UI and API on port 8000 for Hugging Face Spaces
+
+---
+
+## 8. Data Flow Example: Standard Analysis
+
+## Notable Features in v0.2
+
+This diagram illustrates the end-to-end journey of a request when a user uploads a file and clicks "Analyze".
+
+- **Complete Architecture Transformation**: Streamlit → React + FastAPI
+- **Scientific Fidelity Preservation**: Exact ML behavior maintained
+- **Full Provenance Tracking**: QC checks, preprocessing parameters, model metadata
+- **RESTful API**: Programmatic access to all functionality
+- **Single-Container Deployment**: Optimized for Hugging Face Spaces
+- **Type Safety**: Full TypeScript implementation
+- **Professional UI**: Production-ready interface design
+- **Comprehensive Testing**: API validation and frontend testing
+- **Performance Optimization**: Async processing and memory management
+- **Documentation**: Auto-generated API docs and comprehensive README
+
+``` mermaid
+sequenceDiagram
+    participant User
+    participant StandardAnalysis as StandardAnalysis.tsx
+    participant apiClient as apiClient.ts
+    participant FastAPI as Backend (main.py)
+    participant MLService as ML Service (service.py)
+    User->>StandardAnalysis: Drops a file into the upload zone
+    StandardAnalysis->>apiClient: Calls `uploadSpectrum(file)`
+    apiClient->>FastAPI: POST /api/v1/upload with file data
+    FastAPI-->>apiClient: Returns parsed `SpectrumData` JSON
+    apiClient-->>StandardAnalysis: Returns `SpectrumData` object
+    StandardAnalysis->>StandardAnalysis: Sets spectrum state, UI updates to show chart
+    User->>StandardAnalysis: Clicks "Analyze Spectrum" button
+    StandardAnalysis->>apiClient: Calls `analyzeSpectrum({ spectrum, model_name, ... })`
+    apiClient->>FastAPI: POST /api/v1/analyze with `AnalysisRequest` JSON
+    Note over FastAPI: Validates request against Pydantic model
+    FastAPI->>MLService: Calls `run_inference(spectrum, model_name, ...)`
+    Note over MLService: Preprocesses data, loads model, runs prediction
+    MLService-->>FastAPI: Returns a complete `PredictionResult` object
+    FastAPI-->>apiClient: Returns `PredictionResult` as JSON
+    apiClient-->>StandardAnalysis: Returns `PredictionResult` object
+    StandardAnalysis->>StandardAnalysis: Sets result state, passing data to `<ResultsDisplay />`
+    Note over StandardAnalysis: UI updates to show prediction, confidence, and all metadata.
+```
+
+---
+
+## 9. Developer Workflow Guide
+
+This section outlines common development tasks and best practices for this project.
+
+### How to Add a New Model
+
+1.  **Create the Model File**: Add your PyTorch model class to a new file in `backend/models/` (e.g., `my_new_cnn.py`).
+2.  **Register the Model**: In `backend/models/registry.py`:
+    - Import your new model class.
+    - Add a new entry to the `_REGISTRY` dictionary with a unique key and a lambda function to instantiate your model.
+    - Add a corresponding entry to the `_MODEL_SPECS` dictionary with its metadata (description, performance, etc.).
+3.  **Add Weights**: Place the trained model weights (`.pth` file) in the `backend/model_weights/` directory, ensuring the filename matches the key you used in the registry (e.g., `my_new_cnn_model.pth`).
+4.  **Verify**: Restart the application. The new model should automatically appear in the "Select Model" dropdown in the UI.
+
+How to Add a New API Endpoint
+
+1. **Backend (`models.py`)**: Define the Pydantic models for the request body and response. This enforces the data contract.
+2. **Backend (`service.py`)**: Add the core business logic for the new feature into a new method in the `MLInferenceService`.
+3. **Backend (`main.py`)**: Create a new FastAPI route (e.g., `@app.post("/api/v1/my-new-endpoint")`) that takes the request model, calls the new service method, and returns the response model.
+4. **Update API Schema**: After adding the endpoint, regenerate the `openapi-schema.json` file. You can do this by running the app and copying the JSON from `http://localhost:8000/api/v1/openapi.json`.
+5. **Update Frontend Types**: Run the `openapi-typescript` tool to regenerate `frontend/src/types/api.ts` from the new schema.
+
+   ```bash
+   npx openapi-typescript openapi-schema.json --output frontend/src/types/api.ts
+   ```
+
+6. **Frontend (`apiClient.ts`)**: Add a new method to the `ApiClient` class that calls the new endpoint and uses the newly generated TypeScript types.
+7. **Frontend (Component)**: Create or update a React component to call the new `apiClient` method and display the results.
+
+```mermaid
+flowchart TD
+    subgraph Frontend [Frontend: React + TypeScript]
+        UI[UI Components]
+        APIClient[apiClient.ts]
+    end
+    subgraph Backend [Backend: FastAPI + PyTorch]
+        Main[main.py (routes + startup)]
+        Service[service.py (MLInferenceService)]
+        Models[backend/models/registry.py]
+        Utils[backend/utils/*]
+    end
+    subgraph Data [Persistence & Artifacts]
+        Weights[Model Weights .pth]
+        Logs[Performance Logs]
+    end
+    UI --> APIClient
+    APIClient --> Main
+    Main --> Service
+    Service --> Models
+    Service --> Utils
+    Models --> Weights
+    Service --> Logs
+```

Dockerfile

@@ -0,0 +1,44 @@
+# --- STAGE 1: Frontend Builder ---
+FROM node:18-slim AS frontend-builder
+WORKDIR /app/frontend
+COPY frontend/package*.json ./
+RUN npm ci
+COPY frontend/ ./
+RUN npm run build
+
+# --- STAGE 2: Backend Runtime ---
+FROM python:3.12-slim
+
+# Install ONLY runtime libraries (no compilers/git)
+RUN apt-get update && apt-get install -y \
+    curl \
+    libopenblas0 \
+    && rm -rf /var/lib/apt/lists/*
+
+# Set up user
+RUN useradd -m -u 1000 user
+ENV HOME=/home/user PATH=/home/user/.local/bin:$PATH
+WORKDIR $HOME/app
+
+# We need compilers ONLY for the pip install phase
+# Use a temporary root session to install, then clean up
+COPY --chown=user requirements.txt ./
+RUN apt-get update && apt-get install -y build-essential \
+    && pip install --no-cache-dir --user -r requirements.txt \
+    && apt-get purge -y build-essential && apt-get autoremove -y \
+    && rm -rf /var/lib/apt/lists/*
+
+USER user
+
+# Copy backend code
+COPY --chown=user backend/ ./backend/
+
+# Copy React build from builder stage
+# NOTE: Using 'build' because that is the default React output folder
+COPY --chown=user --from=frontend-builder /app/frontend/build ./frontend/dist
+
+# Expose port
+EXPOSE 7860
+
+# Start FastAPI
+CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "7860"]

LICENSE

@@ -0,0 +1,201 @@
+ Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2025 Jaser H.
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

README.md

@@ -0,0 +1,177 @@
+---
+title: Polymer Aging with ML
+emoji: 🔬
+colorFrom: blue
+colorTo: green
+sdk: docker
+app_port: 7860
+pinned: false
+license: mit
+---
+
+# PolymerOS: AI-Driven Polymer Aging Prediction and Classification
+
+![React](https://img.shields.io/badge/React-18.2-brightgreen)
+![FastAPI](https://img.shields.io/badge/FastAPI-0.116%2B-blue)
+![PyTorch](https://img.shields.io/badge/PyTorch-ML--Core-red)
+![Docker](https://img.shields.io/badge/docker-ready-blue)
+![License](https://img.shields.io/badge/License-Apache%202.0-blue)
+
+---
+
+## Overview
+
+**PolymerOS** is a full-stack AI application that classifies the degradation state of polymers using **Raman** and **FTIR spectroscopy**.
+It enables scientists, engineers, and researchers to upload spectroscopic data and receive predictions on whether materials are **stable (unweathered)** or **weathered (degraded)**.
+
+This platform was designed for both research reproducibility and production-grade deployment. Every prediction includes **scientific provenance tracking**, ensuring transparency in preprocessing, QC checks, and model metadata.
+
+---
+
+## Features
+
+- **Multi-Modal Input**: Supports Raman and FTIR data
+- **Flexible File Formats**: Accepts `.txt`, `.csv`, `.json` (auto-detected)
+- **Model Zoo**: Figure2CNN, ResNet1D, ResNet18Vision, and custom CNNs
+- **Batch & Single-Spectrum Analysis**
+- **Multi-Model Comparison**
+- **Provenance Tracking**: QC checks + preprocessing metadata
+- **Performance Dashboard**
+- **RESTful API** for programmatic access
+- **Modern UI**: React + TypeScript, charting with Recharts
+
+---
+
+## Architecture
+
+PolymerOS is a **monorepo** with a React frontend and FastAPI backend, packaged for **single-container deployment**.
+
+```text
+polymeros/
+├── backend/
+│   ├── main.py               # FastAPI entrypoint: routes, middleware, startup
+│   ├── service.py            # Core MLInferenceService logic
+│   ├── pydantic_models.py    # API contracts for request/response validation
+│   ├── utils/                # Preprocessing, performance, orchestration
+│   └── models/
+│       ├── registry.py       # Model registry for AI models
+│       └── figure2_cnn.py    # Example CNN implementation
+├── frontend/
+│   ├── src/
+│   │   ├── App.tsx           # Main React app shell
+│   │   ├── apiClient.ts      # Centralized API calls
+│   │   ├── components/       # Reusable UI components
+│   │   └── types/api.ts      # Auto-generated TypeScript types
+│   └── package.json
+├── models/weights/           # Directory for trained model weights (.pth)
+├── Dockerfile                # Multi-stage build for backend+frontend
+└── main.py                   # Root entrypoint: serves frontend + backend
+```
+
+---
+
+## Data Flow
+
+Example: Standard Spectrum Analysis
+
+```mermaid
+sequenceDiagram
+    participant User
+    participant UI as React UI
+    participant API as FastAPI Backend
+    participant ML as ML Service
+    User->>UI: Upload spectrum file
+    UI->>API: POST /api/v1/upload
+    API->>ML: Parse + preprocess spectrum
+    ML->>ML: Run model inference
+    ML-->>API: PredictionResult (class + confidence + provenance)
+    API-->>UI: JSON response
+    UI-->>User: Display prediction & visualization
+```
+
+---
+
+## Getting Started
+
+### Prerequisites
+
+- Python 3.10+
+- Node.js 16+
+- npm
+- Git
+
+### Setup
+
+```bash
+# Clone repo
+git clone https://github.com/devjas1/polymeros.git
+cd polymeros
+
+# Backend setup
+pip install -r backend/requirements.txt
+
+# Frontend setup
+cd frontend
+npm install
+cd ..
+```
+
+### Run (Dev Mode)
+
+```bash
+# Backend (FastAPI with live reload)
+uvicorn backend.main:app --reload --port 8000
+
+# Frontend (React)
+cd frontend
+npm start
+```
+
+### Run (Single-Container)
+
+```bash
+python main.py
+```
+
+Access the app at: **http://localhost:8000**
+
+---
+
+## Tests
+
+```bash
+# From repo root
+export PYTHONPATH=$PWD
+pytest backend/tests
+```
+
+---
+
+## Deployment
+
+### Docker
+
+```bash
+docker build -t polymeros .
+docker run -p 8000:8000 polymeros
+```
+
+### Hugging Face Spaces
+
+PolymerOS is optimized for deployment on Hugging Face Spaces with unified API + UI on a single port.
+
+---
+
+## Contributors
+
+- **Jaser Hasan** — Author & Developer
+- **Dr. Sanmukh Kuppannagari** — Mentor
+- **Dr. Metin Karailyan** — Mentor
+
+---
+
+## License
+
+Apache 2.0 — see [LICENSE](LICENSE)
+
+---

backend/__init__.py

@@ -0,0 +1 @@
+# Backend Package

backend/config.py

@@ -0,0 +1,4 @@
+# Configuration constants for the backend API
+TARGET_LEN = 500
+LABEL_MAP = {0: "Stable", 1: "Weathered"}
+SAMPLE_DATA_DIR = "sample_data"

backend/main.py

@@ -0,0 +1,598 @@
+# pylint: disable=unused-argument, unused-import, redefined-outer-name,
+# missing-class-docstring, wrong-import-order, missing-function-docstring
+# missing-module-docstring, broad-except, too-many-locals, too-many-stat
+"""
+FastAPI main application.
+Serves both the API endpoints and React frontend static files.
+Single-container deployment for Hugging Face Spaces compatibility.
+"""
+
+import os
+from datetime import datetime
+from typing import List, Dict, Any, Optional
+from pathlib import Path
+import uuid
+import statistics
+import math
+from pydantic import BaseModel, Field
+
+
+from fastapi import FastAPI, HTTPException, File, UploadFile, BackgroundTasks, APIRouter
+from fastapi.middleware.cors import CORSMiddleware
+from fastapi.staticfiles import StaticFiles
+from fastapi.responses import FileResponse, JSONResponse
+from fastapi.exceptions import RequestValidationError
+from contextlib import asynccontextmanager
+
+from backend.service import ml_service, MLServiceError
+from backend.utils.model_manager import model_manager
+from backend.utils.enhanced_ml_service import enhanced_ml_service
+from .pydantic_models import (
+    SpectrumData,
+    AnalysisRequest,
+    BatchAnalysisRequest,
+    ComparisonRequest,
+    PredictionResult,
+    ExplanationResult,
+    BatchPredictionResult,
+    ComparisonResult,
+    ModelInfo,
+    SystemInfo,
+    SystemHealth,
+    ErrorResponse,
+    BatchError,
+)
+
+from backend.utils.prepare_data import prepare_data as run_prepare_data
+from backend.utils.train import train as run_training_job
+from backend.utils.multifile import parse_spectrum_data
+
+
+@asynccontextmanager
+async def lifespan(app: FastAPI):
+    """Application lifespan manager"""
+    # Startup
+    print("🚀 Starting Polymer Aging ML API...")
+
+    # Warmup models (load them into cache)
+    # Use the centralized model_manager for loading
+    try:
+        print("Pre-loading models via ModelManager...")
+        available_models_info = model_manager.get_available_models()
+        print(f"✅ Discovered {len(available_models_info)} models.")
+
+        # Warmup with a dummy spectrum if models are available
+        loaded_models_count = 0
+        for model_info in available_models_info:
+            if model_info.available:
+                ml_service.model_manager.load_model(
+                    model_info.name
+                )  # Ensure models are loaded into ml_service's manager
+                loaded_models_count += 1
+        print(f"✅ {loaded_models_count} models loaded into ModelManager.")
+
+        if loaded_models_count > 0:
+            dummy_spectrum = SpectrumData(
+                x_values=list(range(200, 4000, 10)),
+                y_values=[0.5] * len(list(range(200, 4000, 10))),
+                filename="warmup",
+            )
+            print("✅ Models warmed up successfully")
+    except (
+        KeyError,
+        ValueError,
+        RuntimeError,
+    ) as e:  # Replace with specific exceptions
+        print(f"⚠️ Model warmup failed: {e}")
+
+    yield
+
+    # Shutdown
+    print("🔄 Shutting down Polymer Aging ML API...")
+
+
+# --- In-memory DB for Training Jobs ---
+training_jobs: Dict[str, Dict[str, Any]] = {}
+
+# --- Pydantic Models Building Blocks for Training API ---
+
+
+class PrepareDataRequest(BaseModel):
+    raw_data_path: str = Field(
+        ..., description="Path to the raw dataset (e.g., a single CSV or a directory)."
+    )
+    output_path: str = Field(
+        default="data/processed",
+        description="Directory to save the processed train/val/test splits.",
+    )
+
+
+class TrainingJobConfig(BaseModel):
+    experiment_name: str = "PolymerAgingClassification"
+    run_name: str = Field(
+        default_factory=lambda: f"run_{datetime.now().strftime('%Y%m%d_%H%M%S')}"
+    )
+    data_dir: str = "data/processed"
+    train_csv: str = "train.csv"
+    val_csv: str = "validation.csv"
+    model_name: str
+    epochs: int = 50
+    batch_size: int = 32
+    learning_rate: float = 0.001
+    optimizer: str = "Adam"
+    loss_function: str = "CrossEntropyLoss"
+
+
+class TrainingJobStatus(BaseModel):
+    job_id: str
+    status: str  # PENDING, RUNNING, COMPLETED, FAILED
+    config: TrainingJobConfig
+    progress: float = 0.0
+    current_epoch: int = 0
+    mlflow_run_id: Optional[str] = None
+    metrics: Dict[str, list] = Field(
+        default_factory=lambda: {"train_loss": [], "val_loss": []}
+    )
+    error: Optional[str] = None
+    created_at: str
+
+
+app = FastAPI(
+    title="Polymer Aging ML API",
+    description="AI-driven polymer aging prediction and classification using Raman and FTIR spectroscopy",
+    version="1.0.0",
+    docs_url="/api/v1/docs",
+    redoc_url="/api/v1/redoc",
+    openapi_url="/api/v1/openapi.json",
+    lifespan=lifespan,
+)
+
+# --- Hardened CORS Configuration ---
+# Only allow specific origins. In production (HF), the frontend is same-origin.
+_allowed = os.getenv(
+    "CORS_ALLOWED_ORIGINS", "http://localhost:3000,http://localhost:7860"
+)
+origins = [o.strip() for o in _allowed.split(",") if o.strip()]
+
+app.add_middleware(
+    CORSMiddleware,
+    allow_origins=origins,
+    allow_credentials=True,
+    allow_methods=["GET", "POST", "OPTIONS"],  # Limit to only what the app needs
+    allow_headers=["*"],
+)
+
+
+# Error handlers
+@app.exception_handler(RequestValidationError)
+async def validation_exception_handler(request, exc):
+    # Sanitize validation errors to ensure they are JSON serializable
+    cleaned_errors = []
+    for error in exc.errors():
+        cleaned_error = error.copy()
+        if "ctx" in cleaned_error and isinstance(cleaned_error["ctx"], dict):
+            # The context can contain non-serializable objects like exceptions.
+            # We'll convert all context values to strings for a safe response.
+            cleaned_error["ctx"] = {k: str(v) for k, v in cleaned_error["ctx"].items()}
+        cleaned_errors.append(cleaned_error)
+    return JSONResponse(
+        status_code=422,
+        content=ErrorResponse(
+            error="Validation Error",
+            error_code="VALIDATION_ERROR",
+            details={"validation_errors": cleaned_errors},
+            timestamp=datetime.now().isoformat(),
+            request_id=str(uuid.uuid4()),
+        ).dict(),
+    )
+
+
+@app.exception_handler(Exception)
+async def general_exception_handler(request, exc):
+    return JSONResponse(
+        status_code=500,
+        content=ErrorResponse(
+            error=str(exc) if str(exc) else "Internal server error",
+            error_code="INTERNAL_ERROR",
+            details={},  # Provide an empty dictionary as the default value
+            timestamp=datetime.now().isoformat(),
+            request_id=str(uuid.uuid4()),
+        ).dict(),
+    )
+
+
+# --- API Router for Training ---
+training_router = APIRouter(prefix="/api/v1/training", tags=["Training"])
+
+
+@training_router.post("/prepare-data", summary="Prepare and Split Dataset")
+def prepare_data_endpoint(request: PrepareDataRequest):
+    """
+    Triggers the data preparation script to create train/validation/test splits.
+    In a real web app, this would handle an uploaded zip file.
+    """
+    try:
+        raw_path = Path(request.raw_data_path)
+        output_path = Path(request.output_path)
+        run_prepare_data(data_path=raw_path, output_path=output_path)
+        return {"message": f"Data preparation complete. Splits saved to {output_path}."}
+    except Exception as e:
+        raise HTTPException(status_code=500, detail=str(e)) from e
+
+
+@training_router.post(
+    "/start",
+    response_model=TrainingJobStatus,
+    status_code=202,
+    summary="Start a New Training Job",
+)
+def start_training(config: TrainingJobConfig, background_tasks: BackgroundTasks):
+    """
+    Starts a new model training job in the background.
+    """
+    job_id = str(uuid.uuid4())
+    job_status = TrainingJobStatus(
+        job_id=job_id,
+        status="PENDING",
+        config=config,
+        created_at=datetime.now().isoformat(),
+    ).dict()
+    training_jobs[job_id] = job_status
+
+    # Add the long-running training task to the background
+    background_tasks.add_task(
+        run_training_job, config=config.dict(), jobs_db=training_jobs, job_id=job_id
+    )
+
+    return job_status
+
+
+@training_router.get(
+    "/jobs", response_model=List[TrainingJobStatus], summary="List All Training Jobs"
+)
+def list_training_jobs():
+    """Retrieves the status of all training jobs."""
+    return list(training_jobs.values())
+
+
+@training_router.get(
+    "/jobs/{job_id}",
+    response_model=TrainingJobStatus,
+    summary="Get Training Job Status",
+)
+def get_training_job_status(job_id: str):
+    """Retrieves the status of a specific training job by its ID."""
+    if job_id not in training_jobs:
+        raise HTTPException(status_code=404, detail="Training job not found")
+    return training_jobs[job_id]
+
+
+# API Routes
+@app.get("/api/v1/health")
+async def health_check():
+    """Health check endpoint"""
+    return {"status": "healthy", "timestamp": datetime.now().isoformat()}
+
+
+@app.get("/api/v1/system", response_model=SystemInfo)
+async def get_system_info():
+    """Get system information and available models"""
+    return ml_service.get_system_info()
+
+
+@app.get("/api/v1/models", response_model=List[ModelInfo])
+async def get_models():
+    """Get list of available models"""
+    print("🔍 Fetching available models...")
+    # Directly use the centralized model manager
+    models = model_manager.get_available_models()
+    if not models:
+        print(
+            "⚠️ No models found via ModelManager. Falling back to filesystem scan (this should ideally not be needed)."
+        )
+        # This fallback is now less critical as ModelManager should handle discovery
+        # but keeping it for extreme resilience as per original request.
+        # The ModelManager itself already checks for weight file existence.
+    return models
+
+
+@app.post("/api/v1/analyze", response_model=PredictionResult)
+async def analyze_spectrum(request: AnalysisRequest):
+    """Analyze a single spectrum"""
+    try:
+        result = ml_service.run_inference(
+            request.spectrum,
+            request.model_name,
+            request.modality,
+            request.include_provenance,
+        )
+        return result
+    except MLServiceError as e:
+        raise HTTPException(status_code=400, detail=str(e)) from e
+
+
+# ** fix-429e36db-a89a-42f9-8b64-9bdfd16b01bc
+@app.post("/api/v1/explain")
+async def explain_spectrum(request: AnalysisRequest):
+    """Analyze a spectrum with explainability features"""
+    try:
+        # Ensure we pass modality and use the same include_provenance flag
+        result = enhanced_ml_service.predict_with_explanation(
+            request.spectrum,  # SpectrumData
+            request.model_name,  # model name
+            modality=request.modality,  # pass modality (raman/ftir)
+            include_feature_importance=request.include_provenance,
+        )
+        return result
+    except Exception as e:
+        # Log full traceback for debugging
+        import traceback, sys
+
+        print(
+            "[explain] Error during prediction with explanation:",
+            str(e),
+            file=sys.stderr,
+        )
+        traceback.print_exc()
+        raise HTTPException(status_code=400, detail=str(e)) from e
+
+
+@app.post("/api/v1/explain/batch")
+async def explain_batch_spectra(request: BatchAnalysisRequest):
+    """Analyze multiple spectra with explainability features"""
+    if len(request.spectra) > 50:  # Lower limit for explanation requests
+        raise HTTPException(
+            status_code=400,
+            detail="Batch explainability requests limited to 50 spectra",
+        )
+
+    try:
+        results = enhanced_ml_service.batch_predict_with_explanation(
+            request.spectra,
+            request.model_name,
+            modality=request.modality,  # Pass modality to the enhanced service
+            include_feature_importance=request.include_provenance,  # Use include_provenance for feature importance
+        )
+
+        return {
+            "results": results,
+            "total_processed": len(results),
+            "model_used": request.model_name,
+            "timestamp": datetime.now().isoformat(),
+        }
+    except Exception as e:
+        raise HTTPException(status_code=400, detail=str(e)) from e
+
+
+# ** fix-429e36db-a89a-42f9-8b64-9bdfd16b01bc
+
+
+@app.post("/api/v1/analyze/batch", response_model=BatchPredictionResult)
+async def analyze_batch(request: BatchAnalysisRequest):
+    """Analyze multiple spectra in batch"""
+    if len(request.spectra) > 100:
+        raise HTTPException(
+            status_code=400, detail="Batch size cannot exceed 100 spectra"
+        )
+
+    start_time = datetime.now()
+    results = []
+    errors = []
+
+    for spectrum in request.spectra:
+        try:
+            result = ml_service.run_inference(
+                spectrum,
+                request.model_name,
+                request.modality,
+                request.include_provenance,
+            )
+            results.append(result)
+        except (ValueError, KeyError, RuntimeError) as e:
+            errors.append(BatchError(filename=spectrum.filename, error=str(e)))
+
+    total_time = (datetime.now() - start_time).total_seconds()
+
+    # Initialize summary statistics with default values
+    average_confidence = 0.0
+    confidence_std = 0.0
+    min_confidence = 0.0
+    max_confidence = 0.0
+    predictions = []
+
+    # Calculate summary statistics only on successful results
+    if results:
+        # Calculate summary statistics
+        confidences = [r.confidence for r in results]
+        predictions = [r.prediction for r in results]
+
+        if confidences:
+            average_confidence = statistics.mean(confidences)
+            confidence_std = (
+                statistics.stdev(confidences) if len(confidences) > 1 else 0.0
+            )
+            min_confidence = min(confidences)
+            max_confidence = max(confidences)
+
+    summary = {
+        "total_spectra_requested": len(request.spectra),
+        "total_spectra_processed": len(results),
+        "total_spectra_failed": len(errors),
+        "stable_count": sum(1 for p in predictions if p == 0) if results else 0,
+        "weathered_count": sum(1 for p in predictions if p == 1) if results else 0,
+        "average_confidence": average_confidence if results else 0.0,
+        "confidence_std": confidence_std if results else 0.0,
+        "min_confidence": min_confidence if results else 0.0,
+        "max_confidence": max_confidence if results else 0.0,
+    }
+
+    return BatchPredictionResult(
+        results=results,
+        errors=errors,
+        summary=summary,
+        total_processing_time=total_time,
+        timestamp=datetime.now().isoformat(),
+    )
+
+
+@app.post("/api/v1/compare", response_model=ComparisonResult)
+async def compare_models(request: ComparisonRequest):
+    """Compare multiple models on a single spectrum"""
+    try:
+        available_models = ml_service.get_available_models()
+
+        if request.model_names:
+            models_to_test = [
+                m.name
+                for m in available_models
+                if m.name in request.model_names and m.available
+            ]
+        else:
+            models_to_test = [m.name for m in available_models if m.available]
+
+        if not models_to_test:
+            raise HTTPException(status_code=400, detail="No available models found")
+
+        spectrum_id = str(uuid.uuid4())
+        model_results = {}
+        confidences = []
+        predictions = []
+
+        for model_name in models_to_test:
+            result = ml_service.run_inference(
+                request.spectrum,
+                model_name,
+                request.modality,
+                request.include_provenance,
+            )
+            model_results[model_name] = result
+            confidences.append(result.confidence)
+            predictions.append(result.prediction)
+
+        # Calculate consensus and agreement
+        if predictions:
+            # Simple majority vote for consensus
+            prediction_counts = {0: predictions.count(0), 1: predictions.count(1)}
+            consensus = max(prediction_counts, key=prediction_counts.get)
+
+            # Agreement score: percentage of models that agree with consensus
+            agreement_score = prediction_counts[consensus] / len(predictions)
+
+            # Confidence variance
+            if len(confidences) > 1:
+                confidence_variance = statistics.variance(confidences)
+            else:
+                confidence_variance = 0.0
+        else:
+            consensus = None
+            agreement_score = 0.0
+            confidence_variance = 0.0
+
+        return ComparisonResult(
+            spectrum_id=spectrum_id,
+            model_results=model_results,
+            consensus_prediction=consensus,
+            confidence_variance=confidence_variance,
+            agreement_score=agreement_score,
+            timestamp=datetime.now().isoformat(),
+        )
+
+    except (MLServiceError, KeyError, ValueError, RuntimeError) as e:
+        raise HTTPException(status_code=400, detail=str(e)) from e
+
+
+@app.post("/api/v1/upload", response_model=SpectrumData)
+async def upload_spectrum_file(file: UploadFile = File(...)):
+    """Upload and parse a spectrum file"""
+    try:
+        # Read file content
+        content = await file.read()
+
+        # Parse spectrum data using existing utility
+        x_data, y_data = parse_spectrum_data(
+            content.decode("utf-8"), file.filename or "unknown_filename"
+        )
+
+        return SpectrumData(
+            x_values=x_data.tolist(), y_values=y_data.tolist(), filename=file.filename
+        )
+
+    except Exception as e:
+        raise HTTPException(
+            status_code=400, detail=f"Failed to parse spectrum file: {str(e)}"
+        ) from e
+
+
+# Backward compatibility routes (redirect to v1)
+@app.get("/api/health")
+async def health_check_legacy():
+    """Legacy health check endpoint - redirects to v1"""
+    return await health_check()
+
+
+@app.get("/api/system")
+async def get_system_info_legacy():
+    """Legacy system info endpoint - redirects to v1"""
+    return await get_system_info()
+
+
+@app.get("/api/models")
+async def get_models_legacy():
+    """Legacy models endpoint - redirects to v1"""
+    return await get_models()
+
+
+# Static file serving for React frontend
+# frontend_dist_path = Path("frontend/dist")
+BASE_DIR = Path(__file__).resolve().parent.parent
+frontend_dist_path = BASE_DIR / "frontend" / "dist"
+if frontend_dist_path.exists() and frontend_dist_path.is_dir():
+    # Mount static files for built React app
+    app.mount("/static", StaticFiles(directory="frontend/dist/static"), name="static")
+
+    @app.get("/")
+    async def serve_frontend():
+        """Serve React frontend"""
+        index_path = frontend_dist_path / "index.html"
+        if index_path.exists():
+            return FileResponse(index_path)
+        return JSONResponse(
+            content={"error": "Frontend index.html not found"}, status_code=404
+        )
+
+    @app.get("/{path:path}")
+    async def serve_frontend_routes(path: str):
+        """Serve React frontend for all non-API routes (SPA routing)"""
+        if path.startswith("api/"):
+            raise HTTPException(status_code=404, detail="API endpoint not found")
+
+        file_path = frontend_dist_path / path
+        if file_path.exists() and file_path.is_file():
+            return FileResponse(file_path)
+        else:
+            # For SPA routing, return index.html if it exists
+            index_path = frontend_dist_path / "index.html"
+            if index_path.exists():
+                return FileResponse(index_path)
+            raise HTTPException(status_code=404, detail="Frontend not found")
+
+else:
+
+    @app.get("/")
+    async def root():
+        """Root endpoint when frontend is not built"""
+        return {
+            "message": "Polymer Aging ML API",
+            "status": "Frontend not built. Build React frontend and place in frontend/dist/",
+            "api_docs": "/api/docs",
+            "version": "1.0.0",
+        }
+
+
+# Include the new training router in the main application
+app.include_router(training_router)
+
+if __name__ == "__main__":
+    import uvicorn
+
+    uvicorn.run(app, host="0.0.0.0", port=8000)

backend/models/enhanced_cnn.py

@@ -0,0 +1,410 @@
+"""
+All neural network blocks and architectures in models/enhanced_cnn.py are custom implementations, developed to expand the model registry for advanced polymer spectral classification. While inspired by established deep learning concepts (such as residual connections, attention mechanisms, and multi-scale convolutions), they are are unique to this project and tailored for 1D spectral data.
+
+Registry expansion: The purpose is to enrich the available models.
+Literature inspiration: SE-Net, ResNet, Inception.
+"""
+
+import torch
+import torch.nn as nn
+
+
+class AttentionBlock1D(nn.Module):
+    """1D attention mechanism for spectral data."""
+
+    def __init__(self, channels: int, reduction: int = 8):
+        super().__init__()
+        self.channels = channels
+        self.global_pool = nn.AdaptiveAvgPool1d(1)
+        self.fc = nn.Sequential(
+            nn.Linear(channels, channels // reduction),
+            nn.ReLU(inplace=True),
+            nn.Linear(channels // reduction, channels),
+            nn.Sigmoid(),
+        )
+
+    def forward(self, x):
+        # x shape: [batch, channels, length]
+        b, c, _ = x.size()
+
+        # Global average pooling
+        y = self.global_pool(x).view(b, c)
+
+        # Fully connected layers
+        y = self.fc(y).view(b, c, 1)
+
+        # Apply attention weights
+        return x * y.expand_as(x)
+
+
+class EnhancedResidualBlock1D(nn.Module):
+    """Enhanced residual block with attention and improved normalization."""
+
+    def __init__(
+        self,
+        in_channels: int,
+        out_channels: int,
+        kernel_size: int = 3,
+        use_attention: bool = True,
+        dropout_rate: float = 0.1,
+    ):
+        super().__init__()
+        padding = kernel_size // 2
+
+        self.conv1 = nn.Conv1d(in_channels, out_channels, kernel_size, padding=padding)
+        self.bn1 = nn.BatchNorm1d(out_channels)
+        self.relu = nn.ReLU(inplace=True)
+
+        self.conv2 = nn.Conv1d(out_channels, out_channels, kernel_size, padding=padding)
+        self.bn2 = nn.BatchNorm1d(out_channels)
+
+        self.dropout = nn.Dropout1d(dropout_rate) if dropout_rate > 0 else nn.Identity()
+
+        # Skip connection
+        self.skip = (
+            nn.Identity()
+            if in_channels == out_channels
+            else nn.Sequential(
+                nn.Conv1d(in_channels, out_channels, kernel_size=1),
+                nn.BatchNorm1d(out_channels),
+            )
+        )
+
+        # Attention mechanism
+        self.attention = (
+            AttentionBlock1D(out_channels) if use_attention else nn.Identity()
+        )
+
+    def forward(self, x):
+        identity = self.skip(x)
+
+        out = self.conv1(x)
+        out = self.bn1(out)
+        out = self.relu(out)
+        out = self.dropout(out)
+
+        out = self.conv2(out)
+        out = self.bn2(out)
+
+        # Apply attention
+        out = self.attention(out)
+
+        out = out + identity
+        return self.relu(out)
+
+
+class MultiScaleConvBlock(nn.Module):
+    """Multi-scale convolution block for capturing features at different scales."""
+
+    def __init__(self, in_channels: int, out_channels: int):
+        super().__init__()
+
+        # Different kernel sizes for multi-scale feature extraction
+        self.conv1 = nn.Conv1d(in_channels, out_channels // 4, kernel_size=3, padding=1)
+        self.conv2 = nn.Conv1d(in_channels, out_channels // 4, kernel_size=5, padding=2)
+        self.conv3 = nn.Conv1d(in_channels, out_channels // 4, kernel_size=7, padding=3)
+        self.conv4 = nn.Conv1d(in_channels, out_channels // 4, kernel_size=9, padding=4)
+
+        self.bn = nn.BatchNorm1d(out_channels)
+        self.relu = nn.ReLU(inplace=True)
+
+    def forward(self, x):
+        # Parallel convolutions with different kernel sizes
+        out1 = self.conv1(x)
+        out2 = self.conv2(x)
+        out3 = self.conv3(x)
+        out4 = self.conv4(x)
+
+        # Concatenate along channel dimension
+        out = torch.cat([out1, out2, out3, out4], dim=1)
+        out = self.bn(out)
+        return self.relu(out)
+
+
+class EnhancedCNN(nn.Module):
+    """Enhanced CNN with attention, multi-scale features, and improved architecture."""
+
+    def __init__(
+        self,
+        input_length: int = 500,
+        num_classes: int = 2,
+        dropout_rate: float = 0.2,
+        use_attention: bool = True,
+    ):
+        super().__init__()
+
+        self.input_length = input_length
+        self.num_classes = num_classes
+
+        # Initial feature extraction
+        self.initial_conv = nn.Sequential(
+            nn.Conv1d(1, 32, kernel_size=7, padding=3),
+            nn.BatchNorm1d(32),
+            nn.ReLU(inplace=True),
+            nn.MaxPool1d(kernel_size=2),
+        )
+
+        # Multi-scale feature extraction
+        self.multiscale_block = MultiScaleConvBlock(32, 64)
+        self.pool1 = nn.MaxPool1d(kernel_size=2)
+
+        # Enhanced residual blocks
+        self.res_block1 = EnhancedResidualBlock1D(64, 96, use_attention=use_attention)
+        self.pool2 = nn.MaxPool1d(kernel_size=2)
+
+        self.res_block2 = EnhancedResidualBlock1D(96, 128, use_attention=use_attention)
+        self.pool3 = nn.MaxPool1d(kernel_size=2)
+
+        self.res_block3 = EnhancedResidualBlock1D(128, 160, use_attention=use_attention)
+
+        # Global feature extraction
+        self.global_pool = nn.AdaptiveAvgPool1d(1)
+
+        # Calculate feature size after convolutions
+        self.feature_size = 160
+
+        # Enhanced classifier with dropout
+        self.classifier = nn.Sequential(
+            nn.Linear(self.feature_size, 256),
+            nn.BatchNorm1d(256),
+            nn.ReLU(inplace=True),
+            nn.Dropout(dropout_rate),
+            nn.Linear(256, 128),
+            nn.BatchNorm1d(128),
+            nn.ReLU(inplace=True),
+            nn.Dropout(dropout_rate),
+            nn.Linear(128, 64),
+            nn.BatchNorm1d(64),
+            nn.ReLU(inplace=True),
+            nn.Dropout(dropout_rate / 2),
+            nn.Linear(64, num_classes),
+        )
+
+        # Initialize weights
+        self._initialize_weights()
+
+    def _initialize_weights(self):
+        """Initialize model weights using Xavier initialization."""
+        for m in self.modules():
+            if isinstance(m, nn.Conv1d):
+                nn.init.xavier_uniform_(m.weight)
+                if m.bias is not None:
+                    nn.init.constant_(m.bias, 0)
+            elif isinstance(m, nn.Linear):
+                nn.init.xavier_uniform_(m.weight)
+                nn.init.constant_(m.bias, 0)
+            elif isinstance(m, nn.BatchNorm1d):
+                nn.init.constant_(m.weight, 1)
+                nn.init.constant_(m.bias, 0)
+
+    def forward(self, x):
+        # Ensure input is 3D: [batch, channels, length]
+        if x.dim() == 2:
+            x = x.unsqueeze(1)
+
+        # Feature extraction
+        x = self.initial_conv(x)
+        x = self.multiscale_block(x)
+        x = self.pool1(x)
+
+        x = self.res_block1(x)
+        x = self.pool2(x)
+
+        x = self.res_block2(x)
+        x = self.pool3(x)
+
+        x = self.res_block3(x)
+
+        # Global pooling
+        x = self.global_pool(x)
+        x = x.view(x.size(0), -1)
+
+        # Classification
+        x = self.classifier(x)
+
+        return x
+
+    def get_feature_maps(self, x):
+        """Extract intermediate feature maps for visualization."""
+        if x.dim() == 2:
+            x = x.unsqueeze(1)
+
+        features = {}
+
+        x = self.initial_conv(x)
+        features["initial"] = x
+
+        x = self.multiscale_block(x)
+        features["multiscale"] = x
+        x = self.pool1(x)
+
+        x = self.res_block1(x)
+        features["res1"] = x
+        x = self.pool2(x)
+
+        x = self.res_block2(x)
+        features["res2"] = x
+        x = self.pool3(x)
+
+        x = self.res_block3(x)
+        features["res3"] = x
+
+        return features
+
+
+class EfficientSpectralCNN(nn.Module):
+    """Efficient CNN designed for real-time inference with good performance."""
+
+    def __init__(self, input_length: int = 500, num_classes: int = 2):
+        super().__init__()
+
+        self.input_length = int(input_length)
+        self.num_classes = int(num_classes)
+
+        # Efficient feature extraction with depthwise separable convolutions
+        self.features = nn.Sequential(
+            # Initial convolution
+            nn.Conv1d(1, 32, kernel_size=7, padding=3),
+            nn.BatchNorm1d(32),
+            nn.ReLU(inplace=True),
+            nn.MaxPool1d(2),
+            # Depthwise separable convolutions
+            self._make_depthwise_sep_conv(32, 64),
+            nn.MaxPool1d(2),
+            self._make_depthwise_sep_conv(64, 96),
+            nn.MaxPool1d(2),
+            self._make_depthwise_sep_conv(96, 128),
+            nn.MaxPool1d(2),
+            # Final feature extraction
+            nn.Conv1d(128, 160, kernel_size=3, padding=1),
+            nn.BatchNorm1d(160),
+            nn.ReLU(inplace=True),
+            nn.AdaptiveAvgPool1d(1),
+        )
+
+        # Lightweight classifier
+        self.classifier = nn.Sequential(
+            nn.Linear(160, 64),
+            nn.ReLU(inplace=True),
+            nn.Dropout(0.1),
+            nn.Linear(64, num_classes),
+        )
+
+        self._initialize_weights()
+
+    def _make_depthwise_sep_conv(self, in_channels, out_channels):
+        """Create depthwise separable convolution block."""
+        return nn.Sequential(
+            # Depthwise convolution
+            nn.Conv1d(
+                in_channels, in_channels, kernel_size=3, padding=1, groups=in_channels
+            ),
+            nn.BatchNorm1d(in_channels),
+            nn.ReLU(inplace=True),
+            # Pointwise convolution
+            nn.Conv1d(in_channels, out_channels, kernel_size=1),
+            nn.BatchNorm1d(out_channels),
+            nn.ReLU(inplace=True),
+        )
+
+    def _initialize_weights(self):
+        """Initialize model weights."""
+        for m in self.modules():
+            if isinstance(m, nn.Conv1d):
+                nn.init.kaiming_normal_(m.weight, mode="fan_out", nonlinearity="relu")
+                if m.bias is not None:
+                    nn.init.constant_(m.bias, 0)
+            elif isinstance(m, nn.Linear):
+                nn.init.xavier_uniform_(m.weight)
+                nn.init.constant_(m.bias, 0)
+            elif isinstance(m, nn.BatchNorm1d):
+                nn.init.constant_(m.weight, 1)
+                nn.init.constant_(m.bias, 0)
+
+    def forward(self, x):
+        if x.dim() == 2:
+            x = x.unsqueeze(1)
+
+        x = self.features(x)
+        x = x.view(x.size(0), -1)
+        x = self.classifier(x)
+
+        return x
+
+
+class HybridSpectralNet(nn.Module):
+    """Hybrid network combining CNN and attention mechanisms."""
+
+    def __init__(self, input_length: int = 500, num_classes: int = 2):
+        super().__init__()
+
+        self.input_length = int(input_length)
+        self.num_classes = int(num_classes)
+
+        # CNN backbone
+        self.cnn_backbone = nn.Sequential(
+            nn.Conv1d(1, 64, kernel_size=7, padding=3),
+            nn.BatchNorm1d(64),
+            nn.ReLU(inplace=True),
+            nn.MaxPool1d(2),
+            nn.Conv1d(64, 128, kernel_size=5, padding=2),
+            nn.BatchNorm1d(128),
+            nn.ReLU(inplace=True),
+            nn.MaxPool1d(2),
+            nn.Conv1d(128, 256, kernel_size=3, padding=1),
+            nn.BatchNorm1d(256),
+            nn.ReLU(inplace=True),
+        )
+
+        # Self-attention layer
+        self.attention = nn.MultiheadAttention(
+            embed_dim=256, num_heads=8, dropout=0.1, batch_first=True
+        )
+
+        # Final pooling and classification
+        self.global_pool = nn.AdaptiveAvgPool1d(1)
+        self.classifier = nn.Sequential(
+            nn.Linear(256, 128),
+            nn.ReLU(inplace=True),
+            nn.Dropout(0.2),
+            nn.Linear(128, num_classes),
+        )
+
+    def forward(self, x):
+        if x.dim() == 2:
+            x = x.unsqueeze(1)
+
+        # CNN feature extraction
+        x = self.cnn_backbone(x)
+
+        # Prepare for attention: [batch, length, channels]
+        x = x.transpose(1, 2)
+
+        # Self-attention
+        attn_out, _ = self.attention(x, x, x)
+
+        # Back to [batch, channels, length]
+        x = attn_out.transpose(1, 2)
+
+        # Global pooling and classification
+        x = self.global_pool(x)
+        x = x.view(x.size(0), -1)
+        x = self.classifier(x)
+
+        return x
+
+
+def create_enhanced_model(model_type: str = "enhanced", **kwargs):
+    """Factory function to create enhanced models."""
+    models = {
+        "enhanced": EnhancedCNN,
+        "efficient": EfficientSpectralCNN,
+        "hybrid": HybridSpectralNet,
+    }
+
+    if model_type not in models:
+        raise ValueError(
+            f"Unknown model type: {model_type}. Available: {list(models.keys())}"
+        )
+
+    return models[model_type](**kwargs)

backend/models/figure2_cnn.py

@@ -0,0 +1,77 @@
+# 📌 MODEL DESIGNATION:
+# Figure2CNN is validated ONLY for RAMAN spectra input.
+# Any use for FTIR modeling is invalid and deprecated.
+# See milestone: @figure2cnn-raman-only-milestone
+
+import torch
+import torch.nn as nn
+
+
+class Figure2CNN(nn.Module):
+    """ 
+    CNN architecture based on Figure 2 of the referenced research paper.
+    Designed for 1D spectral data input of length 500
+    """
+
+    def __init__(self, input_length=500, input_channels=1):
+        super(Figure2CNN, self).__init__()
+
+        self.input_channels = input_channels
+
+
+        self.conv_block = nn.Sequential(
+            nn.Conv1d(input_channels, 16, kernel_size=5, padding=2),
+            nn.ReLU(),
+            nn.MaxPool1d(kernel_size=2),
+
+            nn.Conv1d(16, 32, kernel_size=5, padding=2),
+            nn.ReLU(),
+            nn.MaxPool1d(kernel_size=2),
+
+            nn.Conv1d(32, 64, kernel_size=5, padding=2),
+            nn.ReLU(),
+            nn.MaxPool1d(kernel_size=2),
+
+            nn.Conv1d(64, 128, kernel_size=5, padding=2),
+            nn.ReLU(),
+            nn.MaxPool1d(kernel_size=2),
+        )
+
+        # Dynamically calculate flattened size after conv + pooling
+        self.flattened_size = self._get_flattened_size(input_channels, input_length)
+
+        self.classifier = nn.Sequential(
+            nn.Linear(self.flattened_size, 256),
+            nn.ReLU(),
+            nn.Linear(256, 128),
+            nn.ReLU(),
+            nn.Linear(128, 2)  # Binary output
+        )
+
+    def _get_flattened_size(self,input_channels, input_length):
+        with torch.no_grad():
+            dummy_input = torch.zeros(1, input_channels, input_length)
+            out = self.conv_block(dummy_input)
+            return out.view(1, -1).shape[1]
+
+    def forward(self, x):
+        """ 
+        Defines the forward pass of the Figure2CNN model.
+
+        Args:
+            x (torch.Tensor): Input tensor of shape (batch_size, channels, input_length).
+
+        Returns:
+            torch.Tensor: Output tensor containing class scores.
+        """
+        x = self.conv_block(x)
+        x = x.view(x.size(0), -1)  # Flatten
+        return self.classifier(x)
+
+    def describe_model(self):
+        """Print architecture and flattened size (for debug). """
+        print(r"\n Model Summary:")
+        print(r" - Conv Block: 4 Layers")
+        print(f" - Input length: {self.flattened_size} after conv/pool")
+        print(f" - Classifier: {self.classifier}\n")
+        

backend/models/registry.py

@@ -0,0 +1,246 @@
+# models/registry.py
+from typing import Callable, Dict, List, Any
+from .figure2_cnn import Figure2CNN
+from .resnet_cnn import ResNet1D
+from .resnet18_vision import ResNet18Vision
+from .enhanced_cnn import EnhancedCNN, EfficientSpectralCNN, HybridSpectralNet
+
+def _resolve_key(name: str, registry: Dict[str, Any]) -> str:
+    """Resolve a case-insensitive model name to its canonical key."""
+    for key in registry:
+        if key.lower() == name.lower():
+            return key
+    raise KeyError(f"Unknown model '{name}'. Available: {list(registry.keys())}")
+
+# Internal registry of model builders keyed by short name.
+_REGISTRY: Dict[str, Callable[[int], object]] = {
+    "Figure2": lambda L: Figure2CNN(input_length=L),
+    "ResNet": lambda L: ResNet1D(input_length=L),
+    "ResNet18Vision": lambda L: ResNet18Vision(input_length=L),
+    "Enhanced_cnn": lambda L: EnhancedCNN(input_length=L),
+    "Efficient_cnn": lambda L: EfficientSpectralCNN(input_length=L),
+    "Hybrid_Net": lambda L: HybridSpectralNet(input_length=L),
+}
+
+# Model specifications with metadata for enhanced features
+_MODEL_SPECS: Dict[str, Dict[str, Any]] = {
+    "Figure2": {
+        "input_length": 500,
+        "num_classes": 2,
+        "description": "Figure 2 baseline custom implementation",
+        "modalities": ["raman", "ftir"],
+        "citation": "Neo et al., 2023, Resour. Conserv. Recycl., 188, 106718",
+        "performance": {"accuracy": 0.948, "f1_score": 0.943},
+        "parameters": "~500K",
+        "speed": "fast",
+    },
+    "ResNet": {
+        "input_length": 500,
+        "num_classes": 2,
+        "description": "(Residual Network) uses skip connections to train much deeper networks",
+        "modalities": ["raman", "ftir"],
+        "citation": "Custom ResNet implementation",
+        "performance": {"accuracy": 0.962, "f1_score": 0.959},
+        "parameters": "~100K",
+        "speed": "very_fast",
+    },
+    "ResNet18Vision": {
+        "input_length": 500,
+        "num_classes": 2,
+        "description": "excels at image recognition tasks by using 'residual blocks' to train more efficiently",
+        "modalities": ["raman", "ftir"],
+        "citation": "ResNet18 Vision adaptation",
+        "performance": {"accuracy": 0.945, "f1_score": 0.940},
+        "parameters": "~11M",
+        "speed": "medium",
+    },
+    "Enhanced_cnn": {
+        "input_length": 500,
+        "num_classes": 2,
+        "description": "Enhanced CNN with attention mechanisms and multi-scale feature extraction",
+        "modalities": ["raman", "ftir"],
+        "citation": "Custom enhanced architecture with attention",
+        "performance": {"accuracy": 0.975, "f1_score": 0.973},
+        "parameters": "~800K",
+        "speed": "medium",
+        "features": ["attention", "multi_scale", "batch_norm", "dropout"],
+    },
+    "Efficient_cnn": {
+        "input_length": 500,
+        "num_classes": 2,
+        "description": "Efficient CNN optimized for real-time inference with depthwise separable convolutions",
+        "modalities": ["raman", "ftir"],
+        "citation": "Custom efficient architecture",
+        "performance": {"accuracy": 0.955, "f1_score": 0.952},
+        "parameters": "~200K",
+        "speed": "very_fast",
+        "features": ["depthwise_separable", "lightweight", "real_time"],
+    },
+    "Hybrid_Net": {
+        "input_length": 500,
+        "num_classes": 2,
+        "description": "Hybrid network combining CNN backbone with self-attention mechanisms",
+        "modalities": ["raman", "ftir"],
+        "citation": "Custom hybrid CNN-Transformer architecture",
+        "performance": {"accuracy": 0.968, "f1_score": 0.965},
+        "parameters": "~1.2M",
+        "speed": "medium",
+        "features": ["self_attention", "cnn_backbone", "transformer_head"],
+    },
+}
+
+# Placeholder for future model expansions
+_FUTURE_MODELS = {
+    "densenet1d": {
+        "description": "DenseNet1D for spectroscopy with dense connections",
+        "status": "planned",
+        "modalities": ["raman", "ftir"],
+        "features": ["dense_connections", "parameter_efficient"],
+    },
+    "ensemble_cnn": {
+        "description": "Ensemble of multiple CNN variants for robust predictions",
+        "status": "planned",
+        "modalities": ["raman", "ftir"],
+        "features": ["ensemble", "robust", "high_accuracy"],
+    },
+    "vision_transformer": {
+        "description": "Vision Transformer adapted for 1D spectral data",
+        "status": "planned",
+        "modalities": ["raman", "ftir"],
+        "features": ["transformer", "attention", "state_of_art"],
+    },
+    "autoencoder_cnn": {
+        "description": "CNN with autoencoder for unsupervised feature learning",
+        "status": "planned",
+        "modalities": ["raman", "ftir"],
+        "features": ["autoencoder", "unsupervised", "feature_learning"],
+    },
+}
+
+
+def choices():
+    """Return the list of available model keys."""
+    return list(_REGISTRY.keys())
+
+
+def planned_models():
+    """Return the list of planned future model keys."""
+    return list(_FUTURE_MODELS.keys())
+
+
+def build(name: str, input_length: int):
+    """Instantiate a model by short name with the given input length."""
+    key = _resolve_key(name, _REGISTRY)
+    return _REGISTRY[key](input_length)
+
+
+def build_multiple(names: List[str], input_length: int) -> Dict[str, Any]:
+    """Nuild multiple models for comparison."""
+    models = {}
+    for name in names:
+        key = _resolve_key(name, _REGISTRY)
+        models[key] = _REGISTRY[key](input_length)
+    return models
+
+
+def register_model(
+    name: str, builder: Callable[[int], object], spec: Dict[str, Any]
+) -> None:
+    """Dynamically register a new model."""
+    if not callable(builder):
+        raise TypeError("Builder must be a callable that accepts an integer argument.")
+    try:
+        existing_key = _resolve_key(name, _REGISTRY)
+        raise ValueError(f"Model '{name}' already registered as '{existing_key}'.")
+    except KeyError:
+        _REGISTRY[name] = builder
+        _MODEL_SPECS[name] = spec
+
+
+def registry_spec(name: str):
+    """Return expected input length and number of classes for a model key."""
+    key = _resolve_key(name, _MODEL_SPECS)
+    return _MODEL_SPECS[key].copy()
+
+
+def get_model_info(name: str) -> Dict[str, Any]:
+    """Get comprehensive model information including metadata."""
+    try:
+        key = _resolve_key(name, _MODEL_SPECS)
+        return _MODEL_SPECS[key].copy()
+    except KeyError:
+        try:
+            key = _resolve_key(name, _FUTURE_MODELS)
+            return _FUTURE_MODELS[key].copy()
+        except KeyError:
+            raise KeyError(f"Unknown model '{name}'")
+
+
+def models_for_modality(modality: str) -> List[str]:
+    """Get list of models that support a specific modality."""
+    compatible = []
+    for name, spec_info in _MODEL_SPECS.items():
+        if modality in spec_info.get("modalities", []):
+            compatible.append(name)
+    return compatible
+
+
+def validate_model_list(names: List[str]) -> List[str]:
+    """Validate and return list of available models from input list."""
+    valid_models = []
+    for name in names:
+        try:
+            key = _resolve_key(name, _REGISTRY)
+            valid_models.append(key)
+        except KeyError:
+            pass
+
+
+def get_models_metadata() -> Dict[str, Dict[str, Any]]:
+    """Get metadata for all registered models."""
+    return {name: _MODEL_SPECS[name].copy() for name in _MODEL_SPECS}
+
+
+def is_model_compatible(name: str, modality: str) -> bool:
+    """Check if a model is compatible with a specific modality."""
+    try:
+        key = _resolve_key(name, _MODEL_SPECS)
+        return modality in _MODEL_SPECS[key].get("modalities", [])
+    except KeyError:
+        return False
+
+
+def get_model_capabilities(name: str) -> Dict[str, Any]:
+    """Get detailed capabilities of a model."""
+    key = _resolve_key(name, _MODEL_SPECS)
+    spec = _MODEL_SPECS[key].copy()
+    spec.update(
+        {
+            "available": True,
+            "status": "active",
+            "supported_tasks": ["binary_classification"],
+            "performance_metrics": {
+                "supports_confidence": True,
+                "supports_batch": True,
+                "memory_efficient": spec.get("description", "").lower().find("resnet")
+                != -1,
+            },
+        }
+    )
+    return spec
+
+
+__all__ = [
+    "choices",
+    "build",
+    "registry_spec",
+    "build_multiple",
+    "register_model",
+    "get_model_info",
+    "models_for_modality",
+    "validate_model_list",
+    "planned_models",
+    "get_models_metadata",
+    "is_model_compatible",
+    "get_model_capabilities",
+]

backend/models/resnet18_vision.py

@@ -0,0 +1,82 @@
+# models/resnet18_vision.py
+# 1D ResNet-18 style model for spectra: input (B, 1, L)
+import torch
+import torch.nn as nn
+from typing import Callable, List
+
+class BasicBlock1D(nn.Module):
+    expansion = 1
+    def __init__(self, in_planes: int, planes: int, stride: int = 1, downsample: nn.Module | None = None):
+        super().__init__()
+        self.conv1 = nn.Conv1d(in_planes, planes, kernel_size=3, stride=stride, padding=1, bias=False)
+        self.bn1   = nn.BatchNorm1d(planes)
+        self.relu  = nn.ReLU(inplace=True)
+        self.conv2 = nn.Conv1d(planes, planes, kernel_size=3, stride=1, padding=1, bias=False)
+        self.bn2   = nn.BatchNorm1d(planes)
+        self.downsample = downsample
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        identity = x
+        out = self.relu(self.bn1(self.conv1(x)))
+        out = self.bn2(self.conv2(out))
+        if self.downsample is not None:
+            identity = self.downsample(x)
+        out += identity
+        out = self.relu(out)
+        return out
+
+def _make_layer(block: Callable[..., nn.Module], in_planes: int, planes: int, blocks: int, stride: int) -> nn.Sequential:
+    downsample = None
+    if stride != 1 or in_planes != planes * block.expansion:
+        downsample = nn.Sequential(
+            nn.Conv1d(in_planes, planes * block.expansion, kernel_size=1, stride=stride, bias=False),
+            nn.BatchNorm1d(planes * block.expansion),
+        )
+    layers: List[nn.Module] = [block(in_planes, planes, stride, downsample)]
+    in_planes = planes * block.expansion
+    for _ in range(1, blocks):
+        layers.append(block(in_planes, planes))
+    return nn.Sequential(*layers)
+
+class ResNet18Vision(nn.Module):
+    def __init__(self, input_length: int = 500, num_classes: int = 2):
+        super().__init__()
+
+        self.input_length = int(input_length)
+        self.num_classes = int(num_classes)
+
+        # 1D stem
+        self.conv1 = nn.Conv1d(1, 64, kernel_size=7, stride=2, padding=3, bias=False)
+        self.bn1   = nn.BatchNorm1d(64)
+        self.relu  = nn.ReLU(inplace=True)
+        self.maxpool = nn.MaxPool1d(kernel_size=3, stride=2, padding=1)
+
+        # ResNet-18: 2 blocks per layer
+        self.layer1 = _make_layer(BasicBlock1D, 64,  64, blocks=2, stride=1)
+        self.layer2 = _make_layer(BasicBlock1D, 64, 128, blocks=2, stride=2)
+        self.layer3 = _make_layer(BasicBlock1D, 128, 256, blocks=2, stride=2)
+        self.layer4 = _make_layer(BasicBlock1D, 256, 512, blocks=2, stride=2)
+
+        # Global pooling + classifier
+        self.avgpool = nn.AdaptiveAvgPool1d(1)
+        self.fc = nn.Linear(512 * BasicBlock1D.expansion, num_classes)
+
+        # Kaiming init
+        for m in self.modules():
+            if isinstance(m, nn.Conv1d):
+                nn.init.kaiming_normal_(m.weight, mode="fan_out", nonlinearity="relu")
+            elif isinstance(m, (nn.BatchNorm1d, nn.GroupNorm)):
+                nn.init.ones_(m.weight); nn.init.zeros_(m.bias)
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        # x: (B, 1, L)
+        x = self.relu(self.bn1(self.conv1(x)))
+        x = self.maxpool(x)
+        x = self.layer1(x)
+        x = self.layer2(x)
+        x = self.layer3(x)
+        x = self.layer4(x)
+        x = self.avgpool(x)     # (B, C, 1)
+        x = torch.flatten(x, 1) # (B, C)
+        x = self.fc(x)          # (B, num_classes)
+        return x

backend/models/resnet_cnn.py

@@ -0,0 +1,73 @@
+"""
+📌 MODEL DESIGNATION:
+Figure2CNN is validated ONLY for RAMAN spectra input.
+Any use for FTIR modeling is invalid and deprecated.
+See milestone: @figure2cnn-raman-only-milestone
+"""
+import torch
+import torch.nn as nn
+
+
+class ResidualBlock1D(nn.Module):
+    """
+    Basic 1-D residual block:
+    Conv1d -> ReLU -> Conv1d (+ skip connection).
+    If channel count changes, a 1x1 Conv aligns the skip path.
+    """
+
+    def __init__(self, in_channels: int, out_channels: int, kernel_size: int = 3):
+        super().__init__()
+        padding = kernel_size // 2
+
+        self.conv1 = nn.Conv1d(in_channels, out_channels,
+                               kernel_size, padding=padding)
+        self.relu = nn.ReLU(inplace=True)
+        self.conv2 = nn.Conv1d(out_channels, out_channels,
+                               kernel_size, padding=padding)
+
+        self.skip = (
+            nn.Identity()
+            if in_channels == out_channels
+            else nn.Conv1d(in_channels, out_channels, kernel_size=1)
+        )
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        identity = self.skip(x)
+        out = self.relu(self.conv1(x))
+        out = self.conv2(out)
+        return self.relu(out + identity)
+
+    def describe_model(self):
+        """Print architecture and flattened size (for debug). """
+        print(r"\n Model Summary:")
+        print(r" - Conv Block: 4 Layers")
+        print(f" - Input length: {self.flattened_size} after conv/pool")
+        print(f" - Classifier: {self.classifier}\n")
+
+
+class ResNet1D(nn.Module):
+    """
+    Lightweight 1-D ResNet for Raman spectra (length 500, single channel).
+    """
+
+    def __init__(self, input_length: int = 500, num_classes: int = 2):
+        super().__init__()
+
+        self.input_length = int(input_length)
+        self.num_classes = int(num_classes)
+
+        # Three residual stages
+        self.stage1 = ResidualBlock1D(1, 16)
+        self.stage2 = ResidualBlock1D(16, 32)
+        self.stage3 = ResidualBlock1D(32, 64)
+
+        # Global aggregation + classifier
+        self.global_pool = nn.AdaptiveAvgPool1d(1)  # -> [B, 64, 1]
+        self.fc = nn.Linear(64, num_classes)
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        x = self.stage1(x)
+        x = self.stage2(x)
+        x = self.stage3(x)
+        x = self.global_pool(x).squeeze(-1)     # -> [B, 64]
+        return self.fc(x)

backend/models/weights/efficient_cnn_model.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:08ae3befe95b73d80111f669e040d2b185c05e63043850644b9765a4c3013a7d
+size 405858

backend/models/weights/enhanced_cnn_model.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e3d05e9826be3690d5c906a3a814b21d4d778a6cf3f290cd2a1342db8d8dab59
+size 1741892

backend/models/weights/figure2_model.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:852247bf0540aa947c9887a7e004c0858d622cfa0413e9b26bd9f5dab359ad5e
+size 4418520

backend/models/weights/hybrid_net_model.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c6ae29a09550a7cd2bcf6aa63585e8b7713f8d438b41a6e7ac99a7dc0a4334af
+size 1762856

backend/models/weights/resnet18vision_model.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8e08016742f05a0e3d34270a885b67ef0b6d938fcbe8b8ab83256fc0ff1d019d
+size 15458340

backend/models/weights/resnet_model.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4f1d1b5541ade480077eeae8c627b8e2372076cc52f0be4e69a3b063895653a9
+size 114450

backend/pydantic_models.py

@@ -0,0 +1,353 @@
+# pylint: disable=unused-import
+"""
+Pydantic models for API request/response validation.
+Maintains strict contract between React frontend and FastAPI backend.
+"""
+
+import time
+from typing import List, Dict, Any, Optional, Union, Literal
+from pydantic import BaseModel, Field, field_validator, model_validator
+import numpy as np
+
+
+class SpectrumData(BaseModel):
+    """Single spectrum data for analysis"""
+
+    x_values: List[float] = Field(..., description="Wavenumber values (cm⁻¹)")
+    y_values: List[float] = Field(..., description="Intensity values")
+    filename: Optional[str] = Field(None, description="Original filename")
+
+    @field_validator("x_values", "y_values")
+    @classmethod
+    def validate_arrays(cls, v: List[float]) -> List[float]:
+        """
+        Validate that the input arrays have at least 2 values.
+
+        Args:
+            v (list): The array to validate.
+
+        Returns:
+            list: The validated array.
+
+        Raises:
+            ValueError: If the array has fewer than 2 values.
+        """
+        if len(v) < 2:
+            raise ValueError("Arrays must have at least 2 values")
+        return v
+
+    @model_validator(mode="after")
+    def validate_equal_length(self) -> "SpectrumData":
+        """
+        Ensure that y_values has the same length as x_values.
+
+        Args:
+            v (list): The y_values list to validate.
+            values (dict): The dictionary containing other field values.
+
+        Returns:
+            list: The validated y_values list.
+
+        Raises:
+            ValueError: If y_values and x_values do not have the same length.
+        """
+        if len(self.x_values) != len(self.y_values):
+            raise ValueError("x_values and y_values must have equal length")
+        return self
+
+
+class AnalysisRequest(BaseModel):
+    """Request for single spectrum analysis"""
+
+    spectrum: SpectrumData
+    model_name: str = Field(..., description="Model name to use for analysis")
+    modality: Literal["raman", "ftir"] = Field(
+        "raman", description="Spectroscopy modality"
+    )
+    include_provenance: bool = Field(
+        True, description="Include full provenance metadata"
+    )
+
+
+class BatchAnalysisRequest(BaseModel):
+    """Request for batch spectrum analysis"""
+
+    spectra: List[SpectrumData] = Field(..., min_length=1, max_length=100)
+    model_name: str = Field(..., description="Model name to use for analysis")
+    modality: Literal["raman", "ftir"] = Field(
+        "raman", description="Spectroscopy modality"
+    )
+    include_provenance: bool = Field(
+        True, description="Include full provenance metadata"
+    )
+
+
+class ComparisonRequest(BaseModel):
+    """Request for multi-model comparison"""
+
+    spectrum: SpectrumData
+    model_names: Optional[List[str]] = Field(
+        None, description="Models to compare (all if None)"
+    )
+    modality: Literal["raman", "ftir"] = Field(
+        "raman", description="Spectroscopy modality"
+    )
+    include_provenance: bool = Field(
+        True, description="Include full provenance metadata"
+    )
+
+
+class PreprocessingMetadata(BaseModel):
+    """Preprocessing provenance metadata"""
+
+    target_length: int = Field(..., description="Target resampling length")
+    baseline_degree: int = Field(...,
+                                 description="Polynomial baseline removal degree")
+    smooth_window: int = Field(..., description="Smoothing window length")
+    smooth_polyorder: int = Field(...,
+                                  description="Smoothing polynomial order")
+    normalization_method: str = Field(...,
+                                      description="Normalization method applied")
+    modality_validated: bool = Field(
+        ..., description="Whether modality validation passed"
+    )
+    validation_issues: List[str] = Field(
+        default_factory=list, description="Any validation issues found"
+    )
+    original_length: int = Field(..., description="Original spectrum length")
+    wavenumber_range: List[float] = Field(
+        ..., min_length=2, max_length=2, description="[min, max] wavenumber range"
+    )
+
+
+class QualityControlMetadata(BaseModel):
+    """Quality control check results"""
+
+    signal_to_noise_ratio: Optional[float] = Field(
+        None, description="Estimated SNR")
+    baseline_stability: Optional[float] = Field(
+        None, description="Baseline stability metric"
+    )
+    spectral_resolution: Optional[float] = Field(
+        None, description="Estimated spectral resolution"
+    )
+    cosmic_ray_detected: bool = Field(
+        False, description="Cosmic ray spikes detected")
+    saturation_detected: bool = Field(
+        False, description="Signal saturation detected")
+    issues: List[str] = Field(default_factory=list,
+                              description="QC issues found")
+
+
+class ModelMetadata(BaseModel):
+    """Model metadata and calibration details"""
+
+    model_name: str = Field(..., description="Model identifier")
+    model_description: str = Field(..., description="Model description")
+    model_version: Optional[str] = Field(None, description="Model version")
+    training_date: Optional[str] = Field(
+        None, description="Model training date")
+    input_length: int = Field(..., description="Expected input length")
+    num_classes: int = Field(..., description="Number of output classes")
+    parameters_count: Optional[str] = Field(
+        None, description="Number of parameters")
+    performance_metrics: Dict[str, float] = Field(
+        default_factory=dict, description="Training performance"
+    )
+    supported_modalities: List[str] = Field(
+        default_factory=list, description="Supported spectroscopy modalities"
+    )
+    citation: Optional[str] = Field(
+        None, description="Model citation/reference")
+    weights_loaded: bool = Field(...,
+                                 description="Whether trained weights were loaded")
+    weights_path: Optional[str] = Field(
+        None, description="Path to loaded weights")
+
+
+class PredictionResult(BaseModel):
+    """Single prediction result with full provenance"""
+
+    prediction: int = Field(...,
+                            description="Predicted class (0=Stable, 1=Weathered)")
+    prediction_label: str = Field(...,
+                                  description="Human-readable prediction label")
+    confidence: float = Field(
+        ..., ge=0.0, le=1.0, description="Prediction confidence score"
+    )
+    probabilities: List[float] = Field(..., description="Class probabilities")
+    logits: List[float] = Field(..., description="Raw model logits")
+
+    # Provenance metadata
+    preprocessing: PreprocessingMetadata
+    quality_control: QualityControlMetadata
+    model_metadata: ModelMetadata
+
+    # Performance tracking
+    inference_time: float = Field(..., ge=0.0,
+                                  description="Inference time in seconds")
+    preprocessing_time: float = Field(
+        ..., ge=0.0, description="Preprocessing time in seconds"
+    )
+    total_time: float = Field(
+        ..., ge=0.0, description="Total processing time in seconds"
+    )
+    memory_usage_mb: float = Field(..., ge=0.0,
+                                   description="Memory usage in MB")
+
+    # Input data (for audit trail)
+    original_spectrum: SpectrumData
+    processed_spectrum: SpectrumData
+
+    # Timestamps
+    timestamp: str = Field(...,
+                           description="Processing timestamp (ISO format)")
+
+
+class BatchError(BaseModel):
+    """Details of a single error within a batch request"""
+
+    filename: Optional[str] = Field(
+        None, description="Filename of the spectrum that failed"
+    )
+    error: str = Field(..., description="The error message")
+
+
+class BatchPredictionResult(BaseModel):
+    """Batch prediction results"""
+
+    results: List[PredictionResult] = Field(
+        default_factory=list, description="Individual prediction results"
+    )
+    errors: List[BatchError] = Field(
+        default_factory=list,
+        description="Errors for spectra that failed processing",
+    )
+    summary: Dict[str, Any] = Field(
+        default_factory=dict, description="Batch summary statistics"
+    )
+    total_processing_time: float = Field(
+        ..., ge=0.0, description="Total batch processing time"
+    )
+    timestamp: str = Field(..., description="Batch processing timestamp")
+
+
+class ComparisonResult(BaseModel):
+    """Multi-model comparison results"""
+
+    spectrum_id: str = Field(...,
+                             description="Unique identifier for the spectrum")
+    model_results: Dict[str, PredictionResult] = Field(
+        default_factory=dict, description="Results per model"
+    )
+    consensus_prediction: Optional[int] = Field(
+        None, description="Consensus prediction if available"
+    )
+    confidence_variance: float = Field(
+        ..., ge=0.0, description="Variance in confidence scores"
+    )
+    agreement_score: float = Field(
+        ..., ge=0.0, le=1.0, description="Model agreement score"
+    )
+    timestamp: str = Field(..., description="Comparison timestamp")
+
+
+class FeatureImportanceSummary(BaseModel):
+    """Summary of feature importance scores"""
+    max_importance: float
+    mean_importance: float
+    important_region_start: int
+    important_region_end: int
+
+
+class TopFeatures(BaseModel):
+    """Top features identified by explainability analysis"""
+    indices: List[int]
+    values: List[float]
+
+
+class FeatureImportance(BaseModel):
+    """Feature importance results from explainability analysis"""
+    method: str
+    importance_scores: List[float]
+    top_features: TopFeatures
+    summary: FeatureImportanceSummary
+
+
+class ExplanationResult(BaseModel):
+    """Result from an explainability analysis"""
+    prediction: int
+    confidence: float
+    probabilities: List[float]
+    class_labels: List[str]
+    model_used: str
+    spectrum_filename: Optional[str] = None
+    feature_importance: Optional[FeatureImportance] = None
+
+    class Config:
+        """Pydantic model configuration"""
+        from_attributes = True
+
+
+class ModelInfo(BaseModel):
+    """Model information and capabilities"""
+
+    name: str = Field(..., description="Model identifier")
+    description: str = Field(..., description="Model description")
+    input_length: int = Field(..., description="Expected input length")
+    num_classes: int = Field(..., description="Number of output classes")
+    supported_modalities: List[str] = Field(
+        default_factory=list, description="Supported modalities"
+    )
+    performance: Dict[str, float] = Field(
+        default_factory=dict, description="Performance metrics"
+    )
+    parameters: Optional[str] = Field(None, description="Parameter count")
+    speed: Optional[str] = Field(None, description="Relative speed category")
+    citation: Optional[str] = Field(None, description="Citation/reference")
+    available: bool = Field(...,
+                            description="Whether model is available for inference")
+
+
+class SystemHealth(BaseModel):
+    """System health metrics"""
+    status: str = Field(..., description="Overall system status, e.g., 'ok'.")
+    timestamp: float = Field(...,
+                             description="The server timestamp of the health check.")
+    models_loaded: int
+    total_models: int
+    memory_usage_mb: float
+    torch_version: str
+    cuda_available: bool
+
+
+class SystemInfo(BaseModel):
+    """System information and health"""
+
+    version: str = Field(..., description="API version")
+    available_models: List[ModelInfo] = Field(
+        default_factory=list, description="Available models"
+    )
+    supported_modalities: List[str] = Field(
+        default_factory=list, description="Supported spectroscopy modalities"
+    )
+    max_batch_size: int = Field(100, ge=1, description="Maximum batch size")
+    target_spectrum_length: int = Field(
+        500, ge=1, description="Target spectrum length")
+    system_health: SystemHealth = Field(
+        ..., description="System health metrics"
+    )
+
+
+class ErrorResponse(BaseModel):
+    """Standardized error response"""
+
+    error: str = Field(..., description="Error message")
+    error_code: str = Field(...,
+                            description="Error code for programmatic handling")
+    details: Optional[Dict[str, Any]] = Field(
+        None, description="Additional error details"
+    )
+    timestamp: str = Field(..., description="Error timestamp")
+    request_id: Optional[str] = Field(
+        None, description="Request ID for tracking")

backend/registry.py

@@ -0,0 +1,237 @@
+# models/registry.py
+from typing import Callable, Dict, List, Any
+from models.figure2_cnn import Figure2CNN
+from models.resnet_cnn import ResNet1D
+from models.resnet18_vision import ResNet18Vision
+from models.enhanced_cnn import EnhancedCNN, EfficientSpectralCNN, HybridSpectralNet
+
+# Internal registry of model builders keyed by short name.
+_REGISTRY: Dict[str, Callable[[int], object]] = {
+    "figure2": lambda L: Figure2CNN(input_length=L),
+    "resnet": lambda L: ResNet1D(input_length=L),
+    "resnet18vision": lambda L: ResNet18Vision(input_length=L),
+    "enhanced_cnn": lambda L: EnhancedCNN(input_length=L),
+    "efficient_cnn": lambda L: EfficientSpectralCNN(input_length=L),
+    "hybrid_net": lambda L: HybridSpectralNet(input_length=L),
+}
+
+# Model specifications with metadata for enhanced features
+_MODEL_SPECS: Dict[str, Dict[str, Any]] = {
+    "figure2": {
+        "input_length": 500,
+        "num_classes": 2,
+        "description": "Figure 2 baseline custom implementation",
+        "modalities": ["raman", "ftir"],
+        "citation": "Neo et al., 2023, Resour. Conserv. Recycl., 188, 106718",
+        "performance": {"accuracy": 0.948, "f1_score": 0.943},
+        "parameters": "~500K",
+        "speed": "fast",
+    },
+    "resnet": {
+        "input_length": 500,
+        "num_classes": 2,
+        "description": "(Residual Network) uses skip connections to train much deeper networks",
+        "modalities": ["raman", "ftir"],
+        "citation": "Custom ResNet implementation",
+        "performance": {"accuracy": 0.962, "f1_score": 0.959},
+        "parameters": "~100K",
+        "speed": "very_fast",
+    },
+    "resnet18vision": {
+        "input_length": 500,
+        "num_classes": 2,
+        "description": "excels at image recognition tasks by using 'residual blocks' to train more efficiently",
+        "modalities": ["raman", "ftir"],
+        "citation": "ResNet18 Vision adaptation",
+        "performance": {"accuracy": 0.945, "f1_score": 0.940},
+        "parameters": "~11M",
+        "speed": "medium",
+    },
+    "enhanced_cnn": {
+        "input_length": 500,
+        "num_classes": 2,
+        "description": "Enhanced CNN with attention mechanisms and multi-scale feature extraction",
+        "modalities": ["raman", "ftir"],
+        "citation": "Custom enhanced architecture with attention",
+        "performance": {"accuracy": 0.975, "f1_score": 0.973},
+        "parameters": "~800K",
+        "speed": "medium",
+        "features": ["attention", "multi_scale", "batch_norm", "dropout"],
+    },
+    "efficient_cnn": {
+        "input_length": 500,
+        "num_classes": 2,
+        "description": "Efficient CNN optimized for real-time inference with depthwise separable convolutions",
+        "modalities": ["raman", "ftir"],
+        "citation": "Custom efficient architecture",
+        "performance": {"accuracy": 0.955, "f1_score": 0.952},
+        "parameters": "~200K",
+        "speed": "very_fast",
+        "features": ["depthwise_separable", "lightweight", "real_time"],
+    },
+    "hybrid_net": {
+        "input_length": 500,
+        "num_classes": 2,
+        "description": "Hybrid network combining CNN backbone with self-attention mechanisms",
+        "modalities": ["raman", "ftir"],
+        "citation": "Custom hybrid CNN-Transformer architecture",
+        "performance": {"accuracy": 0.968, "f1_score": 0.965},
+        "parameters": "~1.2M",
+        "speed": "medium",
+        "features": ["self_attention", "cnn_backbone", "transformer_head"],
+    },
+}
+
+# Placeholder for future model expansions
+_FUTURE_MODELS = {
+    "densenet1d": {
+        "description": "DenseNet1D for spectroscopy with dense connections",
+        "status": "planned",
+        "modalities": ["raman", "ftir"],
+        "features": ["dense_connections", "parameter_efficient"],
+    },
+    "ensemble_cnn": {
+        "description": "Ensemble of multiple CNN variants for robust predictions",
+        "status": "planned",
+        "modalities": ["raman", "ftir"],
+        "features": ["ensemble", "robust", "high_accuracy"],
+    },
+    "vision_transformer": {
+        "description": "Vision Transformer adapted for 1D spectral data",
+        "status": "planned",
+        "modalities": ["raman", "ftir"],
+        "features": ["transformer", "attention", "state_of_art"],
+    },
+    "autoencoder_cnn": {
+        "description": "CNN with autoencoder for unsupervised feature learning",
+        "status": "planned",
+        "modalities": ["raman", "ftir"],
+        "features": ["autoencoder", "unsupervised", "feature_learning"],
+    },
+}
+
+
+def choices():
+    """Return the list of available model keys."""
+    return list(_REGISTRY.keys())
+
+
+def planned_models():
+    """Return the list of planned future model keys."""
+    return list(_FUTURE_MODELS.keys())
+
+
+def build(name: str, input_length: int):
+    """Instantiate a model by short name with the given input length."""
+    if name not in _REGISTRY:
+        raise ValueError(f"Unknown model '{name}'. Choices: {choices()}")
+    return _REGISTRY[name](input_length)
+
+
+def build_multiple(names: List[str], input_length: int) -> Dict[str, Any]:
+    """Nuild multiple models for comparison."""
+    models = {}
+    for name in names:
+        if name in _REGISTRY:
+            models[name] = build(name, input_length)
+        else:
+            raise ValueError(f"Unknown model '{name}'. Available: {choices()}")
+    return models
+
+
+def register_model(
+    name: str, builder: Callable[[int], object], spec: Dict[str, Any]
+) -> None:
+    """Dynamically register a new model."""
+    if name in _REGISTRY:
+        raise ValueError(f"Model '{name}' already registered.")
+    if not callable(builder):
+        raise TypeError("Builder must be a callable that accepts an integer argument.")
+    _REGISTRY[name] = builder
+    _MODEL_SPECS[name] = spec
+
+
+def spec(name: str):
+    """Return expected input length and number of classes for a model key."""
+    if name in _MODEL_SPECS:
+        return _MODEL_SPECS[name].copy()
+    raise KeyError(f"Unknown model '{name}'. Available: {choices()}")
+
+
+def get_model_info(name: str) -> Dict[str, Any]:
+    """Get comprehensive model information including metadata."""
+    if name in _MODEL_SPECS:
+        return _MODEL_SPECS[name].copy()
+    elif name in _FUTURE_MODELS:
+        return _FUTURE_MODELS[name].copy()
+    else:
+        raise KeyError(f"Unknown model '{name}'")
+
+
+def models_for_modality(modality: str) -> List[str]:
+    """Get list of models that support a specific modality."""
+    compatible = []
+    for name, spec_info in _MODEL_SPECS.items():
+        if modality in spec_info.get("modalities", []):
+            compatible.append(name)
+    return compatible
+
+
+def validate_model_list(names: List[str]) -> List[str]:
+    """Validate and return list of available models from input list."""
+    available = choices()
+    valid_models = []
+    for name in names:
+        if name in available:  # Fixed: was using 'is' instead of 'in'
+            valid_models.append(name)
+    return valid_models
+
+
+def get_models_metadata() -> Dict[str, Dict[str, Any]]:
+    """Get metadata for all registered models."""
+    return {name: _MODEL_SPECS[name].copy() for name in _MODEL_SPECS}
+
+
+def is_model_compatible(name: str, modality: str) -> bool:
+    """Check if a model is compatible with a specific modality."""
+    if name not in _MODEL_SPECS:
+        return False
+    return modality in _MODEL_SPECS[name].get("modalities", [])
+
+
+def get_model_capabilities(name: str) -> Dict[str, Any]:
+    """Get detailed capabilities of a model."""
+    if name not in _MODEL_SPECS:
+        raise KeyError(f"Unknown model '{name}'")
+
+    spec = _MODEL_SPECS[name].copy()
+    spec.update(
+        {
+            "available": True,
+            "status": "active",
+            "supported_tasks": ["binary_classification"],
+            "performance_metrics": {
+                "supports_confidence": True,
+                "supports_batch": True,
+                "memory_efficient": spec.get("description", "").lower().find("resnet")
+                != -1,
+            },
+        }
+    )
+    return spec
+
+
+__all__ = [
+    "choices",
+    "build",
+    "spec",
+    "build_multiple",
+    "register_model",
+    "get_model_info",
+    "models_for_modality",
+    "validate_model_list",
+    "planned_models",
+    "get_models_metadata",
+    "is_model_compatible",
+    "get_model_capabilities",
+]

backend/service.py

@@ -0,0 +1,331 @@
+# pylint:  wrong-import-order, unused-import, import-outside-toplevel
+"""
+Backend service layer for ML inference.
+Extracts and preserves the current Streamlit application logic for FastAPI.
+Maintains scientific fidelity and deterministic outputs.
+"""
+import time
+import gc
+from typing import Tuple, List
+from pathlib import Path
+from datetime import datetime
+import psutil
+import torch
+import torch.nn.functional as F
+import numpy as np
+
+from backend.utils.preprocessing import (
+    # We will replace these with SpectrumPreprocessor
+    # remove_baseline, smooth_spectrum, normalize_spectrum,
+    validate_spectrum_modality,
+    MODALITY_PARAMS
+)
+from backend.models.registry import get_model_info as get_registry_model_info
+from backend.utils.performance  import log_model_performance
+from .config import TARGET_LEN, LABEL_MAP
+
+from .pydantic_models import (
+    SpectrumData,
+    PredictionResult,
+    PreprocessingMetadata,
+    QualityControlMetadata,
+    ModelMetadata,
+    ModelInfo,
+    SystemInfo,
+    SystemHealth
+)
+from backend.utils.model_manager import model_manager
+from backend.utils.preprocessing_fixed import SpectrumPreprocessor
+
+
+class MLServiceError(Exception):
+    """Custom exception for ML service errors."""
+
+    pass
+
+
+class MLInferenceService:
+    """
+    Core ML inference service that preserves the exact behavior of the Streamlit app.
+    Maintains scientific fidelity and deterministic outputs.
+    """
+
+    def __init__(self, model_manager_instance=model_manager):
+        self.model_manager = model_manager_instance
+        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+
+    def get_memory_usage(self) -> float:
+        """Get current memory usage in MB"""
+        try:
+            process = psutil.Process()
+            return process.memory_info().rss / 1024 / 1024
+        except ImportError:
+            return 0.0
+
+    def cleanup_memory(self):
+        """Clean up memory after inference"""
+        gc.collect()
+        if torch.cuda.is_available():
+            torch.cuda.empty_cache()
+
+    def create_preprocessing_metadata(
+        self,
+        modality: str,
+        original_length: int,
+        x_data: np.ndarray,
+        validation_result: Tuple[bool, List[str]]
+    ) -> PreprocessingMetadata:
+        """Create preprocessing provenance metadata"""
+        params = MODALITY_PARAMS.get(modality, MODALITY_PARAMS["raman"])
+        is_valid, issues = validation_result
+
+        return PreprocessingMetadata(
+            target_length=TARGET_LEN,
+            baseline_degree=params["baseline_degree"],
+            smooth_window=params["smooth_window"],
+            smooth_polyorder=params["smooth_polyorder"],
+            normalization_method="min_max",
+            modality_validated=is_valid,
+            validation_issues=issues,
+            original_length=original_length,
+            wavenumber_range=[float(np.min(x_data)), float(np.max(x_data))]
+        )
+
+    def create_quality_control_metadata(
+        self,
+        y_data: np.ndarray,
+        y_processed: np.ndarray
+    ) -> QualityControlMetadata:
+        """Create quality control metadata with basic checks"""
+        issues = []
+
+        # Basic signal quality checks
+        signal_range = np.max(y_data) - np.min(y_data)
+        noise_estimate = np.std(np.diff(y_data))
+        snr = signal_range / noise_estimate if noise_estimate > 0 else None
+
+        # Check for saturation (values at extremes)
+        if np.any(y_data >= 0.99 * np.max(y_data)):
+            issues.append("Potential signal saturation detected")
+
+        # Check for cosmic rays (sudden spikes)
+        diff = np.abs(np.diff(y_data))
+        if len(diff) > 0:
+            threshold = np.mean(diff) + 5 * np.std(diff)
+            cosmic_ray_detected = np.any(diff > threshold)
+            if cosmic_ray_detected:
+                issues.append("Potential cosmic ray spikes detected")
+        else:
+            cosmic_ray_detected = False
+
+        # Baseline stability
+        baseline_stability = 0.0
+        if len(y_processed) >= 100:
+            baseline_stability = 1.0 - \
+                (np.std(y_processed[:50]) + np.std(y_processed[-50:])) / 2
+            baseline_stability = max(0.0, min(1.0, float(baseline_stability)))
+
+        return QualityControlMetadata(
+            signal_to_noise_ratio=snr,
+            baseline_stability=baseline_stability if baseline_stability > 0 else None,
+            spectral_resolution=None,
+            cosmic_ray_detected=bool(cosmic_ray_detected),
+            saturation_detected=any("saturation" in issue.lower()
+                                    for issue in issues),
+            issues=issues
+        )
+
+    def create_model_metadata(
+        self,
+        model_name: str,
+        weights_loaded: bool,
+        weights_path: Path
+    ) -> ModelMetadata:
+        """Create model metadata with calibration details"""
+        info = get_registry_model_info(model_name)
+
+        return ModelMetadata(
+            model_name=model_name,
+            model_description=info.get("description", ""),
+            model_version=None,
+            training_date=None,
+            input_length=info.get("input_length", TARGET_LEN),
+            num_classes=info.get("num_classes", 2),
+            parameters_count=info.get("parameters", "Unknown"),
+            performance_metrics=info.get("performance", {}),
+            supported_modalities=info.get("modalities", ["raman", "ftir"]),
+            citation=info.get("citation", ""),
+            weights_loaded=weights_loaded, # This comes from model_manager
+            weights_path=str(weights_path) if weights_loaded else None
+        )
+
+    def run_inference(
+        self,
+        spectrum: SpectrumData,
+        model_name: str,
+        modality: str,
+        include_provenance: bool = True
+    ) -> PredictionResult:
+        """
+        Run model inference preserving exact Streamlit behavior.
+        Returns complete result with full provenance metadata.
+        """
+        start_total = time.time()
+        start_memory = self.get_memory_usage()
+
+        # Convert input data
+        x_data = np.array(spectrum.x_values)
+        y_data = np.array(spectrum.y_values)
+        original_length = len(y_data)
+
+        if original_length < 2:
+            raise MLServiceError("Spectrum must have at least 2 data points")
+
+        # Validate modality
+        validation_result = validate_spectrum_modality(x_data, y_data, modality)
+
+        # Preprocessing
+        start_preprocess = time.time()
+        # Use SpectrumPreprocessor for consistent preprocessing
+        preprocessor = SpectrumPreprocessor(
+            target_len=TARGET_LEN,
+            do_baseline=True, # Assuming these are desired for standard analysis
+            do_smooth=True,
+            do_normalize=True,
+            modality=modality
+        )
+        y_processed = preprocessor.preprocess_single_spectrum(x_data, y_data, use_fitted_stats=False)
+        # For x_resampled, we can just generate it based on target_len and original range
+        x_resampled = np.linspace(np.min(x_data), np.max(x_data), TARGET_LEN)
+
+        preprocessing_time = time.time() - start_preprocess
+
+        # Load model
+        model, weights_loaded, weights_path = self.model_manager.load_model(model_name)
+        if model is None:
+            raise MLServiceError(f"Model '{model_name}' not available")
+
+        # Ensure model is on the correct device before inference
+        model.to(self.device)
+
+        # Create input tensor
+        input_tensor = torch.tensor(y_processed, dtype=torch.float32).unsqueeze(0).unsqueeze(0)
+        input_tensor = input_tensor.to(self.device) # Move to device
+        # Inference
+        start_inference = time.time()
+        model.eval()
+        with torch.no_grad():
+            logits = model(input_tensor)
+            prediction = torch.argmax(logits, dim=1).item()
+            logits_list = logits.detach().numpy().tolist()[0]
+            probs = F.softmax(logits.detach(), dim=1).cpu().numpy().flatten()
+
+        inference_time = time.time() - start_inference
+        total_time = time.time() - start_total
+        end_memory = self.get_memory_usage()
+        memory_usage = max(end_memory - start_memory, 0)
+
+        # Log performance metrics for benchmarking
+        log_model_performance(
+            model_name=model_name,
+            inference_time=inference_time,
+            preprocessing_time=preprocessing_time,
+            total_time=total_time,
+            memory_usage=memory_usage,
+            spectrum_length=original_length
+        )
+
+        # Calculate confidence
+        confidence = float(max(probs)) if probs is not None and len(
+            probs) > 0 else 0.0
+
+        # Create metadata
+        if include_provenance:
+            preprocessing_metadata = self.create_preprocessing_metadata(
+                modality, original_length, x_data, validation_result
+            )
+            qc_metadata = self.create_quality_control_metadata(
+                y_data, y_processed)
+            model_metadata = self.create_model_metadata(
+                model_name, weights_loaded, weights_path)
+        else:
+            preprocessing_metadata = PreprocessingMetadata(
+                target_length=TARGET_LEN,
+                baseline_degree=2,
+                smooth_window=11,
+                smooth_polyorder=2,
+                normalization_method="min_max",
+                modality_validated=validation_result[0],
+                validation_issues=validation_result[1],
+                original_length=original_length,
+                wavenumber_range=[float(np.min(x_data)), float(np.max(x_data))]
+            )
+            qc_metadata = QualityControlMetadata(
+                signal_to_noise_ratio=None,
+                baseline_stability=None,
+                spectral_resolution=None,
+                cosmic_ray_detected=False,
+                saturation_detected=False,
+                issues=[]
+            )
+            model_metadata = self.create_model_metadata(
+                model_name, weights_loaded, weights_path) # Still need model metadata
+
+        # Create processed spectrum data
+        processed_spectrum = SpectrumData(
+            x_values=x_resampled.tolist(),
+            y_values=y_processed.tolist(),
+            filename=f"processed_{spectrum.filename}" if spectrum.filename else None
+        )
+
+        # Clean up memory
+        self.cleanup_memory()
+
+        return PredictionResult(
+            prediction=prediction,
+            prediction_label=LABEL_MAP[prediction] if prediction in LABEL_MAP else "Unknown",
+            confidence=confidence,
+            probabilities=probs.tolist(),
+            logits=logits_list,
+            preprocessing=preprocessing_metadata,
+            quality_control=qc_metadata,
+            model_metadata=model_metadata,
+            inference_time=inference_time,
+            preprocessing_time=preprocessing_time,
+            total_time=total_time,
+            memory_usage_mb=memory_usage,
+            original_spectrum=spectrum,
+            processed_spectrum=processed_spectrum,
+            timestamp=datetime.now().isoformat()
+        )
+
+    def get_available_models(self) -> List[ModelInfo]:
+        """Get list of available models with their information"""
+        return self.model_manager.get_available_models()
+
+    def get_system_info(self) -> SystemInfo:
+        """Get system information and health status"""
+        models = self.model_manager.get_available_models()
+
+        system_health_data = SystemHealth(
+            status="ok",
+            timestamp=time.time(),
+            models_loaded=sum(1 for m in models if m.available),
+            total_models=len(models),
+            memory_usage_mb=self.get_memory_usage(),
+            torch_version=torch.__version__,
+            cuda_available=torch.cuda.is_available()
+        )
+
+        return SystemInfo(
+            version="1.0.0",
+            available_models=models,
+            supported_modalities=["raman", "ftir"],
+            max_batch_size=100,
+            target_spectrum_length=TARGET_LEN,
+            system_health=system_health_data
+        )
+
+
+# Global service instance
+ml_service = MLInferenceService()

backend/tests/test_api.py

@@ -0,0 +1,17 @@
+from fastapi.testclient import TestClient
+from backend.main import app
+
+client = TestClient(app)
+
+def test_analyze_spectrum():
+    payload = {
+        "spectrum": {
+            "x_values": [200, 210],           # At least 2 points
+            "y_values": [0.5, 0.6],           # At least 2 points
+            "filename": "test.txt"
+        },
+        "modality": "raman",
+        "model_name": "figure2"
+    }
+    response = client.post("/api/v1/analyze", json=payload)
+    assert response.status_code == 200

backend/tests/test_service.py

@@ -0,0 +1,54 @@
+"""
+Unit tests for backend.service.run_inference and /api/v1/analyze endpoint.
+"""
+
+import unittest
+from unittest.mock import patch
+from backend.service import ml_service
+from backend.pydantic_models import SpectrumData  # Adjust import if needed
+from fastapi.testclient import TestClient
+from backend.main import app
+
+class TestService(unittest.TestCase):
+    """Tests for ml_service.run_inference."""
+
+    @patch('backend.service.log_model_performance')
+    def test_run_inference_calls_log_model_performance(self, mock_log_model_performance):
+        """Test that run_inference calls log_model_performance with valid input."""
+        # Build a real SpectrumData instance with required fields only
+        dummy_spectrum = SpectrumData(
+            x_values=[200, 210],
+            y_values=[0.5, 0.6],
+            filename="dummy.txt"
+        )
+        model_name = "figure2"
+        modality = "raman"
+
+        # Call with separate model_name and modality args (not as SpectrumData attributes)
+        ml_service.run_inference(dummy_spectrum, model_name, modality)
+
+        mock_log_model_performance.assert_called_once()
+
+class TestAPI(unittest.TestCase):
+    """Tests for /api/v1/analyze endpoint."""
+
+    def setUp(self):
+        self.client = TestClient(app)
+
+    def test_analyze_spectrum_valid_payload(self):
+        """Test /api/v1/analyze with valid payload."""
+        payload = {
+            "spectrum": {
+                "x_values": [200, 210],
+                "y_values": [0.5, 0.6],
+                "filename": "dummy.txt"
+            },
+            "modality": "raman",
+            "model_name": "figure2"
+        }
+        response = self.client.post("/api/v1/analyze", json=payload)
+        assert response.status_code == 200
+        # Optionally, check response.json() for expected keys
+
+if __name__ == "__main__":
+    unittest.main()

backend/utils/confidence.py

@@ -0,0 +1,97 @@
+"""Confidence calculation and visualization utilities.
+Provides normalized softmax confidence and color-coded badges"""
+
+from typing import Tuple, List
+import numpy as np
+import torch
+import torch.nn.functional as F
+
+
+def calculate_softmax_confidence(
+    logits: torch.Tensor,
+) -> Tuple[np.ndarray, float, str, str]:
+    """Calculate normalized confidence using softmax
+    Args:
+        logits: Raw model logits tensor
+    Returns:
+        Tuple of (probabilities, max_confidence, confidence_level, confidence_emoji)
+    """
+    # ===Apply softmax to get probabilities===
+    probs_np = F.softmax(logits, dim=1).cpu().numpy().flatten()
+
+    # ===Get maximum probability as confidence===
+    max_confidence = float(np.max(probs_np))
+
+    # ===Determine confidence level and emoji===
+    if max_confidence >= 0.80:
+        confidence_level = "HIGH"
+        confidence_emoji = "🟢"
+    elif max_confidence >= 0.60:
+        confidence_level = "MEDIUM"
+        confidence_emoji = "🟡"
+    else:
+        confidence_level = "LOW"
+        confidence_emoji = "🔴"
+
+    return probs_np, max_confidence, confidence_level, confidence_emoji
+
+
+def get_confidence_badge(confidence: float) -> Tuple[str, str]:
+    """Get confidence badge emoji and level description
+    Args:
+        confidence: Confidence value (0-1)
+    Returns:
+        Tuple of (emoji, level)
+    """
+    if confidence >= 0.80:
+        return "🟢", "HIGH"
+    elif confidence >= 0.60:
+        return "🟡", "MEDIUM"
+    else:
+        return "🔴", "LOW"
+
+
+def format_confidence_display(confidence: float, level: str, emoji: str) -> str:
+    """
+    Format confidence for display in UI
+
+    Args:
+        confidence: Confidence value (0-1)
+        level: Confidence level (HIGH/MEDIUM/LOW)
+        emoji: Confidence emoji
+
+    Returns:
+        Formatted confidence string
+    """
+    return f"{emoji} **{level}** ({confidence:.1%})"
+
+
+def calculate_legacy_confidence(logits_list: List[float]) -> Tuple[float, str, str]:
+    """
+    Calculate confidence using legacy logit margin method for backward compatibility
+
+    Args:
+        logits_list: List of raw logits
+
+    Returns:
+        Tuple of (margin, confidence_level, confidence_emoji)
+    """
+    if len(logits_list) < 2:
+        return 0.0, "LOW", "🔴"
+
+    logits_array = np.array(logits_list)
+    sorted_logits = np.sort(logits_array)[::-1]  # Descending order
+    margin = sorted_logits[0] - sorted_logits[1]
+
+    # ===Define thresholds for margin-based confidence===
+    if margin >= 2.0:
+        confidence_level = "HIGH"
+        confidence_emoji = "🟢"
+    elif margin >= 1.0:
+        confidence_level = "MEDIUM"
+        confidence_emoji = "🟡"
+    else:
+        confidence_level = "LOW"
+        confidence_emoji = "🔴"
+
+    return margin, confidence_level, confidence_emoji

backend/utils/enhanced_ml_service.py

@@ -0,0 +1,317 @@
+# pylint:  disable=wrong-import-order, unused-import
+"""
+Enhanced API endpoints with explainability features.
+Extends the existing FastAPI backend with SHAP-based model explanations
+and improved prediction capabilities.
+"""
+from backend.config import TARGET_LEN  # Import TARGET_LEN for model loading
+import numpy as np
+import torch
+from typing import Dict, Any, List, Optional
+from fastapi import HTTPException  # Keep HTTPException for API errors
+# PredictionResult is not directly returned by this service
+from backend.pydantic_models import SpectrumData
+from backend.models.registry import build as build_model, choices, registry_spec
+from backend.utils.preprocessing_fixed import SpectrumPreprocessor
+
+import os
+# Import moved here to the toplevel
+from backend.utils.model_manager import model_manager
+
+
+class EnhancedMLService:
+    """
+    Enhanced ML service with explainability features.
+    Provides predictions with feature importance and model confidence.
+    """
+
+    def __init__(self):
+        self.model_manager = model_manager
+        # Local cache for loaded models (model, preprocessor)
+        self._model_cache = {}
+        self.device = torch.device(
+            "cuda" if torch.cuda.is_available() else "cpu")
+        print(f"✅ Enhanced ML Service initialized on {self.device}")
+
+
+    def cache_model(self, model_name: str, model_instance, preprocessor):
+        """Public method to cache a model and its preprocessor."""
+        self._model_cache[model_name] = {
+            'model': model_instance,
+            'preprocessor': preprocessor
+        }
+
+    def predict_with_explanation(
+        self,
+        spectrum_data: SpectrumData,
+        model_name: str,
+        modality: str = "raman",
+        include_feature_importance: bool = True
+    ) -> Dict[str, Any]:
+        """
+        Make prediction with explainability features.
+
+        Args:
+            spectrum_data (SpectrumData): Input spectrum data
+            model_name (str): Name of model to use
+            modality (str): The spectroscopy modality ('raman' or 'ftir')
+            include_feature_importance (bool): Whether to compute feature importance
+
+        Returns:
+            dict: Prediction results with explanations
+        """
+        if model_name not in self._model_cache:
+            # Attempt to load model via centralized manager if not in local cache
+            model_instance, weights_loaded, _ = self.model_manager.load_model(
+                model_name)
+            if model_instance is None or not weights_loaded:
+                raise HTTPException(
+                    status_code=400,
+                    detail=f"Model {model_name} not loaded or weights not found"
+                )
+
+            # Determine model input length robustly: prefer model attribute,
+            # fallback to registry/spec, then TARGET_LEN
+            input_len = getattr(model_instance, 'input_length', None)
+            if input_len is None:
+                try:
+                    spec_info = registry_spec(model_name)
+                    input_len = int(spec_info.get("input_length", TARGET_LEN))
+                except Exception:
+                    input_len = TARGET_LEN
+
+            # Create preprocessor for this model (use resolved input_len)
+            preprocessor = SpectrumPreprocessor(
+                target_len=input_len,
+                do_baseline=True,
+                do_smooth=True,
+                do_normalize=True,
+                modality=modality  # Use the provided modality
+            )
+            self._model_cache[model_name] = {
+                'model': model_instance, 'preprocessor': preprocessor}
+
+        model_entry = self._model_cache.get(model_name)
+        if not model_entry:  # Should not happen if previous block executed
+            raise HTTPException(
+                status_code=400,
+                detail=f"Model {model_name} not loaded"
+            )
+        model = model_entry['model']
+
+        # --- FIX: Ensure preprocessor has the correct modality ---
+        # The preprocessor might have been cached with a default or different modality.
+        # We must ensure it matches the one from the current request.
+        if model_entry['preprocessor'].modality != modality:
+            print(
+                f"🔄 Updating preprocessor modality for '{model_name}' from '{model_entry['preprocessor'].modality}' to '{modality}'")
+            model_entry['preprocessor'] = SpectrumPreprocessor(
+                target_len=model.input_length,
+                do_baseline=True, do_smooth=True, do_normalize=True,
+                modality=modality
+            )
+
+        preprocessor = model_entry['preprocessor']
+
+        try:
+            # Preprocess input data
+            x_data = np.array(spectrum_data.x_values)
+            y_data = np.array(spectrum_data.y_values)
+
+            # Preprocess spectrum
+            processed_spectrum = preprocessor.preprocess_single_spectrum(
+                x_data, y_data, use_fitted_stats=False
+            )
+
+            # Convert to tensor
+            input_tensor = torch.tensor(
+                processed_spectrum, dtype=torch.float32)
+            # Add batch and channel dimensions
+            input_tensor = input_tensor.unsqueeze(0)
+            input_tensor = input_tensor.unsqueeze(0)
+            input_tensor = input_tensor.to(self.device)
+
+            # Make prediction
+            with torch.no_grad():
+                outputs = model(input_tensor)
+                probabilities = torch.softmax(outputs, dim=1)
+                predicted_class = torch.argmax(probabilities, dim=1).item()
+                confidence = torch.max(probabilities).item()
+
+            # Basic prediction result
+            result = {
+                'prediction': predicted_class,
+                'confidence': confidence,
+                'probabilities': probabilities.cpu().numpy().tolist()[0],
+                'class_labels': ['stable', 'weathered'],
+                'model_used': model_name,
+                'spectrum_filename': spectrum_data.filename
+            }
+
+            # Add feature importance if requested
+            if include_feature_importance:
+                feature_importance = self._compute_feature_importance(
+                    model, input_tensor, processed_spectrum
+                )
+                result['feature_importance'] = feature_importance
+
+            return result
+
+        except (RuntimeError, ValueError, TypeError) as e:
+            raise HTTPException(
+                status_code=500,
+                detail=f"Prediction failed: {str(e)}"
+            ) from e
+
+    def _compute_feature_importance(
+        self,
+        model: torch.nn.Module,
+        input_tensor: torch.Tensor,
+        processed_spectrum: np.ndarray
+    ) -> Dict[str, Any]:
+        """
+        Compute feature importance using gradient-based methods.
+
+        Args:
+            model: PyTorch model
+            input_tensor: Preprocessed input tensor
+            processed_spectrum: Original processed spectrum
+
+        Returns:
+            dict: Feature importance information
+        """
+        try:
+            # Enable gradient computation
+            input_tensor.requires_grad_(True)
+            torch.set_grad_enabled(True)
+
+            # Forward pass
+            output = model(input_tensor)
+            predicted_class = torch.argmax(output, dim=1).item()
+
+            # Compute gradients with respect to input
+            class_score = output[0, predicted_class]
+            class_score.backward()
+
+            if input_tensor.grad is not None:
+                gradients = input_tensor.grad.data.cpu().numpy().squeeze()
+            else:
+                raise RuntimeError(
+                    "Gradients were not computed. Ensure requires_grad is set "
+                    "and gradient computation is enabled."
+                )
+            gradients = input_tensor.grad.data.cpu().numpy().squeeze()
+
+            # Compute importance metrics
+            importance_abs = np.abs(gradients)
+
+            # Find most important regions
+            top_indices = np.argsort(importance_abs)[-20:]  # Top 20 features
+
+            # Create interpretable output
+            feature_importance = {
+                'method': 'gradient_saliency',
+                'importance_scores': importance_abs.tolist(),
+                'top_features': {
+                    'indices': top_indices.tolist(),
+                    'values': importance_abs[top_indices].tolist()
+                },
+                'summary': {
+                    'max_importance': float(np.max(importance_abs)),
+                    'mean_importance': float(np.mean(importance_abs)),
+                    'important_region_start': int(top_indices[0]),
+                    'important_region_end': int(top_indices[-1])
+                }
+            }
+
+            return feature_importance
+
+        except (RuntimeError, ValueError, TypeError) as e:
+            print(f"⚠️ Feature importance computation failed: {e}")
+            return {
+                'method': 'gradient_saliency',
+                'error': str(e),
+                'importance_scores': [0.0] * len(processed_spectrum)
+            }
+
+    def get_model_info(self) -> List[Dict[str, Any]]:
+        """
+        Get information about loaded models.
+
+        Returns:
+            list: List of ModelInfo objects from the centralized manager.
+        """
+        return self.model_manager.get_available_models()
+
+    def batch_predict_with_explanation(
+        self,
+        spectra: List[SpectrumData],
+        model_name: str,
+        modality: str,  # Add modality for preprocessor
+        include_feature_importance: bool = True
+    ) -> List[Dict[str, Any]]:
+        """
+        Batch prediction with explanations.
+
+        Args:
+            spectra (list): List of spectrum data
+            model_name (str): Model to use
+            modality (str): Spectroscopy modality
+            include_feature_importance (bool): Whether to include explanations
+
+        Returns:
+            list: List of prediction results
+        """
+        results = []
+
+        for spectrum in spectra:
+            try:
+                result = self.predict_with_explanation(
+                    spectrum,
+                    model_name,
+                    modality=modality,  # Pass modality down
+                    include_feature_importance=include_feature_importance
+                )
+                results.append(result)
+            except (HTTPException, ValueError, RuntimeError) as e:
+                results.append({
+                    'error': str(e),
+                    'spectrum_filename': spectrum.filename
+                })
+
+        return results
+
+
+# Global enhanced service instance
+enhanced_ml_service = EnhancedMLService()
+
+
+def initialize_enhanced_service():
+    """Initialize the enhanced ML service with available models."""
+    print("Initializing Enhanced ML Service models...")
+    # Iterate through all known models in the registry by calling choices() directly
+    for model_name in choices():
+        try:
+            # Attempt to load each model via the centralized manager
+            model_instance, weights_loaded, _ = enhanced_ml_service.model_manager.load_model(
+                model_name, TARGET_LEN)
+            if model_instance and weights_loaded:
+                preprocessor = SpectrumPreprocessor(
+                    target_len=TARGET_LEN,
+                    do_baseline=True,
+                    do_smooth=True,
+                    do_normalize=True,
+                    modality="raman"
+                )
+                enhanced_ml_service.cache_model(model_name, model_instance, preprocessor)
+                print(f"✅ Enhanced ML Service: Prepared model '{model_name}' with preprocessor.")
+            else:
+                print(
+                    f"⚠️ Enhanced ML Service: Model '{model_name}' not fully loaded or weights missing.")
+        except (RuntimeError, ValueError, ImportError) as e:
+            print(
+                f"❌ Enhanced ML Service: Error initializing model '{model_name}': {e}")
+
+
+# Initialize on import
+initialize_enhanced_service()

backend/utils/errors.py

@@ -0,0 +1,36 @@
+"""Centralized error handling utility for the backend API.
+Provides consistent error logging without UI dependencies"""
+
+import traceback
+import logging
+
+# Configure logging
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
+
+class ErrorHandler:
+    """Centralized error handler for backend operations"""
+
+    @staticmethod
+    def log_error(error: Exception, context: str = "", include_traceback: bool = False) -> None:
+        """Log error for backend operations"""
+        error_msg = f"ERROR {context}: {str(error)}" if context else f"ERROR {str(error)}"
+        
+        if include_traceback:
+            error_msg += f"\nTraceback: {traceback.format_exc()}"
+        
+        logger.error(error_msg)
+
+    @staticmethod
+    def log_warning(message: str, context: str = "") -> None:
+        """Log warning for backend operations"""
+        warning_msg = f"WARNING {context}: {message}" if context else f"WARNING {message}"
+        logger.warning(warning_msg)
+
+def safe_execute(func, *args, default_return=None, error_context="", **kwargs):
+    """Safely execute a function and handle errors"""
+    try:
+        return func(*args, **kwargs)
+    except Exception as e:
+        ErrorHandler.log_error(e, error_context)
+        return default_return

backend/utils/model_manager.py

@@ -0,0 +1,177 @@
+import torch
+from typing import Dict, Any, List, Optional, Tuple
+from pathlib import Path
+
+from backend.models.registry import (
+    build as build_model,
+    get_model_info as get_registry_model_info,
+    choices,
+)
+from backend.config import TARGET_LEN
+from backend.pydantic_models import ModelInfo
+
+
+class ModelManager:
+    """
+    Centralized manager for discovering, loading, and caching ML models and their weights.
+    Ensures consistent model loading logic across different services.
+    """
+
+    def __init__(self):
+        self._model_cache: Dict[str, Dict[str, Any]] = {}
+        self._weights_cache: Dict[str, torch.nn.Module] = {}
+        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+        print(f"✅ ModelManager initialized on {self.device}")
+
+    def _load_state_dict(self, model_path: Path) -> Optional[Dict]:
+        """Production load: Strict security with weights_only enforcement."""
+        try:
+            if not model_path.exists():
+                return None
+
+            mtime = model_path.stat().st_mtime
+            cache_key = f"{model_path}:{mtime}"
+
+            if cache_key not in self._weights_cache:
+                # Strictly enforced security load
+                self._weights_cache[cache_key] = torch.load(
+                    model_path, map_location=self.device, weights_only=True
+                )
+            return self._weights_cache[cache_key]
+        except Exception as e:
+            print(f"❌ Security/Load Error for {model_path.name}: {e}")
+            return None
+
+    def load_model(
+        self, model_name: str, target_len: int = TARGET_LEN
+    ) -> Tuple[torch.nn.Module, bool, Path]:
+        """
+        Load a trained model for inference, including its weights.
+        Caches the loaded model.
+
+        Args:
+            model_name (str): Name of the model architecture (from registry).
+            target_len (int): Expected input length for the model.
+
+        Returns:
+            Tuple[torch.nn.Module, bool, Path]: The loaded model, a boolean indicating
+            if weights were successfully loaded, and the path to the loaded weights.
+        """
+        # Always use lowercase for filenames
+        model_name_lower = model_name.lower()
+        # Use absolute path for weights directory
+        weights_dir = Path(__file__).parent.parent / "models" / "weights"
+        potential_weight_paths = [
+            weights_dir / f"{model_name_lower}_model.pth",
+            weights_dir / f"{model_name_lower}.pth",
+        ]
+
+        if model_name_lower in self._model_cache:
+            model_entry = self._model_cache[model_name_lower]
+            return (
+                model_entry["model"],
+                model_entry["weights_loaded"],
+                model_entry["weights_path"],
+            )
+
+        if model_name_lower not in [m.lower() for m in choices()]:
+            print(f"⚠️ Model '{model_name_lower}' not found in registry.")
+            return None, False, Path("")
+
+        model = build_model(model_name_lower, target_len)
+        weights_loaded = False
+        loaded_path = Path("")
+
+        for weight_path in potential_weight_paths:
+            print(f"🔍 Checking for weights at {weight_path}")  # Debug log
+            if weight_path.exists():
+                try:
+                    state_dict = self._load_state_dict(weight_path)
+                    if state_dict:
+                        model.load_state_dict(state_dict, strict=True)
+                        model.to(self.device)
+                        model.eval()
+                        weights_loaded = True
+                        loaded_path = weight_path
+                        print(
+                            f"✅ Loaded weights for {model_name_lower} from {loaded_path}"
+                        )
+                        break
+                except (OSError, RuntimeError, KeyError) as e:
+                    print(
+                        f"❌ Error loading weights for {model_name_lower} from {weight_path}: {e}"
+                    )
+                    continue
+            else:
+                print(f"🔍 Weights not found for {model_name_lower} at {weight_path}")
+
+        if not weights_loaded:
+            print(
+                f"⚠️ No weights loaded for model '{model_name_lower}'. Model will use random initialization."
+            )
+            model.to(self.device)
+            model.eval()  # Ensure model is in eval mode even if no weights loaded
+
+        self._model_cache[model_name_lower] = {
+            "model": model,
+            "weights_loaded": weights_loaded,
+            "weights_path": loaded_path,
+            "target_len": target_len,
+            "device": self.device,
+        }
+        return model, weights_loaded, loaded_path
+
+    def get_model_info(self, model_name: str) -> Optional[Dict[str, Any]]:
+        """Get detailed information for a specific model."""
+        if model_name not in choices():
+            return None
+        info = get_registry_model_info(model_name)
+        # Add runtime info if model is loaded
+        if model_name in self._model_cache:
+            cached_info = self._model_cache[model_name]
+            info["weights_loaded"] = cached_info["weights_loaded"]
+            info["weights_path"] = str(cached_info["weights_path"])
+            info["device"] = str(cached_info["device"])
+            info["available"] = True
+        else:
+            # Check if weights exist even if not loaded yet
+            model_name = model_name.lower()
+            weights_exist = any(
+                (Path("backend/models/weights") / f"{model_name}_model.pth").exists()
+                or (Path("backend/models/weights") / f"{model_name}.pth").exists()
+                for _ in [0]
+            )  # Dummy loop to check both paths
+            info["weights_loaded"] = False
+            info["weights_path"] = None
+            info["device"] = str(self.device)
+            # Mark as available if weights are present
+            info["available"] = weights_exist
+
+        return info
+
+    def get_available_models(self) -> List[ModelInfo]:
+        """Get a list of all models with their availability status."""
+        models_list = []
+        for model_name in choices():
+            info = self.get_model_info(model_name)
+            if info:
+                models_list.append(
+                    ModelInfo(
+                        name=model_name,
+                        description=info.get("description", ""),
+                        input_length=info.get("input_length", TARGET_LEN),
+                        num_classes=info.get("num_classes", 2),
+                        supported_modalities=info.get("modalities", ["raman", "ftir"]),
+                        performance=info.get("performance", {}),
+                        parameters=info.get("parameters"),
+                        speed=info.get("speed"),
+                        citation=info.get("citation"),
+                        # Use the 'available' status from get_model_info
+                        available=info.get("available", False),
+                    )
+                )
+        return models_list
+
+
+# Global instance of the ModelManager
+model_manager = ModelManager()

backend/utils/multifile.py

@@ -0,0 +1,480 @@
+"""Multi-file processing utilities for batch inference.
+Handles multiple file uploads and iterative processing.
+Supports TXT, CSV, and JSON file formats with automatic detection."""
+
+from typing import List, Dict, Any, Tuple, Optional
+import time
+import io
+from pathlib import Path
+import numpy as np
+import json
+import csv
+import hashlib
+
+from backend.utils.preprocessing import preprocess_spectrum
+from backend.utils.errors import ErrorHandler
+from backend.utils.confidence import calculate_softmax_confidence
+from backend.config import TARGET_LEN
+
+
+def detect_file_format(filename: str, content: str) -> str:
+    """Automatically detect file format based on exstention and content
+
+    Args:
+        filename: Name of the file
+        content: Content of the file
+
+    Returns:
+        File format: .'txt', .'csv', .'json'
+    """
+    # First try by extension
+    suffix = Path(filename).suffix.lower()
+    if suffix == ".json":
+        try:
+            json.loads(content)
+            return "json"
+        except json.JSONDecodeError:
+            pass
+    elif suffix == ".csv":
+        return "csv"
+    elif suffix == ".txt":
+        return "txt"
+
+    # If extension doesn't match or is unclear, try content detection
+    content_stripped = content.strip()
+
+    # Try JSON
+    if content_stripped.startswith(("{", "[")):
+        try:
+            json.loads(content)
+            return "json"
+        except json.JSONDecodeError:
+            pass
+
+    # Try CSV (look for commas in first few lines)
+    lines = content_stripped.split("\n")[:5]
+    comma_count = sum(line.count(",") for line in lines)
+    if comma_count > len(lines):  # More commas than lines suggests CSV
+        return "csv"
+
+    # Default to TXT
+    return "txt"
+
+
+def parse_json_spectrum(content: str) -> Tuple[np.ndarray, np.ndarray]:
+    """
+    Parse spectrum data from JSON format.
+
+    Expected formats:
+    - {"wavenumbers": [...], "intensities": [...]}
+    - {"x": [...], "y": [...]}
+    - [{"wavenumber": val, "intensity": val}, ...]
+    """
+
+    try:
+        data = json.loads(content)
+
+        # Format 1: Object with arrays
+        if isinstance(data, dict):
+            x_key = None
+            y_key = None
+
+            # Try common key names for x-axis
+            for key in ["wavenumbers", "wavenumber", "x", "freq", "frequency"]:
+                if key in data:
+                    x_key = key
+                    break
+
+            # Try common key names for y-axis
+            for key in ["intensities", "intensity", "y", "counts", "absorbance"]:
+                if key in data:
+                    y_key = key
+                    break
+
+            if x_key and y_key:
+                x_vals = np.array(data[x_key], dtype=float)
+                y_vals = np.array(data[y_key], dtype=float)
+                return x_vals, y_vals
+
+        # Format 2: Array of objects
+        elif isinstance(data, list) and len(data) > 0 and isinstance(data[0], dict):
+            x_vals = []
+            y_vals = []
+
+            for item in data:
+                # Try to find x and y values
+                x_val = None
+                y_val = None
+
+                for x_key in ["wavenumber", "wavenumbers", "x", "freq"]:
+                    if x_key in item:
+                        x_val = float(item[x_key])
+                        break
+
+                for y_key in ["intensity", "intensities", "y", "counts"]:
+                    if y_key in item:
+                        y_val = float(item[y_key])
+                        break
+
+                if x_val is not None and y_val is not None:
+                    x_vals.append(x_val)
+                    y_vals.append(y_val)
+
+            if x_vals and y_vals:
+                return np.array(x_vals), np.array(y_vals)
+
+        raise ValueError(
+            "JSON format not recognized. Expected wavenumber/intensity pairs."
+        )
+
+    except json.JSONDecodeError as e:
+        raise ValueError(f"Invalid JSON format: {str(e)}") from e
+    except Exception as e:
+        raise ValueError(f"Failed to parse JSON spectrum: {str(e)}") from e
+
+
+def parse_csv_spectrum(
+    content: str, filename: str = "unknown"
+) -> Tuple[np.ndarray, np.ndarray]:
+    """
+    Parse spectrum data from CSV format.
+
+    Handles various CSV formats with headers or without.
+    """
+    try:
+        # Use StringIO to treat string as file-like object
+        csv_file = io.StringIO(content)
+
+        # Try to detect delimiter
+        sample = content[:1024]
+        delimiter = ","
+        if sample.count(";") > sample.count(","):
+            delimiter = ";"
+        elif sample.count("\t") > sample.count(","):
+            delimiter = "\t"
+
+        # Read CSV
+        csv_reader = csv.reader(csv_file, delimiter=delimiter)
+        rows = list(csv_reader)
+
+        if not rows:
+            raise ValueError("Empty CSV file")
+
+        # Check if first row is header
+        has_header = False
+        try:
+            # If first row contains non-numeric data, it's likely a header
+            float(rows[0][0])
+            float(rows[0][1])
+        except (ValueError, IndexError):
+            has_header = True
+
+        data_rows = rows[1:] if has_header else rows
+
+        # Extract x and y values
+        x_vals = []
+        y_vals = []
+
+        for i, row in enumerate(data_rows):
+            if len(row) < 2:
+                continue
+
+            try:
+                x_val = float(row[0])
+                y_val = float(row[1])
+                x_vals.append(x_val)
+                y_vals.append(y_val)
+            except ValueError:
+                ErrorHandler.log_warning(
+                    f"Could not parse CSV row {i+1}: {row}", f"Parsing {filename}"
+                )
+                continue
+
+        if len(x_vals) < 10:
+            raise ValueError(
+                f"Insufficient data points ({len(x_vals)}). Need at least 10 points."
+            )
+
+        return np.array(x_vals), np.array(y_vals)
+
+    except Exception as e:
+        raise ValueError(f"Failed to parse CSV spectrum: {str(e)}") from e
+
+
+def parse_spectrum_data(
+    text_content: str, filename: str = "unknown", file_format: Optional[str] = None
+) -> Tuple[np.ndarray, np.ndarray]:
+    """
+    Parse spectrum data from text content with automatic format detection.
+    Args:
+        text_content: Raw text content of the spectrum file
+        filename: Name of the file for error reporting
+        file_format: Force specific format ('txt', 'csv', 'json') or None for auto-detection
+    Returns:
+        Tuple of (x_values, y_values) as numpy arrays
+    Raises:
+        ValueError: If the data cannot be parsed
+    """
+    try:
+        # Detect format if not specified
+        if file_format is None:
+            file_format = detect_file_format(filename, text_content)
+
+        # Parse based on detected/specified format
+        if file_format == "json":
+            x, y = parse_json_spectrum(text_content)
+        elif file_format == "csv":
+            x, y = parse_csv_spectrum(text_content, filename)
+        else:  # Default to TXT format
+            x, y = parse_txt_spectrum(text_content, filename)
+
+        # Common validation for all formats
+        validate_spectrum_data(x, y, filename)
+
+        return x, y
+
+    except Exception as e:
+        raise ValueError(f"Failed to parse spectrum data: {str(e)}") from e
+
+
+def parse_txt_spectrum(
+    content: str, filename: str = "unknown"
+) -> Tuple[np.ndarray, np.ndarray]:
+    """Robustly parse spectrum data from TXT format."""
+    lines = content.strip().split("\n")
+    x_vals, y_vals = [], []
+
+    for i, line in enumerate(lines):
+        line = line.strip()
+        if not line or line.startswith(("#", "%")):
+            continue
+
+        try:
+            # Handle different separators
+            parts = line.replace(",", " ").replace(";", " ").replace("\t", " ").split()
+
+            # Find the first two valid numbers in the line
+            numbers = []
+            for part in parts:
+                if part:  # Skip empty strings from multiple spaces
+                    try:
+                        numbers.append(float(part))
+                    except ValueError:
+                        continue  # Ignore non-numeric parts
+
+            if len(numbers) >= 2:
+                x_vals.append(numbers[0])
+                y_vals.append(numbers[1])
+            else:
+                ErrorHandler.log_warning(
+                    f"Could not find two numbers on line {i+1}: '{line}'",
+                    f"Parsing {filename}",
+                )
+
+        except ValueError as e:
+            ErrorHandler.log_warning(
+                f"Error parsing line {i+1}: '{line}'. Error: {e}",
+                f"Parsing {filename}",
+            )
+            continue
+
+    if len(x_vals) < 10:
+        raise ValueError(
+            f"Insufficient data points ({len(x_vals)}). Need at least 10 points."
+        )
+
+    return np.array(x_vals), np.array(y_vals)
+
+
+def validate_spectrum_data(x: np.ndarray, y: np.ndarray, filename: str) -> None:
+    """
+    Validate parsed spectrum data for common issues.
+    """
+    # Check for NaNs
+    if np.any(np.isnan(x)) or np.any(np.isnan(y)):
+        raise ValueError("Input data contains NaN values")
+
+    # Check monotonic increasing x (sort if needed)
+    if not np.all(np.diff(x) >= 0):
+        # Sort by x values if not monotonic
+        sort_idx = np.argsort(x)
+        x = x[sort_idx]
+        y = y[sort_idx]
+        ErrorHandler.log_warning(
+            "Wavenumbers were not monotonic - data has been sorted",
+            f"Parsing {filename}",
+        )
+
+    # Check reasonable range for spectroscopy
+    if min(x) < 0 or max(x) > 10000 or (max(x) - min(x)) < 100:
+        ErrorHandler.log_warning(
+            f"Unusual wavenumber range: {min(x):.1f} - {max(x):.1f} cm⁻¹",
+            f"Parsing {filename}",
+        )
+
+
+def process_single_file(
+    filename: str,
+    text_content: str,
+    model_choice: str,
+    run_inference_func,
+    label_file_func,
+    modality: str,
+    target_len: int,
+) -> Optional[Dict[str, Any]]:
+    """
+    Process a single spectrum file
+
+    Args:
+        filename: Name of the file
+        text_content: Raw text content
+        model_choice: Selected model name
+        run_inference_func: Function to run inference
+        label_file_func: Function to extract ground truth label
+
+    Returns:
+        Dictionary with processing results or None if failed
+    """
+    start_time = time.time()
+
+    try:
+        # 1. Parse spectrum data
+        x_raw, y_raw = parse_spectrum_data(text_content, filename)
+
+        # 2. Preprocess spectrum using the full, modality-aware pipeline
+        x_resampled, y_resampled = preprocess_spectrum(
+            x_raw, y_raw, modality=modality, target_len=target_len
+        )
+
+        # 3. Run inference, passing modality
+        cache_key = hashlib.md5(
+            f"{y_resampled.tobytes()}{model_choice}".encode()
+        ).hexdigest()
+        prediction, logits_list, probs, logits = run_inference_func(
+            y_resampled, model_choice, modality=modality, cache_key=cache_key
+        )
+
+        if prediction is None:
+            raise ValueError("Inference returned None. Model may have failed to load.")
+
+        # ==Calculate confidence==
+        if logits is not None:
+            probs_np, max_confidence, confidence_level, confidence_emoji = (
+                calculate_softmax_confidence(logits)
+            )
+        else:
+            # Fallback for older models or if logits are not returned
+            probs_np = np.array(probs) if probs is not None else np.array([])
+            max_confidence = float(np.max(probs_np)) if probs_np.size > 0 else 0.0
+            confidence_level = "LOW"
+            confidence_emoji = "🔴"
+
+        # ==Get ground truth==
+        ground_truth = label_file_func(filename)
+        ground_truth = (
+            ground_truth if ground_truth is not None and ground_truth >= 0 else None
+        )
+
+        # ==Get predicted class==
+        label_map = {0: "Stable (Unweathered)", 1: "Weathered (Degraded)"}
+        predicted_class = label_map.get(int(prediction), f"Unknown ({prediction})")
+
+        processing_time = time.time() - start_time
+
+        return {
+            "filename": filename,
+            "success": True,
+            "prediction": int(prediction),
+            "predicted_class": predicted_class,
+            "confidence": max_confidence,
+            "confidence_level": confidence_level,
+            "confidence_emoji": confidence_emoji,
+            "logits": logits_list if logits_list else [],
+            "probabilities": probs_np.tolist() if len(probs_np) > 0 else [],
+            "ground_truth": ground_truth,
+            "processing_time": processing_time,
+            "x_raw": x_raw,
+            "y_raw": y_raw,
+            "x_resampled": x_resampled,
+            "y_resampled": y_resampled,
+        }
+
+    except ValueError as e:
+        ErrorHandler.log_error(e, f"processing {filename}")
+        return {
+            "filename": filename,
+            "success": False,
+            "error": str(e),
+            "processing_time": time.time() - start_time,
+        }
+
+
+def process_multiple_files(
+    uploaded_files: List,
+    model_choice: str,
+    run_inference_func,
+    label_file_func,
+    modality: str,
+    progress_callback=None,
+) -> List[Dict[str, Any]]:
+    """
+    Process multiple uploaded files
+
+    Args:
+        uploaded_files: List of uploaded file objects
+        model_choice: Selected model name
+        run_inference_func: Function to run inference
+        label_file_func: Function to extract ground truth label
+        progress_callback: Optional callback to update progress
+
+    Returns:
+        List of processing results
+    """
+    results = []
+    total_files = len(uploaded_files)
+
+    ErrorHandler.log_message(
+        f"Starting batch processing of {total_files} files with modality '{modality}'"
+    )
+
+    for i, uploaded_file in enumerate(uploaded_files):
+        if progress_callback:
+            progress_callback(i, total_files, uploaded_file.name)
+
+        try:
+            # ==Read file content==
+            raw = uploaded_file.read()
+            text_content = raw.decode("utf-8") if isinstance(raw, bytes) else raw
+
+            # ==Process the file==
+            result = process_single_file(
+                filename=uploaded_file.name,
+                text_content=text_content,
+                model_choice=model_choice,
+                run_inference_func=run_inference_func,
+                label_file_func=label_file_func,
+                modality=modality,
+                target_len=TARGET_LEN,
+            )
+
+            if result:
+                results.append(result)
+
+        except ValueError as e:
+            ErrorHandler.log_error(e, f"reading file {uploaded_file.name}")
+            results.append(
+                {
+                    "filename": uploaded_file.name,
+                    "success": False,
+                    "error": f"Failed to read file: {str(e)}",
+                }
+            )
+
+    if progress_callback:
+        progress_callback(total_files, total_files, "Complete")
+
+    ErrorHandler.log_message(
+        f"Completed batch processing: {sum(1 for r in results if r.get('success', False))}/{total_files} successful"
+    )
+
+    return results

backend/utils/performance.py

@@ -0,0 +1,101 @@
+"""Performance benchmarking and monitoring for the backend API."""
+
+import logging
+import psutil
+import threading
+from typing import Dict, Any, Optional
+from datetime import datetime
+from pathlib import Path
+import os
+import tempfile
+import sys
+# Configure performance logger
+performance_logger = logging.getLogger('performance')
+performance_logger.setLevel(logging.INFO)
+
+# Determine writable log directory (env override), fallback to temp dir
+tmp_base = Path(os.getenv("PERF_LOG_DIR", tempfile.gettempdir()))
+log_dir = tmp_base / "ml_polymer_logs"
+
+# Try to create and use a file handler; if that fails, fallback to stdout StreamHandler
+try:
+    log_dir.mkdir(parents=True, exist_ok=True)
+    file_handler = logging.FileHandler(log_dir / "performance.log", encoding="utf-8")
+    file_handler.setLevel(logging.INFO)
+    formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
+    file_handler.setFormatter(formatter)
+    performance_logger.addHandler(file_handler)
+except Exception as e:
+    # Fallback to stdout so HF Spaces / container logs capture the output
+    stream_handler = logging.StreamHandler(sys.stdout)
+    stream_handler.setLevel(logging.INFO)
+    formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
+    stream_handler.setFormatter(formatter)
+    performance_logger.addHandler(stream_handler)
+    performance_logger.warning("Could not create file handler for performance logs, using stdout: %s", e)
+
+class PerformanceBenchmark:
+    """Context manager for benchmarking operations."""
+
+    def __init__(self, operation_name: str, metadata: Optional[Dict[str, Any]] = None):
+        self.operation_name = operation_name
+        self.metadata = metadata or {}
+        self.start_time = 0
+        self.start_memory = 0
+        self.duration = 0
+        self.memory_delta = 0
+        self.performance_data = {}
+
+    def __enter__(self):
+        self.start_time = time.time()
+        self.start_memory = psutil.Process().memory_info().rss / 1024 / 1024  # MB
+        return self
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        duration = time.time() - self.start_time
+        end_memory = psutil.Process().memory_info().rss / 1024 / 1024  # MB
+        memory_delta = end_memory - self.start_memory
+
+        # Log performance data
+        perf_data = {
+            "operation": self.operation_name,
+            "duration_seconds": round(duration, 4),
+            "memory_start_mb": round(self.start_memory, 2),
+            "memory_end_mb": round(end_memory, 2),
+            "memory_delta_mb": round(memory_delta, 2),
+            "timestamp": datetime.utcnow().isoformat(),
+            **self.metadata
+        }
+
+        performance_logger.info(f"BENCHMARK: {perf_data}")
+
+        # Store in class for retrieval
+        self.duration = duration
+        self.memory_delta = memory_delta
+        self.performance_data = perf_data
+
+def log_model_performance(model_name: str, inference_time: float,
+                         preprocessing_time: float, total_time: float,
+                         memory_usage: float, spectrum_length: int):
+    """Log model inference performance metrics."""
+    perf_data = {
+        "operation": "model_inference",
+        "model_name": model_name,
+        "inference_time": round(inference_time, 4),
+        "preprocessing_time": round(preprocessing_time, 4),
+        "total_time": round(total_time, 4),
+        "memory_usage_mb": round(memory_usage, 2),
+        "spectrum_length": spectrum_length,
+        "timestamp": datetime.utcnow().isoformat()
+    }
+
+    performance_logger.info(f"MODEL_PERF: {perf_data}")
+
+def get_system_performance():
+    """Get current system performance metrics."""
+    return {
+        "cpu_percent": psutil.cpu_percent(interval=1),
+        "memory_percent": psutil.virtual_memory().percent,
+        "memory_available_mb": psutil.virtual_memory().available / 1024 / 1024,
+        "timestamp": datetime.utcnow().isoformat()
+    }

backend/utils/prepare_data.py

@@ -0,0 +1,76 @@
+"""
+Data Preparation Script
+
+This script takes a raw dataset, performs a stratified split into
+training, validation, and test sets, and saves them to a processed
+data directory. This ensures a consistent and reproducible data
+splitting strategy.
+
+Usage:
+    python scripts/prepare_data.py --data-path /path/to/raw/data.csv --output-path data/processed
+"""
+
+from pathlib import Path
+import argparse
+import pandas as pd
+from sklearn.model_selection import train_test_split
+
+
+def prepare_data(data_path: Path, output_path: Path, test_size: float = 0.2, val_size: float = 0.15):
+    """
+    Loads data, performs stratified train-val-test split, and saves the splits.
+
+    Args:
+        data_path (Path): Path to the raw data file (CSV expected).
+        output_path (Path): Directory to save the processed data splits.
+        test_size (float): Proportion of the dataset to include in the test split.
+        val_size (float): Proportion of the training set to use for validation.
+    """
+    if not data_path.exists():
+        raise FileNotFoundError(f"Raw data not found at {data_path}")
+
+    print(f"Loading data from {data_path}...")
+    # This assumes a CSV with a 'spectra' column and a 'label' column.
+    # You will need to adapt this to your actual raw data format.
+    df = pd.read_csv(data_path)
+
+    # Ensure the 'label' column exists in the dataset
+    if 'label' not in df.columns:
+        raise ValueError(
+            "The input data must contain a 'label' column for stratified splitting.")
+
+    # Ensure output directory exists
+    output_path.mkdir(parents=True, exist_ok=True)
+
+    print("Performing stratified train-test split...")
+    # Split off the test set first
+    train_val_df, test_df = train_test_split(
+        df, test_size=test_size, stratify=df['label'], random_state=42
+    )
+
+    # Split the remaining data into training and validation sets
+    train_df, val_df = train_test_split(
+        train_val_df, test_size=val_size, stratify=train_val_df['label'], random_state=42
+    )
+
+    print(f"Train set size: {len(train_df)}")
+    print(f"Validation set size: {len(val_df)}")
+    print(f"Test set size: {len(test_df)}")
+
+    # Save the splits
+    train_df.to_csv(output_path / "train.csv", index=False)
+    val_df.to_csv(output_path / "validation.csv", index=False)
+    test_df.to_csv(output_path / "test.csv", index=False)
+
+    print(f"✅ Data splits saved to {output_path}")
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(
+        description="Prepare and split spectral data.")
+    parser.add_argument("--data-path", type=Path, required=True,
+                        help="Path to the raw data CSV file.")
+    parser.add_argument("--output-path", type=Path, default=Path(
+        "data/processed"), help="Directory to save data splits.")
+    args = parser.parse_args()
+    prepare_data(args.data_path, args.output_path)

backend/utils/preprocessing.py

@@ -0,0 +1,331 @@
+"""
+Preprocessing utilities for polymer classification app.
+Adapted from the original scripts/preprocess_dataset.py for Hugging Face Spaces deployment.
+Supports both Raman and FTIR spectroscopy modalities.
+"""
+
+from __future__ import annotations
+import numpy as np
+from numpy.typing import DTypeLike
+from scipy.interpolate import interp1d
+from scipy.signal import savgol_filter
+from typing import Tuple, Literal, Optional
+
+TARGET_LENGTH = 500  # Frozen default per PREPROCESSING_BASELINE
+
+# Modality-specific validation ranges (cm⁻¹)
+MODALITY_RANGES = {
+    "raman": (200, 4000),  # Typical Raman range
+    "ftir": (400, 4000),  # FTIR wavenumber range
+}
+
+# Modality-specific preprocessing parameters
+MODALITY_PARAMS = {
+    "raman": {
+        "baseline_degree": 2,
+        "smooth_window": 11,
+        "smooth_polyorder": 2,
+        "cosmic_ray_removal": False,
+    },
+    "ftir": {
+        "baseline_degree": 2,
+        "smooth_window": 13,  # Slightly larger window for FTIR
+        "smooth_polyorder": 2,
+        "cosmic_ray_removal": False,
+        "atmospheric_correction": False,  # Placeholder for future implementation
+    },
+}
+
+
+def _ensure_1d_equal(x: np.ndarray, y: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
+    x = np.asarray(x, dtype=float)
+    y = np.asarray(y, dtype=float)
+    if x.ndim != 1 or y.ndim != 1 or x.size != y.size or x.size < 2:
+        raise ValueError("x and y must be 1D arrays of equal length >= 2")
+    return x, y
+
+
+def resample_spectrum(
+    x: np.ndarray, y: np.ndarray, target_len: int = TARGET_LENGTH
+) -> tuple[np.ndarray, np.ndarray]:
+    """Linear re-sampling onto a uniform grid of length target_len."""
+    x, y = _ensure_1d_equal(x, y)
+    order = np.argsort(x)
+    x_sorted, y_sorted = x[order], y[order]
+    x_new = np.linspace(x_sorted[0], x_sorted[-1], int(target_len))
+    f = interp1d(x_sorted, y_sorted, kind="linear", assume_sorted=True)
+    y_new = f(x_new)
+    return x_new, y_new
+
+
+def remove_baseline(y: np.ndarray, degree: int = 2) -> np.ndarray:
+    """Polynomial baseline subtraction (degree=2 default)"""
+    y = np.asarray(y, dtype=float)
+    x_idx = np.arange(y.size, dtype=float)
+    coeffs = np.polyfit(x_idx, y, deg=int(degree))
+    baseline = np.polyval(coeffs, x_idx)
+    return y - baseline
+
+
+def smooth_spectrum(
+    y: np.ndarray, window_length: int = 11, polyorder: int = 2
+) -> np.ndarray:
+    """Savitzky-Golay smoothing with safe/odd window enforcement"""
+    y = np.asarray(y, dtype=float)
+    window_length = int(window_length)
+    polyorder = int(polyorder)
+    # === window must be odd and >= polyorder+1 ===
+    if window_length % 2 == 0:
+        window_length += 1
+    min_win = polyorder + 1
+    if min_win % 2 == 0:
+        min_win += 1
+    window_length = max(window_length, min_win)
+    return savgol_filter(
+        y, window_length=window_length, polyorder=polyorder, mode="interp"
+    )
+
+
+def normalize_spectrum(y: np.ndarray) -> np.ndarray:
+    """Min-max normalization to [0, 1] with constant-signal guard."""
+    y = np.asarray(y, dtype=float)
+    y_min = float(np.min(y))
+    y_max = float(np.max(y))
+    if np.isclose(y_max - y_min, 0.0):
+        return np.zeros_like(y)
+    return (y - y_min) / (y_max - y_min)
+
+
+def validate_spectrum_range(x: np.ndarray, modality: str = "raman") -> bool:
+    """Validate that spectrum wavenumbers are within expected range for modality."""
+    if modality not in MODALITY_RANGES:
+        raise ValueError(
+            f"Unknown modality '{modality}'. Supported: {list(MODALITY_RANGES.keys())}"
+        )
+
+    min_range, max_range = MODALITY_RANGES[modality]
+    x_min, x_max = np.min(x), np.max(x)
+
+    # Check if majority of data points are within range
+    in_range = np.sum((x >= min_range) & (x <= max_range))
+    total_points = len(x)
+
+    return bool((in_range / total_points) >= 0.7)  # At least 70% should be in range
+
+
+def validate_spectrum_modality(
+    x_data: np.ndarray, y_data: np.ndarray, selected_modality: str
+) -> Tuple[bool, list[str]]:
+    """
+    Validate that spectrum characteristics match the selected modality.
+
+    Args:
+        x_data: Wavenumber array (cm⁻¹)
+        y_data: Intensity array
+        selected_modality: Selected modality ('raman' or 'ftir')
+
+    Returns:
+        Tuple of (is_valid, list_of_issues)
+    """
+    x_data = np.asarray(x_data)
+    y_data = np.asarray(y_data)
+    issues = []
+
+    if selected_modality not in MODALITY_RANGES:
+        issues.append(f"Unknown modality: {selected_modality}")
+        return False, issues
+
+    expected_min, expected_max = MODALITY_RANGES[selected_modality]
+    actual_min, actual_max = np.min(x_data), np.max(x_data)
+
+    # Check wavenumber range
+    if actual_min < expected_min * 0.8:  # Allow 20% tolerance
+        issues.append(
+            f"Minimum wavenumber ({actual_min:.0f} cm⁻��) is below typical {selected_modality.upper()} range (>{expected_min} cm⁻¹)"
+        )
+
+    if actual_max > expected_max * 1.2:  # Allow 20% tolerance
+        issues.append(
+            f"Maximum wavenumber ({actual_max:.0f} cm⁻¹) is above typical {selected_modality.upper()} range (<{expected_max} cm⁻¹)"
+        )
+
+    # Check for reasonable data range coverage
+    data_range = actual_max - actual_min
+    expected_range = expected_max - expected_min
+    if data_range < expected_range * 0.3:  # Should cover at least 30% of expected range
+        issues.append(
+            f"Data range ({data_range:.0f} cm⁻¹) seems narrow for {selected_modality.upper()} spectroscopy"
+        )
+
+    # FTIR-specific checks
+    if selected_modality == "ftir":
+        # Check for typical FTIR characteristics
+        if actual_min > 1000:  # FTIR usually includes fingerprint region
+            issues.append(
+                "FTIR data should typically include fingerprint region (400-1500 cm⁻¹)"
+            )
+
+    # Raman-specific checks
+    if selected_modality == "raman":
+        # Check for typical Raman characteristics
+        if actual_max < 1000:  # Raman usually extends to higher wavenumbers
+            issues.append(
+                "Raman data typically extends to higher wavenumbers (>1000 cm⁻¹)"
+            )
+
+    return len(issues) == 0, issues
+
+
+def preprocess_spectrum(
+    x: np.ndarray,
+    y: np.ndarray,
+    *,
+    target_len: int = TARGET_LENGTH,
+    modality: str = "raman",  # New parameter for modality-specific processing
+    do_baseline: bool = True,
+    degree: int | None = None,  # Will use modality default if None
+    do_smooth: bool = True,
+    window_length: int | None = None,  # Will use modality default if None
+    polyorder: int | None = None,  # Will use modality default if None
+    do_normalize: bool = True,
+    out_dtype: DTypeLike = np.float32,
+    validate_range: bool = True,
+) -> tuple[np.ndarray, np.ndarray]:
+    """
+    Modality-aware preprocessing: resample -> baseline -> smooth -> normalize
+
+    Args:
+        x, y: Input spectrum data
+        target_len: Target length for resampling
+        modality: 'raman' or 'ftir' for modality-specific processing
+        do_baseline: Enable baseline correction
+        degree: Polynomial degree for baseline (uses modality default if None)
+        do_smooth: Enable smoothing
+        window_length: Smoothing window length (uses modality default if None)
+        polyorder: Polynomial order for smoothing (uses modality default if None)
+        do_normalize: Enable normalization
+        out_dtype: Output data type
+        validate_range: Check if wavenumbers are in expected range for modality
+
+    Returns:
+        Tuple of (resampled_x, processed_y)
+    """
+    # Validate modality
+    if modality not in MODALITY_PARAMS:
+        raise ValueError(
+            f"Unsupported modality '{modality}'. Supported: {list(MODALITY_PARAMS.keys())}"
+        )
+
+    # Get modality-specific parameters
+    modality_config = MODALITY_PARAMS[modality]
+
+    # Use modality defaults if parameters not specified
+    if degree is None:
+        degree = modality_config["baseline_degree"]
+    if window_length is None:
+        window_length = modality_config["smooth_window"]
+    if polyorder is None:
+        polyorder = modality_config["smooth_polyorder"]
+
+    # Validate spectrum range if requested
+    if validate_range:
+        if not validate_spectrum_range(x, modality):
+            print(
+                f"Warning: Spectrum wavenumbers may not be optimal for {modality.upper()} analysis"
+            )
+
+    # Standard preprocessing pipeline
+    x_rs, y_rs = resample_spectrum(x, y, target_len=target_len)
+
+    if do_baseline:
+        y_rs = remove_baseline(y_rs, degree=degree)
+
+    if do_smooth:
+        y_rs = smooth_spectrum(y_rs, window_length=window_length, polyorder=polyorder)
+
+    # FTIR-specific processing
+    if modality == "ftir":
+        if modality_config.get("atmospheric_correction", False):
+            y_rs = remove_atmospheric_interference(y_rs)
+        if modality_config.get("water_correction", False):
+            y_rs = remove_water_vapor_bands(y_rs, x_rs)
+
+    if do_normalize:
+        y_rs = normalize_spectrum(y_rs)
+
+    # === Coerce to a real dtype to satisfy static checkers & runtime ===
+    out_dt = np.dtype(out_dtype)
+    return x_rs.astype(out_dt, copy=False), y_rs.astype(out_dt, copy=False)
+
+
+def remove_atmospheric_interference(y: np.ndarray) -> np.ndarray:
+    """Remove atmospheric CO2 and H2O interference common in FTIR."""
+    y = np.asarray(y, dtype=float)
+
+    # Simple atmospheric correction using median filtering
+    # This is a basic implementation - in practice would use reference spectra
+    from scipy.signal import medfilt
+
+    # Apply median filter to reduce sharp atmospheric lines
+    corrected = medfilt(y, kernel_size=5)
+
+    # Blend with original to preserve peak structure
+    alpha = 0.7  # Weight for original spectrum
+    return alpha * y + (1 - alpha) * corrected
+
+
+def remove_water_vapor_bands(y: np.ndarray, x: np.ndarray) -> np.ndarray:
+    """Remove water vapor interference bands in FTIR spectra."""
+    y = np.asarray(y, dtype=float)
+    x = np.asarray(x, dtype=float)
+
+    # Common water vapor regions in FTIR (cm⁻¹)
+    water_regions = [(3500, 3800), (1300, 1800)]
+
+    corrected_y = y.copy()
+
+    for low, high in water_regions:
+        # Find indices in water vapor region
+        mask = (x >= low) & (x <= high)
+        if np.any(mask):
+            # Simple linear interpolation across water regions
+            indices = np.where(mask)[0]
+            if len(indices) > 2:
+                start_idx, end_idx = indices[0], indices[-1]
+                if start_idx > 0 and end_idx < len(y) - 1:
+                    # Linear interpolation between boundary points
+                    start_val = y[start_idx - 1]
+                    end_val = y[end_idx + 1]
+                    interp_vals = np.linspace(start_val, end_val, len(indices))
+                    corrected_y[mask] = interp_vals
+
+    return corrected_y
+
+
+def apply_ftir_specific_processing(
+    x: np.ndarray,
+    y: np.ndarray,
+    atmospheric_correction: bool = False,
+    water_correction: bool = False,
+) -> tuple[np.ndarray, np.ndarray]:
+    """Apply FTIR-specific preprocessing steps."""
+    processed_y = y.copy()
+
+    if atmospheric_correction:
+        processed_y = remove_atmospheric_interference(processed_y)
+
+    if water_correction:
+        processed_y = remove_water_vapor_bands(processed_y, x)
+
+    return x, processed_y
+
+
+def get_modality_info(modality: str) -> dict:
+    """Get processing parameters and validation ranges for a modality."""
+    if modality not in MODALITY_PARAMS:
+        raise ValueError(f"Unknown modality '{modality}'")
+
+    return {
+        "range": MODALITY_RANGES[modality],
+        "params": MODALITY_PARAMS[modality].copy(),
+    }

backend/utils/preprocessing_fixed.py

@@ -0,0 +1,301 @@
+# pylint: disable=missing-function-docstring, missing-class-docstring, missing-module-docstring, redefined-outer-name, unused-argument, unused-import, singleton-comparison, invalid-name, wrong-import-position, too-many-arguments, too-many-locals, too-many-statements, wrong-import-order
+"""
+preprocessing_fixed.py
+Data leakage-free preprocessing pipeline for polymer aging classification.
+This module ensures that preprocessing transformations (normalization, scaling, etc.)
+are fitted only on training data within each cross-validation fold.
+CRITICAL: This fixes the data leakage issue where preprocessing was applied
+to the entire dataset before cross-validation splits.
+"""
+
+import os
+import sys
+import numpy as np
+from typing import Tuple, Optional, Dict, Any
+
+# Add parent directory to path for imports
+sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), "../..")))
+
+from .raman_util import list_txt_files, label_file, load_spectrum
+from backend.utils.preprocessing import preprocess_spectrum, TARGET_LENGTH
+
+class SpectrumPreprocessor:
+    """
+    Data leakage-free preprocessing pipeline for spectral data.
+
+    This class ensures that normalization and other transformations
+    are fitted only on training data within each CV fold.
+    """
+
+    def __init__(
+        self,
+        target_len: int = TARGET_LENGTH,
+        do_baseline: bool = True,
+        do_smooth: bool = True,
+        do_normalize: bool = True,
+        modality: str = "raman"
+    ):
+        """
+        Initialize the preprocessor with configuration.
+
+        Args:
+            target_len (int): Target length for resampling
+            do_baseline (bool): Whether to apply baseline correction
+            do_smooth (bool): Whether to apply smoothing
+            do_normalize (bool): Whether to apply normalization
+            modality (str): Spectroscopy modality ('raman' or 'ftir')
+        """
+        self.target_len = target_len
+        self.do_baseline = do_baseline
+        self.do_smooth = do_smooth
+        self.do_normalize = do_normalize
+        self.modality = modality
+
+        # Stats fitted on training data only
+        self.normalization_stats = None
+        self.is_fitted = False
+
+    def load_raw_data(self, dataset_dir: str) -> Tuple[np.ndarray, np.ndarray, list]:
+        """
+        Load raw spectrum data without preprocessing.
+
+        Args:
+            dataset_dir (str): Path to dataset directory
+
+        Returns:
+            tuple: (raw_spectra, labels, file_paths)
+        """
+        txt_paths = list_txt_files(dataset_dir)
+        raw_spectra = []
+        labels = []
+        valid_files = []
+
+        for path in txt_paths:
+            label = label_file(path)
+            if label is None:
+                continue
+
+            try:
+                x_raw, y_raw = load_spectrum(path)
+                if len(x_raw) < 10:
+                    continue  # Skip files with too few points
+
+                raw_spectra.append((x_raw, y_raw))
+                labels.append(int(label))
+                valid_files.append(path)
+
+            except (IOError, ValueError) as e:
+                print(f"⚠️ Warning: Failed to load {path}: {e}")
+                continue
+
+        return np.array(raw_spectra, dtype=object), np.array(labels), valid_files
+
+    def preprocess_single_spectrum(
+        self,
+        x_raw: np.ndarray,
+        y_raw: np.ndarray,
+        use_fitted_stats: bool = False
+    ) -> np.ndarray:
+        """
+        Preprocess a single spectrum.
+
+        Args:
+            x_raw (np.ndarray): Raw wavenumber values
+            y_raw (np.ndarray): Raw intensity values
+            use_fitted_stats (bool): Whether to use fitted normalization stats
+
+        Returns:
+            np.ndarray: Preprocessed spectrum
+        """
+        # Apply resampling, baseline correction, and smoothing
+        # These don't cause data leakage as they're applied per-sample
+        _, y_processed = preprocess_spectrum(
+            np.asarray(x_raw),
+            np.asarray(y_raw),
+            target_len=self.target_len,
+            modality=self.modality,
+            do_baseline=self.do_baseline,
+            do_smooth=self.do_smooth,
+            do_normalize=False,  # We handle normalization separately
+            out_dtype=np.float32
+        )
+
+        # Apply normalization using fitted stats if available
+        if self.do_normalize and use_fitted_stats and self.is_fitted:
+            y_processed = self._apply_fitted_normalization(y_processed)
+        elif self.do_normalize and not use_fitted_stats:
+            # Apply per-sample normalization (min-max)
+            y_min, y_max = y_processed.min(), y_processed.max()
+            if y_max > y_min:
+                y_processed = (y_processed - y_min) / (y_max - y_min)
+
+        return y_processed
+
+    def fit_normalization_stats(self, train_spectra: list) -> None:
+        """
+        Fit normalization statistics on training data only.
+
+        Args:
+            train_spectra (list): List of (x_raw, y_raw) tuples for training
+        """
+        if not self.do_normalize:
+            return
+
+        # Preprocess training spectra without normalization
+        processed_spectra = []
+        for x_raw, y_raw in train_spectra:
+            y_processed = self.preprocess_single_spectrum(
+                x_raw, y_raw, use_fitted_stats=False
+            )
+            processed_spectra.append(y_processed)
+
+        # Calculate global statistics from training data
+        all_values = np.concatenate(processed_spectra)
+        self.normalization_stats = {
+            'mean': np.mean(all_values),
+            'std': np.std(all_values),
+            'min': np.min(all_values),
+            'max': np.max(all_values)
+        }
+
+        self.is_fitted = True
+        print("✅ Fitted normalization statistics on training data")
+
+    def _apply_fitted_normalization(self, spectrum: np.ndarray) -> np.ndarray:
+        """
+        Apply fitted normalization to a spectrum.
+
+        Args:
+            spectrum (np.ndarray): Preprocessed spectrum
+
+        Returns:
+            np.ndarray: Normalized spectrum
+        """
+        if not self.is_fitted:
+            raise ValueError("Normalization stats not fitted. Call fit_normalization_stats first.")
+
+        # Use min-max normalization based on training data
+        stats = self.normalization_stats
+        if stats is not None and stats['max'] > stats['min']:
+            spectrum = (spectrum - stats['min']) / (stats['max'] - stats['min'])
+
+        return spectrum
+
+    def transform_fold(
+        self,
+        raw_spectra: np.ndarray,
+        train_indices: np.ndarray,
+        val_indices: np.ndarray
+    ) -> Tuple[np.ndarray, np.ndarray]:
+        """
+        Transform data for a single CV fold without data leakage.
+
+        Args:
+            raw_spectra (np.ndarray): Array of (x_raw, y_raw) tuples
+            train_indices (np.ndarray): Training indices for this fold
+            val_indices (np.ndarray): Validation indices for this fold
+
+        Returns:
+            tuple: (X_train, X_val) preprocessed data
+        """
+        # Get training and validation raw data
+        train_raw = raw_spectra[train_indices]
+        val_raw = raw_spectra[val_indices]
+
+        # Fit normalization stats on training data only
+        self.fit_normalization_stats(train_raw.tolist())
+
+        # Preprocess training data
+        X_train = []
+        for x_raw, y_raw in train_raw:
+            processed = self.preprocess_single_spectrum(
+                x_raw, y_raw, use_fitted_stats=True
+            )
+            X_train.append(processed)
+
+        # Preprocess validation data using fitted stats
+        X_val = []
+        for x_raw, y_raw in val_raw:
+            processed = self.preprocess_single_spectrum(
+                x_raw, y_raw, use_fitted_stats=True
+            )
+            X_val.append(processed)
+
+        return np.array(X_train), np.array(X_val)
+
+def load_data_for_cv(
+    dataset_dir: str,
+    preprocessor_config: Optional[Dict[str, Any]] = None
+) -> Tuple[np.ndarray, np.ndarray, SpectrumPreprocessor]:
+    """
+    Load raw data for cross-validation without data leakage.
+
+    Args:
+        dataset_dir (str): Path to dataset directory
+        preprocessor_config (dict): Configuration for preprocessor
+
+    Returns:
+        tuple: (raw_spectra, labels, preprocessor)
+    """
+    config = preprocessor_config or {}
+    preprocessor = SpectrumPreprocessor(**config)
+
+    raw_spectra, labels, _ = preprocessor.load_raw_data(dataset_dir)
+
+    print(f"✅ Loaded {len(raw_spectra)} raw spectra for CV")
+    print(f"Class distribution: {np.bincount(labels)}")
+
+    return raw_spectra, labels, preprocessor
+
+def preprocess_holdout_test_set(
+    test_spectra: np.ndarray,
+    fitted_preprocessor: SpectrumPreprocessor
+) -> np.ndarray:
+    """
+    Preprocess hold-out test set using fitted preprocessor.
+
+    Args:
+        test_spectra (np.ndarray): Raw test spectra
+        fitted_preprocessor (SpectrumPreprocessor): Preprocessor fitted on training data
+
+    Returns:
+        np.ndarray: Preprocessed test data
+    """
+    if not fitted_preprocessor.is_fitted:
+        raise ValueError("Preprocessor must be fitted on training data first")
+
+    X_test = []
+    for x_raw, y_raw in test_spectra:
+        processed = fitted_preprocessor.preprocess_single_spectrum(
+            x_raw, y_raw, use_fitted_stats=True
+        )
+        X_test.append(processed)
+
+    return np.array(X_test)
+
+if __name__ == "__main__":
+    # Test the data leakage-free preprocessing pipeline
+    print("Testing data leakage-free preprocessing pipeline...")
+
+    # Test with sample data
+    dataset_dir = "sample_data"
+
+    # Load raw data
+    raw_spectra, labels, preprocessor = load_data_for_cv(dataset_dir)
+
+    # Simulate a single CV fold
+    from sklearn.model_selection import StratifiedKFold
+
+    if len(raw_spectra) >= 2:
+        cv = StratifiedKFold(n_splits=2, shuffle=True, random_state=42)
+        train_idx, val_idx = next(cv.split(raw_spectra, labels))
+
+        # Transform without data leakage
+        X_train, X_val = preprocessor.transform_fold(raw_spectra, train_idx, val_idx)
+
+        print("✅ Fold transformation completed")
+        print(f"   Train: {X_train.shape}")
+        print(f"   Val: {X_val.shape}")
+        print(f"   Normalization fitted: {preprocessor.is_fitted}")
+
+    print("✅ Data leakage-free preprocessing test completed!")

backend/utils/raman_util.py

@@ -0,0 +1,46 @@
+from typing import List, Dict
+import os
+from pathlib import Path
+import matplotlib.pyplot as plt
+
+import json
+
+def list_txt_files(root_dir):
+    """Recursively lists all .txt files in a directory."""
+    txt_files = []
+    for dirpath, _, filenames in os.walk(root_dir):
+        for file in filenames:
+            if file.endswith(".txt"):
+                full_path = os.path.join(dirpath, file)
+                txt_files.append(full_path)
+    return txt_files
+
+def label_file(filepath):
+    """
+    Assigns label based on filename prefix:
+    - 'sta-' => 0 (pristine)
+    - 'wea-' => 1 (weathered)
+    Returns None if prefix is unknown.
+    """
+    filename = os.path.basename(filepath).lower()
+    if filename.startswith("sta-"):
+        return 0
+    elif filename.startswith("wea-"):
+        return 1
+    else:
+        return None  # Unknown or irrelevant
+
+def load_spectrum(filepath):
+    """Loads a Raman spectrum from a two-column .txt file."""
+    x_vals, y_vals = [], []
+    with open(filepath, 'r', encoding='utf-8') as file:
+        for line in file:
+            parts = line.strip().split()
+            if len(parts) == 2:
+                try:
+                    x, y = float(parts[0]), float(parts[1])
+                    x_vals.append(x)
+                    y_vals.append(y)
+                except ValueError:
+                    continue  # Skip lines that can't be converted
+    return x_vals, y_vals

backend/utils/seeds.py

@@ -0,0 +1,157 @@
+# pylint: disable=missing-function-docstring, missing-class-docstring, missing-module-docstring, redefined-outer-name, unused-argument, unused-import, singleton-comparison, broad-except
+"""
+seeds.py
+
+Universal reproducibility controls for the polymer aging ML pipeline.
+Provides centralized seed management to ensure consistent results across
+all random operations in training, validation, and inference.
+
+* NOTE: This module should be imported and used at the start of any script
+*       involving randomness to guarantee reproducible results.
+"""
+
+import os
+import random
+import numpy as np
+import torch
+
+
+def set_global_seeds(seed: int = 42):
+    """
+    Set random seeds for all major libraries to ensure reproducibility.
+
+    Args:
+        seed (int): Random seed value to use across all libraries
+
+    Note:
+        This function should be called at the beginning of any script
+        that involves random operations (training, data splitting, etc.)
+    """
+    # Python built-in random
+    random.seed(seed)
+
+    # NumPy random
+    np.random.seed(seed)
+
+    # PyTorch random
+    torch.manual_seed(seed)
+
+    # PyTorch CUDA random (if available)
+    if torch.cuda.is_available():
+        torch.cuda.manual_seed(seed)
+        torch.cuda.manual_seed_all(seed)
+
+        # Additional CUDA reproducibility settings
+        torch.backends.cudnn.deterministic = True
+        torch.backends.cudnn.benchmark = False
+
+    # Set environment variable for Python hash randomization
+    os.environ['PYTHONHASHSEED'] = str(seed)
+
+    print(f"✅ Global seeds set to {seed} for reproducibility")
+
+
+def get_default_seed():
+    """
+    Get the default seed value used across the project.
+
+    Returns:
+        int: Default seed value (42)
+    """
+    return 42
+
+
+def create_fold_seeds(base_seed: int = 42, num_folds: int = 10):
+    """
+    Create deterministic seeds for cross-validation folds.
+
+    Args:
+        base_seed (int): Base seed for generating fold seeds
+        num_folds (int): Number of CV folds
+
+    Returns:
+        list: List of unique seeds for each fold
+    """
+    # Use base seed to create deterministic but unique seeds for each fold
+    np.random.seed(base_seed)
+    fold_seeds = np.random.randint(0, 2**31-1, size=num_folds)
+    return fold_seeds.tolist()
+
+
+def create_augmentation_seed(base_seed: int = 42, fold: int = 0):
+    """
+    Create a deterministic seed for data augmentation within a specific fold.
+
+    Args:
+        base_seed (int): Base seed
+        fold (int): Current fold number
+
+    Returns:
+        int: Deterministic seed for augmentation in this fold
+    """
+    return base_seed + 1000 + fold
+
+
+def verify_reproducibility():
+    """
+    Verify that random operations are reproducible after setting seeds.
+
+    Returns:
+        bool: True if reproducibility check passes
+    """
+    # Test Python random
+    set_global_seeds(42)
+    python_rand_1 = random.random()
+
+    set_global_seeds(42)
+    python_rand_2 = random.random()
+
+    # Test NumPy random
+    set_global_seeds(42)
+    numpy_rand_1 = np.random.random()
+
+    set_global_seeds(42)
+    numpy_rand_2 = np.random.random()
+
+    # Test PyTorch random
+    set_global_seeds(42)
+    torch_rand_1 = torch.rand(1).item()
+
+    set_global_seeds(42)
+    torch_rand_2 = torch.rand(1).item()
+
+    # Check if all are reproducible
+    python_reproducible = python_rand_1 == python_rand_2
+    numpy_reproducible = numpy_rand_1 == numpy_rand_2
+    torch_reproducible = torch_rand_1 == torch_rand_2
+
+    all_reproducible = python_reproducible and numpy_reproducible and torch_reproducible
+
+    if all_reproducible:
+        print("✅ Reproducibility verification passed")
+    else:
+        print("❌ Reproducibility verification failed")
+        print(f"   Python: {python_reproducible}")
+        print(f"   NumPy: {numpy_reproducible}")
+        print(f"   PyTorch: {torch_reproducible}")
+
+    return all_reproducible
+
+
+if __name__ == "__main__":
+    print("🧪 Testing reproducibility controls...")
+    # Test seed setting
+    set_global_seeds(42)
+
+    # Test fold seed generation
+    fold_seeds = create_fold_seeds(42, 10)
+    print(f"📊 Generated fold seeds: {fold_seeds}")
+
+    # Test augmentation seed generation
+    aug_seeds = [create_augmentation_seed(42, i) for i in range(5)]
+    print(f"📊 Generated augmentation seeds: {aug_seeds}")
+
+    # Verify reproducibility
+    verify_reproducibility()
+
+    print("✅ Reproducibility controls test completed!")

backend/utils/spectrum_preprocessor.py

@@ -0,0 +1,242 @@
+"""
+Centralized Spectrum Preprocessing Module
+Single source of truth for all preprocessing operations across training, validation, testing, and live inference.
+Ensures no drift between different processing stages.
+"""
+
+import hashlib
+import json
+import numpy as np
+from pathlib import Path
+from typing import Dict, Any, Tuple, Optional, List
+from dataclasses import dataclass, asdict
+from datetime import datetime
+
+from backend.utils.preprocessing import (
+    preprocess_spectrum,
+    validate_spectrum_modality,
+    MODALITY_PARAMS,
+    MODALITY_RANGES,
+    TARGET_LENGTH
+)
+
+
+@dataclass
+class PreprocessingConfig:
+    """Immutable preprocessing configuration."""
+    target_length: int = TARGET_LENGTH
+    modality: str = "raman"
+    do_baseline: bool = True
+    baseline_degree: Optional[int] = None  # Uses modality default if None
+    do_smooth: bool = True
+    smooth_window: Optional[int] = None  # Uses modality default if None
+    smooth_polyorder: Optional[int] = None  # Uses modality default if None
+    do_normalize: bool = True
+    validate_range: bool = True
+    version: str = "1.0.0"  # Version for config compatibility
+
+    def __post_init__(self):
+        """Validate configuration and set modality defaults."""
+        if self.modality not in MODALITY_PARAMS:
+            raise ValueError(f"Invalid modality: {self.modality}")
+
+        # Set modality defaults if None
+        modality_config = MODALITY_PARAMS[self.modality]
+        if self.baseline_degree is None:
+            object.__setattr__(self, 'baseline_degree', modality_config['baseline_degree'])
+        if self.smooth_window is None:
+            object.__setattr__(self, 'smooth_window', modality_config['smooth_window'])
+        if self.smooth_polyorder is None:
+            object.__setattr__(self, 'smooth_polyorder', modality_config['smooth_polyorder'])
+
+    def to_dict(self) -> Dict[str, Any]:
+        """Convert to dictionary for serialization."""
+        return asdict(self)
+
+    def get_hash(self) -> str:
+        """Get deterministic hash of configuration."""
+        config_str = json.dumps(self.to_dict(), sort_keys=True)
+        return hashlib.sha256(config_str.encode()).hexdigest()[:16]
+
+
+@dataclass
+class PreprocessingResult:
+    """Result of preprocessing with full provenance."""
+    x_processed: np.ndarray
+    y_processed: np.ndarray
+    x_original: np.ndarray
+    y_original: np.ndarray
+    config: PreprocessingConfig
+    metadata: Dict[str, Any]
+    processing_time: float
+    timestamp: str
+
+    def get_content_hash(self) -> str:
+        """Get hash of processed content for drift detection."""
+        # Combine config hash with data hash
+        config_hash = self.config.get_hash()
+        data_hash = hashlib.sha256(
+            np.concatenate([self.x_processed, self.y_processed]).tobytes()
+        ).hexdigest()[:16]
+        return f"{config_hash}_{data_hash}"
+
+
+class SpectrumPreprocessor:
+    """
+    Centralized spectrum preprocessor ensuring consistent processing across all stages.
+    Single source of truth for preprocessing logic.
+    """
+
+    def __init__(self, config: Optional[PreprocessingConfig] = None):
+        """Initialize with preprocessing configuration."""
+        self.config = config or PreprocessingConfig()
+        self._processing_history: List[Dict[str, Any]] = []
+
+    def process(
+        self,
+        x: np.ndarray,
+        y: np.ndarray,
+        metadata: Optional[Dict[str, Any]] = None
+    ) -> PreprocessingResult:
+        """
+        Process spectrum with full provenance tracking.
+
+        Args:
+            x: Input wavenumber array
+            y: Input intensity array
+            metadata: Optional metadata to include
+
+        Returns:
+            PreprocessingResult with full provenance
+        """
+        import time
+        start_time = time.time()
+
+        # Store original data
+        x_original = np.array(x, copy=True)
+        y_original = np.array(y, copy=True)
+
+        # Process using centralized function
+        x_processed, y_processed = preprocess_spectrum(
+            x, y,
+            target_len=self.config.target_length,
+            modality=self.config.modality,
+            do_baseline=self.config.do_baseline,
+            degree=self.config.baseline_degree,
+            do_smooth=self.config.do_smooth,
+            window_length=self.config.smooth_window,
+            polyorder=self.config.smooth_polyorder,
+            do_normalize=self.config.do_normalize,
+            validate_range=self.config.validate_range
+        )
+
+        processing_time = time.time() - start_time
+
+        # Create metadata
+        result_metadata = {
+            "original_length": len(x_original),
+            "processed_length": len(x_processed),
+            "wavenumber_range": [float(x_processed.min()), float(x_processed.max())],
+            "intensity_range": [float(y_processed.min()), float(y_processed.max())],
+            "modality_validated": validate_spectrum_modality(x_original, y_original, self.config.modality)[0],
+            **(metadata or {})
+        }
+
+        # Create result
+        result = PreprocessingResult(
+            x_processed=x_processed,
+            y_processed=y_processed,
+            x_original=x_original,
+            y_original=y_original,
+            config=self.config,
+            metadata=result_metadata,
+            processing_time=processing_time,
+            timestamp=datetime.now().isoformat()
+        )
+
+        # Track processing
+        self._processing_history.append({
+            "timestamp": result.timestamp,
+            "config_hash": self.config.get_hash(),
+            "content_hash": result.get_content_hash(),
+            "processing_time": processing_time
+        })
+
+        return result
+
+    def process_batch(
+        self,
+        spectra: List[Tuple[np.ndarray, np.ndarray]],
+        metadata_list: Optional[List[Dict[str, Any]]] = None
+    ) -> List[PreprocessingResult]:
+        """Process multiple spectra with consistent configuration."""
+        if metadata_list is None:
+            metadata_list = [None] * len(spectra)
+
+        results = []
+        for i, (x, y) in enumerate(spectra):
+            metadata = metadata_list[i] if i < len(metadata_list) else None
+            result = self.process(x, y, metadata)
+            results.append(result)
+
+        return results
+
+    def get_processing_summary(self) -> Dict[str, Any]:
+        """Get summary of all processing operations."""
+        if not self._processing_history:
+            return {"total_processed": 0}
+
+        return {
+            "total_processed": len(self._processing_history),
+            "config_hash": self.config.get_hash(),
+            "config": self.config.to_dict(),
+            "processing_times": {
+                "min": min(h["processing_time"] for h in self._processing_history),
+                "max": max(h["processing_time"] for h in self._processing_history),
+                "mean": np.mean([h["processing_time"] for h in self._processing_history])
+            },
+            "first_processed": self._processing_history[0]["timestamp"],
+            "last_processed": self._processing_history[-1]["timestamp"]
+        }
+
+
+# Global preprocessor instances for different stages
+TRAINING_PREPROCESSOR = SpectrumPreprocessor(PreprocessingConfig(modality="raman"))
+VALIDATION_PREPROCESSOR = SpectrumPreprocessor(PreprocessingConfig(modality="raman"))
+INFERENCE_PREPROCESSOR = SpectrumPreprocessor(PreprocessingConfig(modality="raman"))
+
+# Factory function for creating stage-specific preprocessors
+def create_preprocessor(stage: str, modality: str = "raman") -> SpectrumPreprocessor:
+    """
+    Create preprocessor for specific stage with identical configuration.
+
+    Args:
+        stage: 'training', 'validation', 'testing', or 'inference'
+        modality: 'raman' or 'ftir'
+
+    Returns:
+        SpectrumPreprocessor configured for the stage
+    """
+    config = PreprocessingConfig(modality=modality)
+    return SpectrumPreprocessor(config)
+
+
+# Utility functions for external compatibility
+def preprocess_for_inference(x: np.ndarray, y: np.ndarray, modality: str = "raman") -> Tuple[np.ndarray, np.ndarray]:
+    """
+    Process spectrum for inference using centralized preprocessor.
+    Maintains compatibility with existing code.
+    """
+    preprocessor = create_preprocessor("inference", modality)
+    result = preprocessor.process(x, y)
+    return result.x_processed, result.y_processed
+
+
+def preprocess_for_training(x: np.ndarray, y: np.ndarray, modality: str = "raman") -> Tuple[np.ndarray, np.ndarray]:
+    """
+    Process spectrum for training using centralized preprocessor.
+    Maintains compatibility with existing code.
+    """
+    preprocessor = create_preprocessor("training", modality)
+    result = preprocessor.process(x, y)
+    return result.x_processed, result.y_processed

backend/utils/train.py

@@ -0,0 +1,163 @@
+"""
+Main Training Script
+
+This script orchestrates the model training process. It is configuration-driven
+and uses MLflow for experiment tracking.
+
+Usage:
+    python scripts/train.py --config-path configs/base_config.yaml
+"""
+
+from pathlib import Path
+import sys
+import argparse
+import yaml
+from typing import Dict, Optional, Any
+
+import pandas as pd
+import torch
+import mlflow
+from torch.utils.data import DataLoader, TensorDataset
+from tqdm import tqdm
+
+# Ensure the backend is in the path to import registry and preprocessing
+sys.path.append(str(Path(__file__).resolve().parents[1]))
+
+from config import TARGET_LEN
+from backend.utils.preprocessing import preprocess_spectrum
+from models.registry import build
+
+
+def load_data(data_path: Path, target_len: int):
+    """Load and preprocess data from a CSV file."""
+    df = pd.read_csv(data_path)
+
+    # This is a placeholder for your actual data loading.
+    # You need to parse your 'spectra' column into x and y values.
+    # For this example, we assume 'y_values' are stored as a string of numbers.
+    # A more robust solution would use np.load or similar if data is saved in binary format.
+
+    all_y = []
+    # This loop is inefficient and for demonstration only. Vectorize in production.
+    for _, row in tqdm(df.iterrows(), total=len(df), desc=f"Processing {data_path.name}"):
+        # Dummy x_values, as preprocess_spectrum primarily uses y_values
+        x_values = range(len(row['spectrum'].split()))
+        y_values = [float(y) for y in row['spectrum'].split()]
+        _, y_processed = preprocess_spectrum(
+            x_values, y_values, modality='raman')
+        all_y.append(y_processed)
+
+    features = torch.tensor(all_y, dtype=torch.float32).unsqueeze(1)
+    labels = torch.tensor(df['label'].values, dtype=torch.long)
+
+    return TensorDataset(features, labels)
+
+
+def train(config: dict, jobs_db: Optional[Dict[str, Any]] = None, job_id: Optional[str] = None):
+    """Main training and validation loop."""
+    try:
+        # --- MLflow Setup ---
+        mlflow.set_experiment(config['experiment_name'])
+        with mlflow.start_run(run_name=config.get('run_name', 'default_run')) as run:
+            mlflow.log_params(config)
+            if jobs_db and job_id:
+                jobs_db[job_id]['mlflow_run_id'] = run.info.run_id
+                jobs_db[job_id]['status'] = 'RUNNING'
+            print(f"MLflow Run ID: {run.info.run_id}")
+
+            # --- Data Loading ---
+            data_dir = Path(config['data_dir'])
+            train_dataset = load_data(data_dir / config['train_csv'], TARGET_LEN)
+            val_dataset = load_data(data_dir / config['val_csv'], TARGET_LEN)
+
+            train_loader = DataLoader(
+                train_dataset, batch_size=config['batch_size'], shuffle=True)
+            val_loader = DataLoader(val_dataset, batch_size=config['batch_size'])
+
+            # --- Model, Optimizer, Loss ---
+            device = "cuda" if torch.cuda.is_available() else "cpu"
+            print(f"Using device: {device}")
+
+            model = build(config['model_name'], TARGET_LEN).to(device)
+            optimizer = getattr(torch.optim, config['optimizer'])(
+                model.parameters(), lr=config['learning_rate'])
+            criterion = getattr(torch.nn, config['loss_function'])()
+
+            # --- Training Loop ---
+            best_val_loss = float('inf')
+            for epoch in range(config['epochs']):
+                model.train()
+                train_loss = 0.0
+                for features, labels in tqdm(train_loader, desc=f"Epoch {epoch+1}/{config['epochs']} [Train]"):
+                    features, labels = features.to(device), labels.to(device)
+
+                    optimizer.zero_grad()
+                    outputs = model(features)
+                    loss = criterion(outputs, labels)
+                    loss.backward()
+                    optimizer.step()
+                    train_loss += loss.item()
+
+                avg_train_loss = train_loss / len(train_loader)
+                mlflow.log_metric("train_loss", avg_train_loss, step=epoch)
+
+                # --- Validation Loop ---
+                model.eval()
+                val_loss = 0.0
+                with torch.no_grad():
+                    for features, labels in tqdm(val_loader, desc=f"Epoch {epoch+1}/{config['epochs']} [Val]"):
+                        features, labels = features.to(device), labels.to(device)
+                        outputs = model(features)
+                        loss = criterion(outputs, labels)
+                        val_loss += loss.item()
+
+                avg_val_loss = val_loss / len(val_loader)
+                mlflow.log_metric("val_loss", avg_val_loss, step=epoch)
+                print(
+                    f"Epoch {epoch+1}: Train Loss: {avg_train_loss:.4f}, Val Loss: {avg_val_loss:.4f}")
+
+                # --- Progress Update for Web UI ---
+                if jobs_db and job_id:
+                    progress = (epoch + 1) / config['epochs']
+                    jobs_db[job_id]['progress'] = progress
+                    jobs_db[job_id]['metrics']['train_loss'].append(avg_train_loss)
+                    jobs_db[job_id]['metrics']['val_loss'].append(avg_val_loss)
+                    jobs_db[job_id]['current_epoch'] = epoch + 1
+
+                # --- Save Best Model ---
+                if avg_val_loss < best_val_loss:
+                    best_val_loss = avg_val_loss
+                    mlflow.pytorch.log_model(
+                        model, "model", registered_model_name=f"{config.get('run_name', 'default_run')}_best")
+                    print(
+                        f"New best model saved at epoch {epoch+1} with validation loss: {best_val_loss:.4f}")
+
+        if jobs_db and job_id:
+            jobs_db[job_id]['status'] = 'COMPLETED'
+            jobs_db[job_id]['progress'] = 1.0
+        print("✅ Training complete.")
+
+    except Exception as e:
+        print(f"❌ Training failed: {e}")
+        if jobs_db and job_id:
+            jobs_db[job_id]['status'] = 'FAILED'
+            jobs_db[job_id]['error'] = str(e)
+        raise
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(
+        description="Train a spectral classification model.")
+    parser.add_argument(
+        "--config-path",
+        type=Path,
+        required=True,
+        help="Path to the YAML configuration file."
+    )
+    args = parser.parse_args()
+
+    with open(args.config_path, 'r', encoding='utf-8') as f:
+        config = yaml.safe_load(f)
+
+    # Run training from CLI without web-specific job tracking
+    train(config=config)

backend/utils/training_engine.py

@@ -0,0 +1,161 @@
+"""
+Core Training Engine for the POLYMEROS project.
+
+This module contains the primary logic for model training and validation,
+encapsulated in a reusable `TrainingEngine` class. It is designed to be
+called by different interfaces, such as the command-line script
+(train_model.py) and the web UI's TrainingManager.
+
+This approach ensures that the core training process is consistent,
+maintainable, and follows the DRY (Don't Repeat Yourself) principle.
+"""
+
+import torch
+import torch.nn as nn
+import numpy as np
+from torch.utils.data import TensorDataset, DataLoader
+from sklearn.metrics import confusion_matrix, accuracy_score
+
+from .training_types import (
+    TrainingConfig,
+    TrainingProgress,
+    get_cv_splitter,
+    augment_spectral_data,
+)
+from backend.registry import build as build_model
+
+
+class TrainingEngine:
+    """Encapsulates the core model training and validation logic."""
+
+    def __init__(self, config: TrainingConfig):
+        """
+        Initializes the TrainingEngine with a given configuration.
+
+        Args:
+            config (TrainingConfig): The configuration object for the training run.
+        """
+        self.config = config
+        self.device = self._get_device()
+
+    def _get_device(self) -> torch.device:
+        """Selects the appropriate compute device."""
+        if self.config.device == "auto":
+            return torch.device("cuda" if torch.cuda.is_available() else "cpu")
+        return torch.device(self.config.device)
+
+    def run(
+        self, X: np.ndarray, y: np.ndarray, progress_callback: callable = None
+    ) -> dict:
+        """
+        Executes the full cross-validation training and evaluation loop.
+        Args:
+            X (np.ndarray): Feature data.
+            y (np.ndarray): Label data.
+            progress_callback (callable, optional):
+                A function to call with
+                progress updates. Defaults to None.
+
+        Returns:
+                dict: A dictionary containing the final
+                results and metrics.
+        """
+        cv_splitter = get_cv_splitter(self.config.cv_strategy, self.config.num_folds)
+
+        fold_accuracies = []
+        all_conf_matrices = []
+        final_model_state = None
+
+        for fold, (train_idx, val_idx) in enumerate(cv_splitter.split(X, y), 1):
+            if progress_callback:
+                progress_callback(
+                    {
+                        "type": "fold_start",
+                        "fold": fold,
+                        "total_folds": self.config.num_folds,
+                    }
+                )
+
+            X_train, X_val = X[train_idx], X[val_idx]
+            y_train, y_val = y[train_idx], y[val_idx]
+
+            # Apply data augmentation if enabled
+            if self.config.enable_augmentation:
+                X_train, y_train = augment_spectral_data(
+                    X_train, y_train, noise_level=self.config.noise_level
+                )
+
+            train_loader = DataLoader(
+                TensorDataset(
+                    torch.tensor(X_train, dtype=torch.float32),
+                    torch.tensor(y_train, dtype=torch.long),
+                ),
+                batch_size=self.config.batch_size,
+                shuffle=True,
+            )
+            val_loader = DataLoader(
+                TensorDataset(
+                    torch.tensor(X_val, dtype=torch.float32),
+                    torch.tensor(y_val, dtype=torch.long),
+                )
+            )
+
+            model = build_model(self.config.model_name, self.config.target_len).to(
+                self.device
+            )
+            optimizer = torch.optim.Adam(
+                model.parameters(), lr=self.config.learning_rate
+            )
+            criterion = nn.CrossEntropyLoss()
+
+            for epoch in range(self.config.epochs):
+                model.train()
+                running_loss = 0.0
+                for inputs, labels in train_loader:
+                    inputs = inputs.unsqueeze(1).to(self.device)
+                    labels = labels.to(self.device)
+
+                    optimizer.zero_grad()
+                    outputs = model(inputs)
+                    loss = criterion(outputs, labels)
+                    loss.backward()
+                    optimizer.step()
+                    running_loss += loss.item()
+
+                if progress_callback:
+                    progress_callback(
+                        {
+                            "type": "epoch_end",
+                            "fold": fold,
+                            "epoch": epoch + 1,
+                            "total_epochs": self.config.epochs,
+                            "loss": running_loss / len(train_loader),
+                        }
+                    )
+
+            # Validation
+            model.eval()
+            all_true, all_pred = [], []
+            with torch.no_grad():
+                for inputs, labels in val_loader:
+                    inputs = inputs.unsqueeze(1).to(self.device)
+                    outputs = model(inputs)
+                    _, predicted = torch.max(outputs, 1)
+                    all_true.extend(labels.cpu().numpy())
+                    all_pred.extend(predicted.cpu().numpy())
+
+            acc = accuracy_score(all_true, all_pred)
+            fold_accuracies.append(acc)
+            all_conf_matrices.append(confusion_matrix(all_true, all_pred).tolist())
+            final_model_state = model.state_dict()
+
+            if progress_callback:
+                progress_callback({"type": "fold_end", "fold": fold, "accuracy": acc})
+
+        return {
+            "fold_accuracies": fold_accuracies,
+            "confusion_matrices": all_conf_matrices,
+            "mean_accuracy": np.mean(fold_accuracies),
+            "std_accuracy": np.std(fold_accuracies),
+            "model_state_dict": final_model_state,
+        }

backend/utils/training_engine_enhanced.py

@@ -0,0 +1,431 @@
+# pylint: disable=missing-function-docstring, missing-class-docstring, missing-module-docstring, redefined-outer-name, unused-argument, unused-import, singleton-comparison, broad-except, invalid-name
+"""
+training_engine_enhanced.py
+
+Enhanced training engine with modern ML practices:
+- L2 Weight Decay (regularization)
+- Early Stopping based on validation loss
+- Learning Rate Scheduling (ReduceLROnPlateau)
+- Data leakage-free preprocessing
+- Comprehensive logging and metrics
+
+* NOTE: This replaces the original training engine to incorporate
+*       best practices for robust model training.
+"""
+
+import os
+import sys
+import torch
+import torch.nn as nn
+import numpy as np
+from torch.utils.data import TensorDataset, DataLoader
+from torch.optim.lr_scheduler import ReduceLROnPlateau
+from sklearn.metrics import confusion_matrix, accuracy_score, classification_report
+from typing import Dict, Any, Optional, Callable
+
+
+from .utils.preprocessing_fixed import SpectrumPreprocessor, load_data_for_cv
+from .utils.seeds import set_global_seeds, create_fold_seeds
+from .training_types import TrainingConfig, get_cv_splitter
+from backend.registry import build as build_model
+
+class EarlyStoppingCallback:
+    """Early stopping callback to prevent overfitting."""
+
+    def __init__(self, patience: int = 7, min_delta: float = 1e-6):
+        self.patience = patience
+        self.min_delta = min_delta
+        self.best_loss = float('inf')
+        self.counter = 0
+        self.early_stop = False
+
+    def __call__(self, val_loss: float) -> bool:
+        """
+        Check if training should stop early.
+
+        Args:
+            val_loss (float): Current validation loss
+
+        Returns:
+            bool: True if training should stop
+        """
+        if val_loss < self.best_loss - self.min_delta:
+            self.best_loss = val_loss
+            self.counter = 0
+        else:
+            self.counter += 1
+            if self.counter >= self.patience:
+                self.early_stop = True
+
+        return self.early_stop
+
+class EnhancedTrainingEngine:
+    """
+    Enhanced training engine with modern ML practices and data leakage prevention.
+    """
+
+    def __init__(self, config: TrainingConfig):
+        """
+        Initialize the enhanced training engine.
+
+        Args:
+            config (TrainingConfig): Training configuration
+        """
+        self.config = config
+        self.device = self._get_device()
+
+        # Enhanced training parameters
+        self.weight_decay = getattr(config, 'weight_decay', 1e-4)
+        self.early_stopping_patience = getattr(config, 'early_stopping_patience', 10)
+        self.lr_scheduler_patience = getattr(config, 'lr_scheduler_patience', 5)
+        self.lr_scheduler_factor = getattr(config, 'lr_scheduler_factor', 0.5)
+        self.min_lr = getattr(config, 'min_lr', 1e-6)
+
+        print("Enhanced Training Engine initialized")
+        print(f"   Device: {self.device}")
+        print(f"   Weight Decay: {self.weight_decay}")
+        print(f"   Early Stopping Patience: {self.early_stopping_patience}")
+        print(f"   LR Scheduler Patience: {self.lr_scheduler_patience}")
+
+    def _get_device(self) -> torch.device:
+        """Select the appropriate compute device."""
+        if self.config.device == "auto":
+            return torch.device("cuda" if torch.cuda.is_available() else "cpu")
+        return torch.device(self.config.device)
+
+    def run(
+        self,
+        dataset_dir: str,
+        progress_callback: Optional[Callable] = None
+    ) -> Dict[str, Any]:
+        """
+        Run the complete training pipeline with data leakage prevention.
+
+        Args:
+            dataset_dir (str): Path to dataset directory
+            progress_callback (callable): Optional progress callback
+
+        Returns:
+            dict: Complete training results and metrics
+        """
+        print("Starting enhanced training pipeline...")
+
+        # Set global seeds for reproducibility
+        set_global_seeds(getattr(self.config, 'random_state', 42))
+
+        # Load raw data without preprocessing
+        preprocessor_config = {
+            'target_len': self.config.target_len,
+            'do_baseline': getattr(self.config, 'baseline_correction', True),
+            'do_smooth': getattr(self.config, 'smoothing', True),
+            'do_normalize': getattr(self.config, 'normalization', True),
+            'modality': getattr(self.config, 'modality', 'raman')
+        }
+
+        raw_spectra, labels, preprocessor = load_data_for_cv(
+            dataset_dir, preprocessor_config
+        )
+
+        # Initialize cross-validation
+        cv_splitter = get_cv_splitter(
+            getattr(self.config, 'cv_strategy', 'stratified_kfold'),
+            self.config.num_folds
+        )
+
+        # Generate fold-specific seeds
+        fold_seeds = create_fold_seeds(
+            getattr(self.config, 'random_state', 42),
+            self.config.num_folds
+        )
+
+        # Results storage
+        fold_results = []
+        all_conf_matrices = []
+
+        for fold, (train_idx, val_idx) in enumerate(cv_splitter.split(raw_spectra, labels), 1):
+            print(f"\nTraining Fold {fold}/{self.config.num_folds}")
+
+            # Set fold-specific seed
+            set_global_seeds(fold_seeds[fold - 1])
+
+            if progress_callback:
+                progress_callback({
+                    "type": "fold_start",
+                    "fold": fold,
+                    "total_folds": self.config.num_folds
+                })
+
+            # Preprocess data for this fold (no data leakage)
+            X_train, X_val = preprocessor.transform_fold(raw_spectra, train_idx, val_idx)
+            y_train, y_val = labels[train_idx], labels[val_idx]
+
+            print(f"   Train: {X_train.shape}, Val: {X_val.shape}")
+
+            # Train model for this fold
+            fold_result = self._train_single_fold(
+                X_train, X_val, y_train, y_val,
+                fold, progress_callback
+            )
+
+            fold_results.append(fold_result)
+            all_conf_matrices.append(fold_result['confusion_matrix'])
+
+            print(f"Fold {fold} completed - Accuracy: {fold_result['accuracy']:.4f}")
+
+        # Aggregate results
+        final_results = self._aggregate_results(fold_results, all_conf_matrices)
+
+        print("\nTraining completed!")
+        print(f"   Mean Accuracy: {final_results['mean_accuracy']:.4f} ± {final_results['std_accuracy']:.4f}")
+        print(f"   Best Fold: {final_results['best_fold']} ({final_results['best_accuracy']:.4f})")
+
+        return final_results
+
+    def _train_single_fold(
+        self,
+        X_train: np.ndarray,
+        X_val: np.ndarray,
+        y_train: np.ndarray,
+        y_val: np.ndarray,
+        fold: int,
+        progress_callback: Optional[Callable] = None
+    ) -> Dict[str, Any]:
+        """
+        Train a model for a single fold with enhanced techniques.
+
+        Args:
+            X_train, X_val, y_train, y_val: Training and validation data
+            fold (int): Current fold number
+            progress_callback (callable): Optional progress callback
+
+        Returns:
+            dict: Results for this fold
+        """
+        # Create data loaders
+        train_loader = DataLoader(
+            TensorDataset(
+                torch.tensor(X_train, dtype=torch.float32),
+                torch.tensor(y_train, dtype=torch.long)
+            ),
+            batch_size=self.config.batch_size,
+            shuffle=True
+        )
+
+        val_loader = DataLoader(
+            TensorDataset(
+                torch.tensor(X_val, dtype=torch.float32),
+                torch.tensor(y_val, dtype=torch.long)
+            ),
+            batch_size=self.config.batch_size,
+            shuffle=False
+        )
+
+        # Initialize model
+        model = build_model(self.config.model_name, self.config.target_len)
+        if not isinstance(model, torch.nn.Module):
+            raise TypeError(f"Expected a PyTorch model, but got {type(model)}")
+        model = model.to(self.device)
+
+        # Enhanced optimizer with weight decay (L2 regularization)
+        optimizer = torch.optim.Adam(
+            model.parameters(),
+            lr=self.config.learning_rate,
+            weight_decay=self.weight_decay
+        )
+
+        # Learning rate scheduler
+        scheduler = ReduceLROnPlateau(
+            optimizer,
+            mode='min',
+            factor=self.lr_scheduler_factor,
+            patience=self.lr_scheduler_patience,
+            min_lr=self.min_lr,
+            verbose='True'
+        )
+
+        # Early stopping
+        early_stopping = EarlyStoppingCallback(patience=self.early_stopping_patience)
+
+        # Loss function
+        criterion = nn.CrossEntropyLoss()
+
+        # Training loop
+        train_losses = []
+        val_losses = []
+        val_accuracies = []
+
+        best_val_loss = float('inf')
+        best_model_state = None
+        epochs_trained = 0
+
+        for epoch in range(self.config.epochs):
+            # Training phase
+            model.train()
+            train_loss = 0.0
+
+            for inputs, labels_batch in train_loader:
+                inputs = inputs.unsqueeze(1).to(self.device)
+                labels_batch = labels_batch.to(self.device)
+
+                optimizer.zero_grad()
+                outputs = model(inputs)
+                loss = criterion(outputs, labels_batch)
+                loss.backward()
+                optimizer.step()
+
+                train_loss += loss.item()
+
+            avg_train_loss = train_loss / len(train_loader)
+            train_losses.append(avg_train_loss)
+
+            # Validation phase
+            model.eval()
+            val_loss = 0.0
+            val_correct = 0
+            val_total = 0
+
+            with torch.no_grad():
+                for inputs, labels_batch in val_loader:
+                    inputs = inputs.unsqueeze(1).to(self.device)
+                    labels_batch = labels_batch.to(self.device)
+
+                    outputs = model(inputs)
+                    loss = criterion(outputs, labels_batch)
+                    val_loss += loss.item()
+
+                    _, predicted = torch.max(outputs, 1)
+                    val_total += labels_batch.size(0)
+                    val_correct += (predicted == labels_batch).sum().item()
+
+            avg_val_loss = val_loss / len(val_loader)
+            val_accuracy = val_correct / val_total
+
+            val_losses.append(avg_val_loss)
+            val_accuracies.append(val_accuracy)
+
+            # Learning rate scheduling
+            scheduler.step(avg_val_loss)
+
+            # Save best model
+            if avg_val_loss < best_val_loss:
+                best_val_loss = avg_val_loss
+                best_model_state = model.state_dict().copy()
+
+            # Progress callback
+            if progress_callback:
+                progress_callback({
+                    "type": "epoch_end",
+                    "fold": fold,
+                    "epoch": epoch + 1,
+                    "total_epochs": self.config.epochs,
+                    "train_loss": avg_train_loss,
+                    "val_loss": avg_val_loss,
+                    "val_accuracy": val_accuracy
+                })
+
+            # Early stopping check
+            if early_stopping(avg_val_loss):
+                print(f"   Early stopping at epoch {epoch + 1}")
+                epochs_trained = epoch + 1
+                break
+
+            epochs_trained = epoch + 1
+
+        # Load best model and evaluate
+        if best_model_state is not None:
+            model.load_state_dict(best_model_state)
+
+        # Final evaluation
+        model.eval()
+        all_true = []
+        all_pred = []
+
+        with torch.no_grad():
+            for inputs, labels_batch in val_loader:
+                inputs = inputs.unsqueeze(1).to(self.device)
+                outputs = model(inputs)
+                _, predicted = torch.max(outputs, 1)
+
+                all_true.extend(labels_batch.cpu().numpy())
+                all_pred.extend(predicted.cpu().numpy())
+
+        # Calculate metrics
+        accuracy = accuracy_score(all_true, all_pred)
+        conf_matrix = confusion_matrix(all_true, all_pred)
+
+        return {
+            'fold': fold,
+            'accuracy': accuracy,
+            'confusion_matrix': conf_matrix.tolist(),
+            'train_losses': train_losses,
+            'val_losses': val_losses,
+            'val_accuracies': val_accuracies,
+            'epochs_trained': epochs_trained,
+            'best_val_loss': best_val_loss,
+            'model_state': best_model_state
+        }
+
+    def _aggregate_results(
+        self,
+        fold_results: list,
+        all_conf_matrices: list
+    ) -> Dict[str, Any]:
+        """
+        Aggregate results across all folds.
+
+        Args:
+            fold_results (list): Results from each fold
+            all_conf_matrices (list): Confusion matrices from each fold
+
+        Returns:
+            dict: Aggregated results
+        """
+        accuracies = [result['accuracy'] for result in fold_results]
+
+        # Find best fold
+        best_fold_idx = np.argmax(accuracies)
+        best_fold = fold_results[best_fold_idx]
+
+        return {
+            'fold_results': fold_results,
+            'accuracies': accuracies,
+            'mean_accuracy': float(np.mean(accuracies)),
+            'std_accuracy': float(np.std(accuracies)),
+            'best_fold': best_fold['fold'],
+            'best_accuracy': float(best_fold['accuracy']),
+            'best_model_state': best_fold['model_state'],
+            'confusion_matrices': all_conf_matrices,
+            'config': self.config.__dict__ if hasattr(self.config, '__dict__') else str(self.config)
+        }
+
+
+if __name__ == "__main__":
+    # Test the enhanced training engine
+    print("Testing Enhanced Training Engine...")
+
+    # Create a minimal config for testing
+    class TestConfig(TrainingConfig):
+        model_name = "figure2"
+        target_len = 500
+        batch_size = 16
+        epochs = 2  # Short for testing
+        learning_rate = 1e-3
+        num_folds = 2  # Small for testing
+        device = "cpu"
+        weight_decay = 1e-4
+        early_stopping_patience = 5
+
+    config = TestConfig(
+        model_name="figure2",
+        dataset_path="sample_data"
+    )
+    engine = EnhancedTrainingEngine(config)
+
+    # Test with sample data (will work even with small dataset)
+    try:
+        results = engine.run("sample_data")
+        print("✅ Enhanced training engine test completed!")
+        print(f"   Results keys: {list(results.keys())}")
+    except Exception as e:
+        print(f"⚠️ Test failed (expected with minimal data): {e}")
+        print("✅ Enhanced training engine structure validated")

backend/utils/training_manager.py

@@ -0,0 +1,638 @@
+"""
+Training job management system for ML Hub functionality.
+Handles asynchronous training jobs, progress tracking, and result management.
+"""
+
+import os
+import sys
+import json
+import time
+import uuid
+import threading
+import concurrent.futures
+import multiprocessing
+from datetime import datetime, timedelta
+from typing import Dict, List, Optional, Callable, Any, Tuple
+from pathlib import Path
+from dataclasses import dataclass, field
+
+import torch
+import torch.nn as nn
+import numpy as np
+from torch.utils.data import TensorDataset, DataLoader
+from sklearn.metrics import confusion_matrix, accuracy_score, f1_score
+from sklearn.metrics.pairwise import cosine_similarity
+from scipy.signal import find_peaks
+from scipy.spatial.distance import euclidean
+
+# Add project-specific imports
+sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))
+from backend.registry import choices as model_choices, build as build_model
+from utils.training_engine import TrainingEngine
+from utils.training_types import (
+    TrainingConfig,
+    TrainingProgress,
+    TrainingStatus,
+    CVStrategy,
+    get_cv_splitter,
+)
+from backend.utils.preprocessing import preprocess_spectrum
+
+
+def spectral_cosine_similarity(y_true: np.ndarray, y_pred: np.ndarray) -> float:
+    """Calculate cosine similarity between spectral predictions and true values"""
+    # Reshape if needed for cosine similarity calculation
+    if y_true.ndim == 1:
+        y_true = y_true.reshape(1, -1)
+    if y_pred.ndim == 1:
+        y_pred = y_pred.reshape(1, -1)
+
+    return float(cosine_similarity(y_true, y_pred)[0, 0])
+
+
+def peak_matching_score(
+    spectrum1: np.ndarray,
+    spectrum2: np.ndarray,
+    height_threshold: float = 0.1,
+    distance: int = 5,
+) -> float:
+    """Calculate peak matching score between two spectra"""
+    try:
+        # Find peaks in both spectra
+        peaks1, _ = find_peaks(spectrum1, height=height_threshold, distance=distance)
+        peaks2, _ = find_peaks(spectrum2, height=height_threshold, distance=distance)
+
+        if len(peaks1) == 0 or len(peaks2) == 0:
+            return 0.0
+
+        # Calculate matching peaks (within tolerance)
+        tolerance = 3  # wavenumber tolerance
+        matches = 0
+
+        for peak1 in peaks1:
+            for peak2 in peaks2:
+                if abs(peak1 - peak2) <= tolerance:
+                    matches += 1
+                    break
+
+        # Return normalized matching score
+        return matches / max(len(peaks1), len(peaks2))
+    except:
+        return 0.0
+
+
+def spectral_euclidean_distance(y_true: np.ndarray, y_pred: np.ndarray) -> float:
+    """Calculate normalized Euclidean distance between spectra"""
+    try:
+        distance = euclidean(y_true.flatten(), y_pred.flatten())
+        # Normalize by the length of the spectrum
+        return distance / len(y_true.flatten())
+    except:
+        return float("inf")
+
+
+def calculate_spectroscopy_metrics(
+    y_true: np.ndarray, y_pred: np.ndarray, probabilities: Optional[np.ndarray] = None
+) -> Dict[str, float]:
+    """Calculate comprehensive spectroscopy-specific metrics"""
+    metrics = {}
+
+    try:
+        # Standard classification metrics
+        metrics["accuracy"] = accuracy_score(y_true, y_pred)
+        metrics["f1_score"] = f1_score(y_true, y_pred, average="weighted")
+
+        # Spectroscopy-specific metrics
+        if probabilities is not None and len(probabilities.shape) > 1:
+            # For classification with probabilities, use cosine similarity on prob distributions
+            unique_classes = np.unique(y_true)
+            if len(unique_classes) > 1:
+                # Convert true labels to one-hot for similarity calculation
+                y_true_onehot = np.eye(len(unique_classes))[y_true]
+                metrics["cosine_similarity"] = float(
+                    cosine_similarity(
+                        y_true_onehot.mean(axis=0).reshape(1, -1),
+                        probabilities.mean(axis=0).reshape(1, -1),
+                    )[0, 0]
+                )
+
+        # Add bias audit metric (class distribution comparison)
+        unique_true, counts_true = np.unique(y_true, return_counts=True)
+        unique_pred, counts_pred = np.unique(y_pred, return_counts=True)
+
+        # Calculate distribution difference (Jensen-Shannon divergence approximation)
+        true_dist = counts_true / len(y_true)
+        pred_dist = np.zeros_like(true_dist)
+
+        for i, class_label in enumerate(unique_true):
+            if class_label in unique_pred:
+                pred_idx = np.where(unique_pred == class_label)[0][0]
+                pred_dist[i] = counts_pred[pred_idx] / len(y_pred)
+
+        # Simple distribution similarity (1 - average absolute difference)
+        metrics["distribution_similarity"] = 1.0 - np.mean(
+            np.abs(true_dist - pred_dist)
+        )
+
+    except Exception as e:
+        print(f"Error calculating spectroscopy metrics: {e}")
+        # Return basic metrics
+        metrics = {
+            "accuracy": accuracy_score(y_true, y_pred) if len(y_true) > 0 else 0.0,
+            "f1_score": (
+                f1_score(y_true, y_pred, average="weighted") if len(y_true) > 0 else 0.0
+            ),
+            "cosine_similarity": 0.0,
+            "distribution_similarity": 0.0,
+        }
+
+    return metrics
+
+
+@dataclass
+class AugmentationConfig:
+    """Data augmentation configuration"""
+
+    enable_augmentation: bool = False
+    noise_level: float = 0.01  # Noise level for augmentation
+
+
+@dataclass
+class PreprocessingConfig:
+    """Preprocessing configuration"""
+
+    baseline_correction: bool = True
+    smoothing: bool = True
+    normalization: bool = True
+
+
+@dataclass
+class TrainingConfig:
+    """Training configuration parameters"""
+
+    model_name: str
+    dataset_path: str
+    target_len: int = 500
+    batch_size: int = 16
+    epochs: int = 10
+    learning_rate: float = 1e-3
+    num_folds: int = 10
+    modality: str = "raman"
+    device: str = "auto"  # auto, cpu, cuda
+    cv_strategy: str = "stratified_kfold"  # New field for CV strategy
+    spectral_weight: float = 0.1  # Weight for spectroscopy-specific metrics
+    augmentation: AugmentationConfig = field(default_factory=AugmentationConfig)
+    preprocessing: PreprocessingConfig = field(default_factory=PreprocessingConfig)
+
+    def to_dict(self) -> Dict[str, Any]:
+        """Convert to dictionary for serialization"""
+        return asdict(self)
+
+
+@dataclass
+class TrainingProgress:
+    """Training progress tracking with enhanced metrics"""
+
+    current_fold: int = 0
+    total_folds: int = 10
+    current_epoch: int = 0
+    total_epochs: int = 10
+    current_loss: float = 0.0
+    current_accuracy: float = 0.0
+    fold_accuracies: List[float] = field(default_factory=list)
+    confusion_matrices: List[List[List[int]]] = field(default_factory=list)
+    spectroscopy_metrics: List[Dict[str, float]] = field(default_factory=list)
+    start_time: Optional[datetime] = None
+    end_time: Optional[datetime] = None
+
+
+@dataclass
+class TrainingJob:
+    """Training job container"""
+
+    job_id: str
+    config: TrainingConfig
+    status: TrainingStatus = TrainingStatus.PENDING
+    progress: TrainingProgress = None
+    error_message: Optional[str] = None
+    created_at: datetime = None
+    started_at: Optional[datetime] = None
+    completed_at: Optional[datetime] = None
+    weights_path: Optional[str] = None
+    logs_path: Optional[str] = None
+
+    def __post_init__(self):
+        if self.progress is None:
+            self.progress = TrainingProgress(
+                total_folds=self.config.num_folds, total_epochs=self.config.epochs
+            )
+        if self.created_at is None:
+            self.created_at = datetime.now()
+
+
+class TrainingManager:
+    """Manager for training jobs with async execution and progress tracking"""
+
+    def __init__(
+        self,
+        max_workers: int = 2,
+        output_dir: str = "outputs",
+        use_multiprocessing: bool = True,
+    ):
+        self.max_workers = max_workers
+        self.use_multiprocessing = use_multiprocessing
+
+        # Use ProcessPoolExecutor for CPU/GPU-bound tasks, ThreadPoolExecutor for I/O-bound
+        if use_multiprocessing:
+            # Limit workers to available CPU cores to prevent oversubscription
+            actual_workers = min(max_workers, multiprocessing.cpu_count())
+            self.executor = concurrent.futures.ProcessPoolExecutor(
+                max_workers=actual_workers
+            )
+        else:
+            self.executor = concurrent.futures.ThreadPoolExecutor(
+                max_workers=max_workers
+            )
+
+        self.jobs: Dict[str, TrainingJob] = {}
+        self.output_dir = Path(output_dir)
+        self.output_dir.mkdir(exist_ok=True)
+        (self.output_dir / "weights").mkdir(exist_ok=True)
+
+    def generate_job_id(self) -> str:
+        """Generate unique job ID"""
+        return f"train_{uuid.uuid4().hex[:8]}_{int(time.time())}"
+
+    def submit_training_job(
+        self, config: TrainingConfig, progress_callback: Optional[Callable] = None
+    ) -> str:
+        """Submit a new training job"""
+        job_id = self.generate_job_id()
+        job = TrainingJob(job_id=job_id, config=config)
+
+        self.jobs[job_id] = job
+
+        # Submit to thread pool
+        self.executor.submit(
+            self._run_training_job, job, progress_callback=progress_callback
+        )
+
+        return job_id
+
+    def _run_training_job(self, job: TrainingJob) -> None:
+        """Execute training job (runs in separate thread)"""
+        try:
+            job.status = TrainingStatus.RUNNING
+            job.started_at = datetime.now()
+            if job.progress:
+                job.progress.start_time = job.started_at
+
+            if progress_callback:
+                progress_callback(job)
+
+            # Load and preprocess data
+            X, y = self._load_and_preprocess_data(job)
+            if X is None or y is None:
+                raise ValueError("Failed to load dataset")
+
+            # Define a callback to update the job's progress object
+            def engine_progress_callback(progress_data: dict):
+                if job.progress:
+                    if progress_data["type"] == "fold_start":
+                        job.progress.current_fold = progress_data["fold"]
+                    elif progress_data["type"] == "epoch_end":
+                        job.progress.current_epoch = progress_data["epoch"]
+                        job.progress.current_loss = progress_data["loss"]
+                if progress_callback:
+                    progress_callback(job)
+
+            # Instantiate and run the training engine
+            engine = TrainingEngine(job.config)
+            results = engine.run(X, y, progress_callback=engine_progress_callback)
+
+            # Update job with results
+            if job.progress:
+                job.progress.fold_accuracies = results["fold_accuracies"]
+                job.progress.confusion_matrices = results["confusion_matrices"]
+
+            # Save model weights and logs
+            self._save_model_weights(job, results["model_state_dict"])
+            self._save_training_results(job)
+
+            job.status = TrainingStatus.COMPLETED
+            job.completed_at = datetime.now()
+            job.progress.end_time = job.completed_at
+
+        except Exception as e:
+            job.status = TrainingStatus.FAILED
+            job.error_message = str(e)
+            job.completed_at = datetime.now()
+
+        finally:
+            if progress_callback:
+                progress_callback(job)
+
+    def _load_and_preprocess_data(
+        self, job: TrainingJob
+    ) -> Tuple[Optional[np.ndarray], Optional[np.ndarray]]:
+        """Load and preprocess dataset with enhanced validation and security"""
+        try:
+            config = job.config
+            dataset_path = Path(config.dataset_path)
+
+            # Enhanced path validation and security
+            if not dataset_path.exists():
+                raise FileNotFoundError(f"Dataset path not found: {dataset_path}")
+
+            # Validate dataset path is within allowed directories (security)
+            try:
+                dataset_path = dataset_path.resolve()
+                allowed_bases = [
+                    Path("datasets").resolve(),
+                    Path("data").resolve(),
+                    Path("/tmp").resolve(),
+                ]
+                if not any(
+                    str(dataset_path).startswith(str(base)) for base in allowed_bases
+                ):
+                    raise ValueError(
+                        f"Dataset path outside allowed directories: {dataset_path}"
+                    )
+            except Exception as e:
+                print(f"Path validation error: {e}")
+                raise ValueError("Invalid dataset path")
+
+            # Load data from dataset directory
+            X, y = [], []
+            total_files = 0
+            processed_files = 0
+            max_files_per_class = 1000  # Limit to prevent memory issues
+            max_file_size = 10 * 1024 * 1024  # 10MB per file
+
+            # Look for data files in the dataset directory
+            for label_dir in dataset_path.iterdir():
+                if not label_dir.is_dir():
+                    continue
+
+                label = 0 if "stable" in label_dir.name.lower() else 1
+                files_in_class = 0
+
+                # Support multiple file formats
+                file_patterns = ["*.txt", "*.csv", "*.json"]
+
+                for pattern in file_patterns:
+                    for file_path in label_dir.glob(pattern):
+                        total_files += 1
+
+                        # Security: Check file size
+                        if file_path.stat().st_size > max_file_size:
+                            print(
+                                f"Skipping large file: {file_path} ({file_path.stat().st_size} bytes)"
+                            )
+                            continue
+
+                        # Limit files per class
+                        if files_in_class >= max_files_per_class:
+                            print(
+                                f"Reached maximum files per class ({max_files_per_class}) for {label_dir.name}"
+                            )
+                            break
+
+                        try:
+                            # Load spectrum data based on file type
+                            if file_path.suffix.lower() == ".txt":
+                                data = np.loadtxt(file_path)
+                                if data.ndim == 2 and data.shape[1] >= 2:
+                                    x_raw, y_raw = data[:, 0], data[:, 1]
+                                elif data.ndim == 1:
+                                    # Single column data
+                                    x_raw = np.arange(len(data))
+                                    y_raw = data
+                                else:
+                                    continue
+
+                            elif file_path.suffix.lower() == ".csv":
+                                import pandas as pd
+
+                                df = pd.read_csv(file_path)
+                                if df.shape[1] >= 2:
+                                    x_raw, y_raw = (
+                                        df.iloc[:, 0].values,
+                                        df.iloc[:, 1].values,
+                                    )
+                                else:
+                                    x_raw = np.arange(len(df))
+                                    y_raw = df.iloc[:, 0].values
+
+                            elif file_path.suffix.lower() == ".json":
+                                with open(file_path, "r") as f:
+                                    data_dict = json.load(f)
+                                if isinstance(data_dict, dict):
+                                    if "x" in data_dict and "y" in data_dict:
+                                        x_raw, y_raw = np.array(
+                                            data_dict["x"]
+                                        ), np.array(data_dict["y"])
+                                    elif "spectrum" in data_dict:
+                                        y_raw = np.array(data_dict["spectrum"])
+                                        x_raw = np.arange(len(y_raw))
+                                    else:
+                                        continue
+                                else:
+                                    continue
+                            else:
+                                continue
+
+                            # Validate data integrity
+                            if len(x_raw) != len(y_raw) or len(x_raw) < 10:
+                                print(
+                                    f"Invalid data in file {file_path}: insufficient data points"
+                                )
+                                continue
+
+                            # Check for NaN or infinite values
+                            if np.any(np.isnan(y_raw)) or np.any(np.isinf(y_raw)):
+                                print(
+                                    f"Invalid data in file {file_path}: NaN or infinite values"
+                                )
+                                continue
+
+                            # Validate reasonable value ranges for spectroscopy
+                            if np.min(y_raw) < -1000 or np.max(y_raw) > 1e6:
+                                print(
+                                    f"Suspicious data values in file {file_path}: outside expected range"
+                                )
+                                continue
+
+                            # Preprocess spectrum
+                            _, y_processed = preprocess_spectrum(
+                                x_raw,
+                                y_raw,
+                                modality=config.modality,
+                                target_len=config.target_len,
+                                do_baseline=config.baseline_correction,
+                                do_smooth=config.smoothing,
+                                do_normalize=config.normalization,
+                            )
+
+                            # Final validation of processed data
+                            if (
+                                y_processed is None
+                                or len(y_processed) != config.target_len
+                            ):
+                                print(f"Preprocessing failed for file {file_path}")
+                                continue
+
+                            X.append(y_processed)
+                            y.append(label)
+                            files_in_class += 1
+                            processed_files += 1
+
+                        except Exception as e:
+                            print(f"Error processing file {file_path}: {e}")
+                            continue
+
+            # Validate final dataset
+            if len(X) == 0:
+                raise ValueError("No valid data files found in dataset")
+
+            if len(X) < 10:
+                raise ValueError(
+                    f"Insufficient data: only {len(X)} samples found (minimum 10 required)"
+                )
+
+            # Check class balance
+            unique_labels, counts = np.unique(y, return_counts=True)
+            if len(unique_labels) < 2:
+                raise ValueError("Dataset must contain at least 2 classes")
+
+            min_class_size = min(counts)
+            if min_class_size < 3:
+                raise ValueError(
+                    f"Insufficient samples in one class: minimum {min_class_size} (need at least 3)"
+                )
+
+            print(f"Dataset loaded: {processed_files}/{total_files} files processed")
+            print(f"Class distribution: {dict(zip(unique_labels, counts))}")
+
+            return np.array(X, dtype=np.float32), np.array(y, dtype=np.int64)
+
+        except Exception as e:
+            print(f"Error loading dataset: {e}")
+            return None, None
+
+    def _save_model_weights(self, job: TrainingJob, model_state_dict: dict):
+        """Saves the model's state dictionary to a file."""
+        weights_dir = self.output_dir / "weights"
+        weights_dir.mkdir(exist_ok=True)
+        job.weights_path = str(weights_dir / f"{job.config.model_name}_model.pth")
+        torch.save(model_state_dict, job.weights_path)
+
+    def _save_training_results(self, job: TrainingJob):
+        """Save training results and logs with enhanced metrics"""
+        logs_dir = self.output_dir / "logs"
+        logs_dir.mkdir(exist_ok=True)
+        job.logs_path = str(logs_dir / f"{job.job_id}_log.json")
+
+        # Calculate comprehensive summary metrics
+        spectro_summary = {}
+        if job.progress.spectroscopy_metrics:
+            # Average across all folds for each metric
+            metric_keys = job.progress.spectroscopy_metrics[0].keys()
+            for key in metric_keys:
+                values = [
+                    fold_metrics.get(key, 0.0)
+                    for fold_metrics in job.progress.spectroscopy_metrics
+                ]
+                spectro_summary[f"mean_{key}"] = float(np.mean(values))
+                spectro_summary[f"std_{key}"] = float(np.std(values))
+
+        results = {
+            "job_id": job.job_id,
+            "config": job.config.to_dict(),
+            "status": job.status.value,
+            "created_at": job.created_at.isoformat(),
+            "started_at": job.started_at.isoformat() if job.started_at else None,
+            "completed_at": job.completed_at.isoformat() if job.completed_at else None,
+            "progress": {
+                "fold_accuracies": job.progress.fold_accuracies,
+                "confusion_matrices": job.progress.confusion_matrices,
+                "spectroscopy_metrics": job.progress.spectroscopy_metrics,
+                "mean_accuracy": (
+                    np.mean(job.progress.fold_accuracies)
+                    if job.progress.fold_accuracies
+                    else 0.0
+                ),
+                "std_accuracy": (
+                    np.std(job.progress.fold_accuracies)
+                    if job.progress.fold_accuracies
+                    else 0.0
+                ),
+                "spectroscopy_summary": spectro_summary,
+            },
+            "weights_path": job.weights_path,
+            "error_message": job.error_message,
+        }
+
+        if job.logs_path:
+            with open(job.logs_path, "w") as f:
+                json.dump(results, f, indent=2)
+
+    def get_job_status(self, job_id: str) -> Optional[TrainingJob]:
+        """Get current status of a training job"""
+        return self.jobs.get(job_id)
+
+    def list_jobs(
+        self, status_filter: Optional[TrainingStatus] = None
+    ) -> List[TrainingJob]:
+        """List all jobs, optionally filtered by status"""
+        jobs = list(self.jobs.values())
+        if status_filter:
+            jobs = [job for job in jobs if job.status == status_filter]
+        return sorted(jobs, key=lambda j: j.created_at, reverse=True)
+
+    def cancel_job(self, job_id: str) -> bool:
+        """Cancel a running job"""
+        job = self.jobs.get(job_id)
+        if job and job.status == TrainingStatus.RUNNING:
+            job.status = TrainingStatus.CANCELLED
+            job.completed_at = datetime.now()
+            # Note: This is a simple cancellation - actual thread termination is more complex
+            return True
+        return False
+
+    def cleanup_old_jobs(self, max_age_hours: int = 24):
+        """Clean up old completed/failed jobs"""
+        cutoff_time = datetime.now() - timedelta(hours=max_age_hours)
+        to_remove = []
+
+        for job_id, job in self.jobs.items():
+            if (
+                job.status
+                in [
+                    TrainingStatus.COMPLETED,
+                    TrainingStatus.FAILED,
+                    TrainingStatus.CANCELLED,
+                ]
+                and job.completed_at
+                and job.completed_at < cutoff_time
+            ):
+                to_remove.append(job_id)
+
+        for job_id in to_remove:
+            del self.jobs[job_id]
+
+    def shutdown(self):
+        """Shutdown the training manager"""
+        self.executor.shutdown(wait=True)
+
+
+# Global training manager instance
+_training_manager = None
+
+
+def get_training_manager() -> TrainingManager:
+    """Get global training manager instance"""
+    global _training_manager
+    if _training_manager is None:
+        _training_manager = TrainingManager()
+    return _training_manager

backend/utils/training_types.py

@@ -0,0 +1,128 @@
+"""
+Defines core data structures and types for the training system.
+
+This module centralizes data classes like TrainingConfig and helper
+functions to avoid circular dependencies between the TrainingManager
+and TrainingEngine.
+"""
+
+from dataclasses import dataclass, asdict, field
+from enum import Enum
+from typing import List, Optional, Dict, Any, Tuple
+from datetime import datetime
+import numpy as np
+
+from sklearn.model_selection import StratifiedKFold, KFold, TimeSeriesSplit
+
+
+class TrainingStatus(Enum):
+    """Training job status enumeration"""
+
+    PENDING = "pending"
+    RUNNING = "running"
+    COMPLETED = "completed"
+    FAILED = "failed"
+    CANCELLED = "cancelled"
+
+
+class CVStrategy(Enum):
+    """Cross-validation strategy enumeration"""
+
+    STRATIFIED_KFOLD = "stratified_kfold"
+    KFOLD = "kfold"
+    TIME_SERIES_SPLIT = "time_series_split"
+
+
+@dataclass
+class TrainingConfig:
+    """Training configuration parameters"""
+
+    model_name: str
+    dataset_path: str
+    target_len: int = 500
+    batch_size: int = 16
+    epochs: int = 10
+    learning_rate: float = 1e-3
+    num_folds: int = 10
+    baseline_correction: bool = True
+    smoothing: bool = True
+    normalization: bool = True
+    modality: str = "raman"
+    device: str = "auto"  # auto, cpu, cuda
+    cv_strategy: str = "stratified_kfold"  # New field for CV strategy
+    spectral_weight: float = 0.1  # Weight for spectroscopy-specific metrics
+    enable_augmentation: bool = False  # Enable data augmentation
+    noise_level: float = 0.01  # Noise level for augmentation
+
+    def to_dict(self) -> Dict[str, Any]:
+        """Convert to dictionary for serialization"""
+        return asdict(self)
+
+
+@dataclass
+class TrainingProgress:
+    """Training progress tracking with enhanced metrics"""
+
+    current_fold: int = 0
+    total_folds: int = 10
+    current_epoch: int = 0
+    total_epochs: int = 10
+    current_loss: float = 0.0
+    current_accuracy: float = 0.0
+    fold_accuracies: List[float] = field(default_factory=list)
+    confusion_matrices: List[List[List[int]]] = field(default_factory=list)
+    spectroscopy_metrics: List[Dict[str, float]] = field(default_factory=list)
+    start_time: Optional[datetime] = None
+    end_time: Optional[datetime] = None
+
+
+def get_cv_splitter(strategy: str, n_splits: int = 10, random_state: int = 42):
+    """Get cross-validation splitter based on strategy"""
+    if strategy == "stratified_kfold":
+        return StratifiedKFold(
+            n_splits=n_splits, shuffle=True, random_state=random_state
+        )
+    elif strategy == "kfold":
+        return KFold(n_splits=n_splits, shuffle=True, random_state=random_state)
+    elif strategy == "time_series_split":
+        return TimeSeriesSplit(n_splits=n_splits)
+    else:
+        # Default to stratified k-fold
+        return StratifiedKFold(
+            n_splits=n_splits, shuffle=True, random_state=random_state
+        )
+
+
+def augment_spectral_data(
+    X: np.ndarray,
+    y: np.ndarray,
+    noise_level: float = 0.01,
+    augmentation_factor: int = 2,
+) -> Tuple[np.ndarray, np.ndarray]:
+    """Augment spectral data with realistic noise and variations"""
+    if augmentation_factor <= 1:
+        return X, y
+
+    augmented_X = [X]
+    augmented_y = [y]
+
+    for i in range(augmentation_factor - 1):
+        # Add Gaussian noise
+        noise = np.random.normal(0, noise_level, X.shape)
+        X_noisy = X + noise
+
+        # Add baseline drift (common in spectroscopy)
+        baseline_drift = np.random.normal(0, noise_level * 0.5, (X.shape[0], 1))
+        X_drift = X_noisy + baseline_drift
+
+        # Add intensity scaling variation
+        intensity_scale = np.random.normal(1.0, 0.05, (X.shape[0], 1))
+        X_scaled = X_drift * intensity_scale
+
+        # Ensure no negative values
+        X_scaled = np.maximum(X_scaled, 0)
+
+        augmented_X.append(X_scaled)
+        augmented_y.append(y)
+
+    return np.vstack(augmented_X), np.hstack(augmented_y)

frontend/README.md

@@ -0,0 +1,120 @@
+# Frontend React Application
+
+This directory contains the React/TypeScript frontend for the Polymer Aging ML application.
+
+## Architecture
+
+The frontend is a modern React application with TypeScript:
+
+```
+frontend/
+├── src/
+│   ├── components/          # React components
+│   │   ├── Header.tsx
+│   │   ├── SpectrumChart.tsx
+│   │   ├── ResultsDisplay.tsx
+│   │   └── ...
+│   ├── apiClient.ts         # Centralized API client
+│   ├── types/
+│   │   ├── api.ts          # Auto-generated API types
+│   │   └── index.ts        # Custom types
+│   ├── App.tsx             # Main application component
+│   └── index.tsx           # Application entry point
+├── public/                 # Static assets
+├── package.json           # Dependencies and scripts
+└── tsconfig.json          # TypeScript configuration
+```
+
+## Key Features
+
+- **Type Safety**: Full TypeScript integration with OpenAPI-generated types
+- **Centralized API Client**: Single source for all backend communication
+- **Component Architecture**: Modular, reusable React components
+- **Responsive Design**: Works across desktop and mobile devices
+- **Error Handling**: Graceful error handling with user feedback
+
+## Setup and Development
+
+### Prerequisites
+
+- Node.js 16+ and npm 8+
+- Backend API server running (for development)
+
+### Installation
+
+```bash
+# Install dependencies
+npm install --legacy-peer-deps
+
+# Verify TypeScript types are up to date
+npm run typegen:file
+```
+
+### Development Server
+
+```bash
+# Start development server with hot reload
+npm start
+
+# Opens http://localhost:3000
+```
+
+### Build for Production
+
+```bash
+# Create production build
+npm run build
+
+# Build files output to build/ directory
+```
+
+## API Integration
+
+### Centralized API Client
+
+All backend communication goes through `src/apiClient.ts`:
+
+```typescript
+import { ApiClient } from './apiClient';
+
+const api = new ApiClient('http://localhost:8000');
+
+// Example usage
+const result = await api.analyzeSpectrum({
+  spectrum: spectrumData,
+  model_name: 'resnet',
+  modality: 'raman'
+});
+```
+
+### Type Generation
+
+API types are automatically generated from the OpenAPI schema:
+
+```bash
+# Generate types from running backend
+npm run typegen
+
+# Generate types from schema file
+npm run typegen:file
+```
+
+## Scripts
+
+Available npm scripts:
+
+```bash
+npm start          # Development server
+npm run build      # Production build
+npm test           # Run tests
+npm run lint       # ESLint checking
+npm run format     # Prettier formatting
+npm run typegen    # Generate API types
+```
+
+## API Contract Adherence
+
+The frontend strictly adheres to the OpenAPI contract:
+- All requests/responses validated by TypeScript types
+- Automatic type generation ensures contract compliance
+- No direct backend imports or cross-boundary dependencies

frontend/package.json

@@ -0,0 +1,65 @@
+{
+  "name": "polymer-aging-ml",
+  "version": "1.0.0",
+  "description": "Full-stack React/TypeScript project for polymer aging ML, deployable in Docker and Hugging Face Space.",
+  "scripts": {
+    "start": "react-scripts start",
+    "build": "react-scripts build",
+    "test": "react-scripts test",
+    "eject": "react-scripts eject",
+    "typegen": "openapi-typescript http://localhost:8000/api/v1/openapi.json --output src/types/api.ts",
+    "typegen:file": "openapi-typescript ../openapi-schema.json --output src/types/api.ts",
+    "check-types-sync": "../scripts/check-typegen-sync.sh",
+    "lint": "eslint . --ext .ts,.tsx,.js,.jsx",
+    "format": "prettier --write \"src/**/*.{ts,tsx,js,jsx,json,css,md}\""
+  },
+  "keywords": [
+    "react",
+    "typescript",
+    "ml",
+    "polymer",
+    "huggingface"
+  ],
+  "author": "devjas1",
+  "license": "MIT",
+  "dependencies": {
+    "axios": "^1.5.0",
+    "openapi-typescript-codegen": "^0.29.0",
+    "react": "^18.2.0",
+    "react-dom": "^18.2.0",
+    "react-dropzone": "^14.3.8",
+    "react-router-dom": "^6.15.0",
+    "react-scripts": "^5.0.0",
+    "recharts": "^3.2.1",
+    "web-vitals": "^5.1.0"
+  },
+  "devDependencies": {
+    "@testing-library/jest-dom": "^6.8.0",
+    "@testing-library/react": "^14.3.1",
+    "@types/jest": "^30.0.0",
+    "@types/react": "^18.0.37",
+    "@types/react-dom": "^18.0.11",
+    "@types/react-dropzone": "^4.2.2",
+    "@types/react-router-dom": "^5.3.3",
+    "@types/recharts": "^1.8.29",
+    "eslint": "^8.56.0",
+    "eslint-config-prettier": "^9.1.0",
+    "eslint-plugin-react": "^7.33.2",
+    "jest": "^28.1.3",
+    "openapi-typescript": "^6.7.0",
+    "prettier": "^3.2.2",
+    "typescript": "^4.9.5"
+  },
+  "browserslist": {
+    "production": [
+      ">0.2%",
+      "not dead",
+      "not op_mini all"
+    ],
+    "development": [
+      "last 1 chrome version",
+      "last 1 firefox version",
+      "last 1 safari version"
+    ]
+  }
+}

frontend/public/index.html

@@ -0,0 +1,40 @@
+<!DOCTYPE html>
+<html lang="en">
+  <head>
+    <meta charset="utf-8" />
+    <link rel="icon" href="%PUBLIC_URL%/favicon.ico" />
+    <meta name="viewport" content="width=device-width, initial-scale=1" />
+    <meta name="theme-color" content="#000000" />
+    <meta
+      name="description"
+      content="Web site created using create-react-app"
+    />
+    <link rel="apple-touch-icon" href="%PUBLIC_URL%/logo192.png" />
+    <!--
+      manifest.json provides metadata used when your web app is installed on a
+      user's mobile device or desktop. See https://developers.google.com/web/fundamentals/web-app-manifest/
+    -->
+    <link rel="manifest" href="%PUBLIC_URL%/manifest.json" />
+    <!--
+      Notice the use of %PUBLIC_URL% in the tags above.
+      It will be replaced with the URL of the `public` folder during the build.
+      Only files inside the `public` folder can be referenced from the HTML.
+      Unlike "/favicon.ico" or "favicon.ico", "%PUBLIC_URL%/favicon.ico" will
+      work correctly both with client-side routing and a non-root public URL.
+      Learn how to configure a non-root public URL by running `npm run build`.
+    -->
+    <title>React App</title>
+  </head>
+  <body>
+    <noscript>You need to enable JavaScript to run this app.</noscript>
+    <div id="root"></div>
+    <!--
+      This HTML file is a template.
+      If you open it directly in the browser, you will see an empty page.
+      You can add webfonts, meta tags, or analytics to this file.
+      The build step will place the bundled scripts into the <body> tag.
+      To begin the development, run `npm start` or `yarn start`.
+      To create a production bundle, use `npm run build` or `yarn build`.
+    -->
+  </body>
+</html>

frontend/public/robots.txt

@@ -0,0 +1,3 @@
+# https://www.robotstxt.org/robotstxt.html
+User-agent: *
+Disallow: