README.md

metadata

title: Sentence-Translator
emoji: 🎓
colorFrom: blue
colorTo: green
sdk: docker
app_file: main.py
pinned: false

🌐 Sentence-Translator: English to Hindi

A robust, enterprise-grade English-to-Hindi translation application built with a Sequence-to-Sequence (Seq2Seq) architecture and modern MLOps principles.

🚀 Key Features

• Advanced Deep Learning: Implements a GRU-based Encoder-Decoder architecture with teacher forcing.
• Memory-Mapped Data (Scale-Ready): Custom np.memmap integration for data transformation and training, allowing the pipeline to handle datasets far larger than available RAM.
• Modular MLOps Pipeline:
  • Data Ingestion: Automated fetching and ingestion.
  • Data Validation: Schema and quality checks.
  • Data Transformation: Fixed-width tokenization and memory-mapped storage.
  • Model Training: Scalable training with configurable hyperparameters.
  • Prediction: Robust inference engine with architecture reconstruction.
• Live Tracking: Integrated with MLflow and DagsHub for experiment tracking.
• Modern UI: Interactive Streamlit dashboard for real-time translation.

📊 Live Experiment Tracking

Monitor training metrics and model performance here: DagsHub MLflow Tracking

🛠️ Tech Stack

• Core: PyTorch, NumPy, Pandas
• Experiment Tracking: MLflow, DagsHub
• Platform: Streamlit
• Dependency Management: uv
• Data Versioning: DVC

📁 Project Structure

├── config/                 # YAML configs for training and validation
├── src/
│   ├── components/        # Pipeline components (Ingestion, Training, etc.)
│   ├── entity/           # Data classes for artifacts and configs
│   ├── pipelines/        # Process-specific workflow pipelines
│   └── utils/            # Shared utilities (Main utils, Async handler)
├── saved_model/            # Production-ready model and vocab artifacts
├── StreamlitApp/           # Interactive web application
└── notebooks/              # Research and experimentation

🏗️ Getting Started

1. Installation

Using uv for lightning-fast setup:

uv sync

2. Training the Pipeline

To run the full end-to-end training process:

uv run python main.py

3. Running Real-time Predictions

Verify the prediction engine with a sample script:

uv run python src/tests/test_prediction_fix.py

4. Launch the Web App

uv run streamlit run StreamlitApp/app.py

🧠 Technical Highlights: Memory Map Optimization

To prevent "Out of Memory" errors during large-scale training, this project uses Memory-Mapped Files (.dat). Instead of loading the entire tokenized dataset into RAM, we map the files directly to disk using np.memmap, ensuring nearly constant memory usage regardless of dataset size.

---

Developed with ❤️ by Vansh Sharma