README.md

metadata

title: Multi-Agent NLP System
emoji: 🤖
colorFrom: blue
colorTo: purple
sdk: docker
app_port: 7860
pinned: false
license: mit

Multi-Agent NLP System

A comprehensive multi-agent system for NLP and multimedia generation tasks, featuring distributed processing with Kafka message queues, Redis caching, and MinIO object storage.

🏗️ Architecture

This system implements a microservices architecture with:

• Kafka Message Broker: Inter-service communication
• Redis Cache: Fast data caching and session management
• MinIO Object Storage: File and media storage
• Multiple Agents: NLP processing, image generation, video processing
• API Gateway: External interface and request routing

🚀 Features

• Distributed Processing: Multiple specialized agents working in parallel
• Scalable Architecture: Microservices-based design for easy scaling
• Real-time Communication: Kafka-based message passing
• Persistent Storage: MinIO for files, Redis for caching
• Health Monitoring: Comprehensive system health checks
• Docker Support: Containerized deployment

🛠️ Technology Stack

• Backend: Python 3.9+ with asyncio
• Message Broker: Apache Kafka with Zookeeper
• Cache: Redis
• Storage: MinIO (S3-compatible)
• Containerization: Docker & Docker Compose
• Monitoring: Custom health check system

📋 Services

Core Infrastructure

• Zookeeper: Kafka coordination
• Kafka: Message broker for inter-service communication
• Redis: High-performance caching layer
• MinIO: Object storage for files and media

Application Services

• API Gateway: External interface and request routing
• NLP Agent: Natural language processing tasks
• Image Agent: Image generation and processing
• Video Agent: Video processing and generation
• Monitoring: System health and performance monitoring

🔧 Installation & Setup

Prerequisites

• Docker and Docker Compose
• Python 3.9+
• 4GB+ RAM recommended

Quick Start

1. Clone the repository

git clone <repository-url>
cd multi-agent-nlp-system

2. Install dependencies

pip install -r requirements-validation.txt

3. Start the system

docker compose -f docker-compose-minimal.yml up -d

4. Validate installation

python3 simple_check.py

Development Setup

For development with full agent services:

docker compose up -d

🧪 Testing

The system includes comprehensive testing:

# Infrastructure validation
python3 simple_check.py

# Kafka client testing
python3 test_kafka_client.py

# Full system validation
python3 run_debug_tests.py

📊 Monitoring

Access the monitoring interfaces:

• MinIO Console: http://localhost:9001 (admin/minioadmin123)
• Kafka Topics: docker exec kafka kafka-topics --list --bootstrap-server localhost:9092
• Redis CLI: docker exec -it redis redis-cli

🌐 API Endpoints

Once deployed, the system will expose:

• Health Check: /health
• NLP Processing: /api/nlp/process
• Image Generation: /api/image/generate
• Video Processing: /api/video/process

🔒 Security

• Environment-based configuration
• Secure inter-service communication
• Input validation and sanitization
• Rate limiting and authentication ready

📈 Performance

• Horizontal Scaling: Add more agent instances
• Load Balancing: Built-in Kafka consumer groups
• Caching: Redis for frequently accessed data
• Async Processing: Non-blocking I/O throughout

🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests for new functionality
5. Submit a pull request

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

🆘 Support

For issues and questions:

• Check the troubleshooting guide in DEBUG_README.md
• Review system logs: docker compose logs
• Run health checks: python3 simple_check.py

🏷️ Version

Current version: 1.0.0 - Production Ready

---

Built with ❤️ for scalable AI applications