# MozDef Architecture: Frontend, Backend & Services Integration Guide ## Table of Contents 1. [Frontend vs Backend Overview](#frontend-vs-backend-overview) 2. [Services Breakdown](#services-breakdown) 3. [How Services Integrate](#how-services-integrate) 4. [Complete Data Flow](#complete-data-flow) 5. [Service Communication Patterns](#service-communication-patterns) 6. [How They Run Together](#how-they-run-together) --- ## Frontend vs Backend Overview ### Frontend Components (User-Facing) **Frontend** = What users interact with directly | Component | Technology | Purpose | Port | |-----------|-----------|---------|------| | **Meteor Web Interface** | Node.js/Meteor | Real-time web UI for incident handling | 80 (via Nginx) | | **Kibana** | JavaScript/React | Log visualization and analysis | 9090 (via Nginx) | **Key Characteristics:** - **Meteor**: Runs on Node.js, provides real-time collaboration - **Kibana**: Standalone visualization tool, connects directly to Elasticsearch - Both are served through **Nginx** as a reverse proxy ### Backend Components (Processing & API) **Backend** = Services that process data and provide APIs | Component | Technology | Purpose | Port | |-----------|-----------|---------|------| | **REST API** | Python/Bottle.py | RESTful API for web interface | 8081 | | **Loginput** | Python/Bottle.py | HTTP endpoint for event ingestion | 8080 | | **MQ Workers** | Python | Process events from RabbitMQ queue | N/A (internal) | | **Alerts Engine** | Python/Celery | Alert correlation and generation | N/A (internal) | | **Alert Actions** | Python | Post-alert processing | N/A (internal) | | **Bot** | Python | Slack/IRC notifications | N/A (internal) | **Key Characteristics:** - All backend services are **Python-based** - Use **uWSGI** for process management - Communicate via **RabbitMQ** message queues - Query/write to **Elasticsearch** and **MongoDB** --- ## Services Breakdown ### Infrastructure Services (Supporting Services) These are the foundational services that other components depend on: | Service | Technology | Purpose | Port | |---------|-----------|---------|------| | **Elasticsearch** | Java | Event storage, indexing, search | 9200 (internal) | | **RabbitMQ** | Erlang | Message queue for event processing | 5672 (internal), 15672 (admin) | | **MongoDB** | C++ | Database for Meteor (incidents, users) | 3002 (internal) | | **Nginx** | C | Reverse proxy and web server | 80, 8080, 8081, 9090 | ### Application Services (MozDef Components) These are the MozDef-specific services: #### **Ingest Services** (Data Ingestion) - **Loginput**: Receives JSON events via HTTP - **MQ Workers**: Process events from queue - **Syslog**: Receives syslog messages #### **Alert Services** (Security Alerting) - **Alerts**: Celery-based alert engine - **Alert Actions**: Post-alert processing - **Bot**: Chat notifications #### **Web Services** (User Interface) - **REST API**: Backend API for web interface - **Meteor**: Frontend web application - **Kibana**: Log visualization (separate service) --- ## How Services Integrate ### Integration Architecture Diagram ``` ┌─────────────────────────────────────────────────────────────────┐ │ USER BROWSER │ └───────────────┬─────────────────────────────────────────────────┘ │ HTTP/WebSocket ▼ ┌─────────────────────────────────────────────────────────────────┐ │ NGINX (Reverse Proxy) │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │ │ Port 80 │ │ Port 8080│ │ Port 8081│ │ Port 9090│ │ │ │ Meteor │ │ Loginput │ │ REST API │ │ Kibana │ │ │ └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘ │ └───────┼─────────────┼──────────────┼─────────────┼──────────────┘ │ │ │ │ ▼ ▼ ▼ ▼ ┌─────────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ METEOR │ │ LOGINPUT │ │ REST API │ │ KIBANA │ │ (Node.js) │ │ (Python) │ │ (Python) │ │ │ └──────┬──────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘ │ │ │ │ │ │ │ │ │ ▼ │ │ │ ┌──────────────┐ │ │ │ │ RABBITMQ │ │ │ │ │ (Message │ │ │ │ │ Queue) │ │ │ │ └──────┬───────┘ │ │ │ │ │ │ │ ▼ │ │ │ ┌──────────────┐ │ │ │ │ MQ WORKERS │ │ │ │ │ (Python) │ │ │ │ └──────┬───────┘ │ │ │ │ │ │ │ │ │ │ └─────────────┼───────────────┼──────────────┘ │ │ ▼ ▼ ┌──────────────┐ ┌──────────────┐ │ ELASTICSEARCH│ │ MONGODB │ │ (Storage) │ │ (Database) │ └──────┬───────┘ └──────┬───────┘ │ │ │ │ ▼ │ ┌──────────────┐ │ │ ALERTS │ │ │ (Celery) │ │ └──────┬───────┘ │ │ │ ▼ │ ┌──────────────┐ │ │ ALERT ACTIONS│ │ │ & BOT │ │ └──────────────┘ │ ``` --- ## Complete Data Flow ### 1. Event Ingestion Flow ``` External Log Shipper │ │ JSON over HTTP POST ▼ ┌─────────────────┐ │ NGINX │ (Port 8080) │ Reverse Proxy │ └────────┬────────┘ │ │ uWSGI socket ▼ ┌─────────────────┐ │ LOGINPUT │ (Python/Bottle.py) │ - Receives JSON│ │ - Validates │ │ - Adds metadata│ └────────┬────────┘ │ │ Publishes to RabbitMQ ▼ ┌─────────────────┐ │ RABBITMQ │ (Message Queue) │ Exchange: │ │ eventtask │ └────────┬────────┘ │ │ Consumes from queue ▼ ┌─────────────────┐ │ MQ WORKERS │ (Python) │ - Normalize │ │ - Enrich │ │ - GeoIP │ │ - Watchlist │ └────────┬────────┘ │ │ Index document ▼ ┌─────────────────┐ │ ELASTICSEARCH │ │ - Stores event │ │ - Indexes │ │ - Searchable │ └─────────────────┘ ``` ### 2. Frontend-Backend Communication Flow ``` User Browser │ │ HTTP Request ▼ ┌─────────────────┐ │ NGINX │ (Port 80) │ Reverse Proxy │ └────────┬────────┘ │ │ Proxy to Meteor ▼ ┌─────────────────┐ │ METEOR │ (Node.js) │ Web Interface │ │ │ │ Client Code: │ │ - UI Components│ │ - Reactivity │ │ │ │ Server Code: │ │ - Methods │ │ - HTTP calls │ └────────┬────────┘ │ │ HTTP POST/GET │ (via mozdef.rootAPI) ▼ ┌─────────────────┐ │ REST API │ (Python/Bottle.py) │ Port 8081 │ │ │ │ Endpoints: │ │ - /events │ │ - /alerts │ │ - /incidents │ │ - /watchitem │ │ - /blockip │ └────────┬────────┘ │ │ Query/Write ├──────────────┐ ▼ ▼ ┌──────────────┐ ┌──────────────┐ │ ELASTICSEARCH│ │ MONGODB │ │ (Events) │ │ (Incidents) │ └──────────────┘ └──────────────┘ ``` **Example: User blocks an IP address** 1. User clicks "Block IP" in Meteor UI 2. Meteor client calls Meteor method: `Meteor.call('blockip', {ip: '1.2.3.4'})` 3. Meteor server method executes: `blockIP(formobj)` 4. Meteor server makes HTTP POST: `HTTP.post(mozdef.rootAPI + '/blockip', {data: formobj})` 5. REST API receives POST at `/blockip` 6. REST API processes request (may call plugins, update watchlist, etc.) 7. REST API returns response 8. Meteor receives response and updates UI reactively ### 3. Alert Processing Flow ``` ┌─────────────────┐ │ ELASTICSEARCH │ │ (Event Store) │ └────────┬────────┘ │ │ Periodic Query │ (Every X minutes) ▼ ┌─────────────────┐ │ ALERTS │ (Python/Celery) │ Alert Engine │ │ │ │ - Queries ES │ │ - Correlates │ │ - Aggregates │ │ - Generates │ └────────┬────────┘ │ │ Creates alert document ├──────────────────────┐ ▼ ▼ ┌──────────────┐ ┌──────────────┐ │ ELASTICSEARCH│ │ RABBITMQ │ │ (alerts-*) │ │ (alerts) │ └──────────────┘ └──────┬───────┘ │ │ Consume alert ▼ ┌──────────────┐ │ ALERT ACTIONS │ │ (Python) │ │ │ │ - Notify │ │ - Enrich │ │ - Respond │ └──────┬───────┘ │ ▼ ┌──────────────┐ │ BOT │ │ (Slack/IRC) │ │ │ │ Sends message│ └──────────────┘ ``` **Example: SSH Brute Force Alert** 1. **Alerts Engine** (Celery) runs every 5 minutes 2. Queries Elasticsearch: "Find SSH failed login events in last 5 minutes" 3. Aggregates by source IP: "Count failures per IP" 4. If IP has > 10 failures: Create alert 5. Alert saved to Elasticsearch `alerts-*` index 6. Alert published to RabbitMQ `alerts` exchange 7. **Alert Actions** worker consumes alert 8. **Alert Actions** enriches alert (GeoIP, threat intel) 9. **Bot** consumes alert and sends to Slack channel ### 4. Real-Time Collaboration Flow (Meteor) ``` User 1 Browser User 2 Browser │ │ │ │ ▼ ▼ ┌─────────┐ ┌─────────┐ │ METEOR │ │ METEOR │ │ Client │ │ Client │ └────┬────┘ └────┬────┘ │ │ │ WebSocket/DDP │ │ (Real-time) │ │ │ └──────────┬──────────┘ │ ▼ ┌──────────────┐ │ METEOR │ │ Server │ │ (Node.js) │ └──────┬───────┘ │ │ Read/Write ▼ ┌──────────────┐ │ MONGODB │ │ (Incidents) │ └──────────────┘ ``` **Example: Two users working on same incident** 1. User 1 opens incident #123 in Meteor UI 2. User 2 opens same incident #123 3. Both users subscribe to MongoDB collection: `incidents` 4. User 1 adds a comment 5. Meteor server updates MongoDB 6. MongoDB change triggers Meteor reactivity 7. Both User 1 and User 2 see the comment appear **instantly** (no page refresh) --- ## Service Communication Patterns ### 1. HTTP/REST Communication **Used for:** - Meteor → REST API - External systems → Loginput - Kibana → Elasticsearch **Protocol:** HTTP/HTTPS **Format:** JSON **Example:** ```javascript // Meteor calling REST API HTTP.post('http://rest:8081/blockip', { data: {ip: '1.2.3.4', reason: 'malicious'} }); ``` ### 2. Message Queue Communication (RabbitMQ) **Used for:** - Loginput → MQ Workers - Alerts → Alert Actions - Alerts → Bot **Protocol:** AMQP (Advanced Message Queuing Protocol) **Format:** JSON messages **Exchanges:** - `eventtask`: Events from loginput - `mozdef.event`: Processed events (topic exchange) - `alerts`: Generated alerts **Example:** ```python # Loginput publishes to RabbitMQ mqConn.ensure(mqproducer, mqproducer.publish, max_retries=10) mqproducer.publish( eventDict, exchange=eventTaskExchange, routing_key=options.taskexchange ) # MQ Worker consumes from RabbitMQ taskConsumer(mqConn, eventTaskQueue, eventTopicExchange, es).run() ``` ### 3. Direct Database Access **Used for:** - REST API → Elasticsearch (queries) - REST API → MongoDB (incidents) - MQ Workers → Elasticsearch (indexing) - Alerts → Elasticsearch (queries) - Meteor → MongoDB (real-time subscriptions) **Protocol:** Native client libraries - Elasticsearch: `elasticsearch` Python client - MongoDB: `pymongo` (Python) or native (Meteor) **Example:** ```python # REST API querying Elasticsearch es = ElasticsearchClient(options.esservers) search_query = SearchQuery(minutes=20) search_query.add_must([TermMatch('category', 'authentication')]) results = es.query(search_query) ``` ### 4. WebSocket/DDP (Meteor) **Used for:** - Browser → Meteor server (real-time updates) **Protocol:** DDP (Distributed Data Protocol) over WebSocket **Format:** JSON **Example:** ```javascript // Meteor automatically handles WebSocket // When MongoDB changes, clients are notified incidents.find({_id: incidentId}).observe({ added: function(doc) { // Real-time update received } }); ``` --- ## How They Run Together ### Docker Compose Orchestration All services are orchestrated via `docker-compose.yml`: ```yaml services: nginx: # Reverse proxy elasticsearch: # Event storage rabbitmq: # Message queue mongodb: # Database kibana: # Visualization # MozDef Services loginput: # Event ingestion mq_worker: # Event processing rest: # REST API meteor: # Web interface alerts: # Alert engine alertactions: # Alert processing bot: # Notifications (optional) cron: # Scheduled tasks ``` ### Startup Sequence 1. **Infrastructure Services Start First:** ``` elasticsearch → rabbitmq → mongodb → kibana ``` 2. **Bootstrap Service:** - Waits for Elasticsearch to be ready - Creates initial indices - Sets up Kibana dashboards 3. **Backend Services Start:** ``` loginput → mq_worker → rest → alerts → alertactions ``` 4. **Frontend Services Start:** ``` meteor (depends on mongodb and rest) ``` 5. **Nginx Starts Last:** - Depends on all services being ready - Routes traffic to appropriate services ### Service Dependencies ``` nginx ├── meteor (port 80) ├── loginput (port 8080) ├── rest (port 8081) └── kibana (port 9090) meteor ├── mongodb (user data, incidents) └── rest (API calls) rest ├── elasticsearch (event queries) └── mongodb (incident storage) loginput └── rabbitmq (publish events) mq_worker ├── rabbitmq (consume events) └── elasticsearch (index events) alerts ├── elasticsearch (query events) ├── rabbitmq (publish alerts) └── rest (sync schedules) alertactions └── rabbitmq (consume alerts) kibana └── elasticsearch (direct connection) ``` ### Health Checks Each service has health checks to ensure proper startup: ```yaml healthcheck: test: ["CMD", "curl", "-f", "127.0.0.1:8080/status"] interval: 30s timeout: 3s retries: 10 ``` Services wait for dependencies to be healthy before starting. ### Configuration Integration **Environment Variables:** - Services read from environment variables - Docker Compose sets these via `environment:` section - Settings cascade: env vars → config files → defaults **Key Configuration Points:** - `OPTIONS_METEOR_ROOTAPI`: REST API URL (default: `http://rest:8081`) - `OPTIONS_METEOR_KIBANAURL`: Kibana URL (default: `http://localhost:9090/app/kibana`) - `OPTIONS_MQPROTOCOL`: Message queue protocol (`amqp` or `sqs`) ### Network Communication All services run in the same Docker network (`default`): - Services can communicate via service names (e.g., `http://rest:8081`) - Ports are exposed only where needed (80, 8080, 8081, 9090) - Internal ports (Elasticsearch 9200, RabbitMQ 5672) are not exposed --- ## Summary ### Frontend - **Meteor**: Real-time web UI (Node.js) - **Kibana**: Log visualization (JavaScript) ### Backend - **REST API**: Backend API (Python) - **Loginput**: Event ingestion (Python) - **MQ Workers**: Event processing (Python) - **Alerts**: Alert engine (Python/Celery) - **Alert Actions**: Alert processing (Python) - **Bot**: Notifications (Python) ### Services - **Elasticsearch**: Event storage - **RabbitMQ**: Message queue - **MongoDB**: Database - **Nginx**: Reverse proxy ### Integration Points 1. **HTTP**: Meteor ↔ REST API, External → Loginput 2. **RabbitMQ**: Loginput → MQ Workers, Alerts → Alert Actions 3. **Elasticsearch**: All services query/write events 4. **MongoDB**: Meteor stores incidents, REST API reads/writes 5. **WebSocket**: Browser ↔ Meteor (real-time) ### How They Run - Docker Compose orchestrates all services - Services start in dependency order - Health checks ensure proper startup - All services communicate via Docker network This architecture allows MozDef to: - **Scale horizontally** (multiple MQ workers, loginput instances) - **Process events asynchronously** (via RabbitMQ) - **Provide real-time collaboration** (via Meteor) - **Handle high event volumes** (300M+ events/day)