ARCHITECTURE_INTEGRATION_GUIDE.md

MozDef Architecture: Frontend, Backend & Services Integration Guide

Table of Contents

1. Frontend vs Backend Overview
2. Services Breakdown
3. How Services Integrate
4. Complete Data Flow
5. Service Communication Patterns
6. How They Run Together

---

Frontend vs Backend Overview

Frontend Components (User-Facing)

Frontend = What users interact with directly

Component	Technology	Purpose	Port
Meteor Web Interface	Node.js/Meteor	Real-time web UI for incident handling	80 (via Nginx)
Kibana	JavaScript/React	Log visualization and analysis	9090 (via Nginx)

Key Characteristics:

• Meteor: Runs on Node.js, provides real-time collaboration
• Kibana: Standalone visualization tool, connects directly to Elasticsearch
• Both are served through Nginx as a reverse proxy

Backend Components (Processing & API)

Backend = Services that process data and provide APIs

Component	Technology	Purpose	Port
REST API	Python/Bottle.py	RESTful API for web interface	8081
Loginput	Python/Bottle.py	HTTP endpoint for event ingestion	8080
MQ Workers	Python	Process events from RabbitMQ queue	N/A (internal)
Alerts Engine	Python/Celery	Alert correlation and generation	N/A (internal)
Alert Actions	Python	Post-alert processing	N/A (internal)
Bot	Python	Slack/IRC notifications	N/A (internal)

Key Characteristics:

• All backend services are Python-based
• Use uWSGI for process management
• Communicate via RabbitMQ message queues
• Query/write to Elasticsearch and MongoDB

---

Services Breakdown

Infrastructure Services (Supporting Services)

These are the foundational services that other components depend on:

Service	Technology	Purpose	Port
Elasticsearch	Java	Event storage, indexing, search	9200 (internal)
RabbitMQ	Erlang	Message queue for event processing	5672 (internal), 15672 (admin)
MongoDB	C++	Database for Meteor (incidents, users)	3002 (internal)
Nginx	C	Reverse proxy and web server	80, 8080, 8081, 9090

Application Services (MozDef Components)

These are the MozDef-specific services:

Ingest Services (Data Ingestion)

• Loginput: Receives JSON events via HTTP
• MQ Workers: Process events from queue
• Syslog: Receives syslog messages

Alert Services (Security Alerting)

• Alerts: Celery-based alert engine
• Alert Actions: Post-alert processing
• Bot: Chat notifications

Web Services (User Interface)

• REST API: Backend API for web interface
• Meteor: Frontend web application
• Kibana: Log visualization (separate service)

---

How Services Integrate

Integration Architecture Diagram

┌─────────────────────────────────────────────────────────────────┐
│                         USER BROWSER                             │
└───────────────┬─────────────────────────────────────────────────┘
                │ HTTP/WebSocket
                ▼
┌─────────────────────────────────────────────────────────────────┐
│                         NGINX (Reverse Proxy)                     │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐        │
│  │ Port 80  │  │ Port 8080│  │ Port 8081│  │ Port 9090│        │
│  │ Meteor   │  │ Loginput │  │ REST API │  │ Kibana   │        │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘  └────┬─────┘        │
└───────┼─────────────┼──────────────┼─────────────┼──────────────┘
        │             │              │             │
        ▼             ▼              ▼             ▼
┌─────────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐
│   METEOR    │  │ LOGINPUT │  │ REST API │  │  KIBANA  │
│  (Node.js)  │  │ (Python) │  │ (Python) │  │          │
└──────┬──────┘  └────┬─────┘  └────┬─────┘  └────┬─────┘
       │              │              │              │
       │              │              │              │
       │              ▼              │              │
       │      ┌──────────────┐       │              │
       │      │   RABBITMQ   │       │              │
       │      │  (Message    │       │              │
       │      │   Queue)     │       │              │
       │      └──────┬───────┘       │              │
       │             │               │              │
       │             ▼               │              │
       │      ┌──────────────┐       │              │
       │      │  MQ WORKERS  │       │              │
       │      │   (Python)   │       │              │
       │      └──────┬───────┘       │              │
       │             │               │              │
       │             │               │              │
       └─────────────┼───────────────┼──────────────┘
                     │               │
                     ▼               ▼
              ┌──────────────┐  ┌──────────────┐
              │ ELASTICSEARCH│  │   MONGODB    │
              │  (Storage)   │  │  (Database)  │
              └──────┬───────┘  └──────┬───────┘
                     │                  │
                     │                  │
                     ▼                  │
              ┌──────────────┐          │
              │   ALERTS     │          │
              │  (Celery)    │          │
              └──────┬───────┘          │
                     │                  │
                     ▼                  │
              ┌──────────────┐          │
              │ ALERT ACTIONS│          │
              │   & BOT      │          │
              └──────────────┘          │

---

Complete Data Flow

1. Event Ingestion Flow

External Log Shipper
    │
    │ JSON over HTTP POST
    ▼
┌─────────────────┐
│     NGINX       │  (Port 8080)
│   Reverse Proxy │
└────────┬────────┘
         │
         │ uWSGI socket
         ▼
┌─────────────────┐
│    LOGINPUT     │  (Python/Bottle.py)
│  - Receives JSON│
│  - Validates    │
│  - Adds metadata│
└────────┬────────┘
         │
         │ Publishes to RabbitMQ
         ▼
┌─────────────────┐
│    RABBITMQ     │  (Message Queue)
│  Exchange:      │
│  eventtask      │
└────────┬────────┘
         │
         │ Consumes from queue
         ▼
┌─────────────────┐
│   MQ WORKERS    │  (Python)
│  - Normalize    │
│  - Enrich       │
│  - GeoIP        │
│  - Watchlist    │
└────────┬────────┘
         │
         │ Index document
         ▼
┌─────────────────┐
│  ELASTICSEARCH  │
│  - Stores event │
│  - Indexes      │
│  - Searchable   │
└─────────────────┘

2. Frontend-Backend Communication Flow

User Browser
    │
    │ HTTP Request
    ▼
┌─────────────────┐
│     NGINX       │  (Port 80)
│   Reverse Proxy │
└────────┬────────┘
         │
         │ Proxy to Meteor
         ▼
┌─────────────────┐
│     METEOR      │  (Node.js)
│  Web Interface  │
│                 │
│  Client Code:   │
│  - UI Components│
│  - Reactivity   │
│                 │
│  Server Code:   │
│  - Methods      │
│  - HTTP calls   │
└────────┬────────┘
         │
         │ HTTP POST/GET
         │ (via mozdef.rootAPI)
         ▼
┌─────────────────┐
│    REST API     │  (Python/Bottle.py)
│  Port 8081      │
│                 │
│  Endpoints:     │
│  - /events      │
│  - /alerts      │
│  - /incidents   │
│  - /watchitem   │
│  - /blockip     │
└────────┬────────┘
         │
         │ Query/Write
         ├──────────────┐
         ▼              ▼
┌──────────────┐  ┌──────────────┐
│ ELASTICSEARCH│  │   MONGODB    │
│  (Events)    │  │ (Incidents)  │
└──────────────┘  └──────────────┘

Example: User blocks an IP address

1. User clicks "Block IP" in Meteor UI
2. Meteor client calls Meteor method: Meteor.call('blockip', {ip: '1.2.3.4'})
3. Meteor server method executes: blockIP(formobj)
4. Meteor server makes HTTP POST: HTTP.post(mozdef.rootAPI + '/blockip', {data: formobj})
5. REST API receives POST at /blockip
6. REST API processes request (may call plugins, update watchlist, etc.)
7. REST API returns response
8. Meteor receives response and updates UI reactively

3. Alert Processing Flow

┌─────────────────┐
│  ELASTICSEARCH  │
│  (Event Store)  │
└────────┬────────┘
         │
         │ Periodic Query
         │ (Every X minutes)
         ▼
┌─────────────────┐
│     ALERTS      │  (Python/Celery)
│  Alert Engine   │
│                 │
│  - Queries ES   │
│  - Correlates   │
│  - Aggregates   │
│  - Generates    │
└────────┬────────┘
         │
         │ Creates alert document
         ├──────────────────────┐
         ▼                      ▼
┌──────────────┐        ┌──────────────┐
│ ELASTICSEARCH│        │   RABBITMQ   │
│ (alerts-*)   │        │ (alerts)     │
└──────────────┘        └──────┬───────┘
                                │
                                │ Consume alert
                                ▼
                        ┌──────────────┐
                        │ ALERT ACTIONS │
                        │   (Python)    │
                        │               │
                        │ - Notify      │
                        │ - Enrich      │
                        │ - Respond     │
                        └──────┬───────┘
                               │
                               ▼
                        ┌──────────────┐
                        │     BOT      │
                        │  (Slack/IRC) │
                        │               │
                        │ Sends message│
                        └──────────────┘

Example: SSH Brute Force Alert

1. Alerts Engine (Celery) runs every 5 minutes
2. Queries Elasticsearch: "Find SSH failed login events in last 5 minutes"
3. Aggregates by source IP: "Count failures per IP"
4. If IP has > 10 failures: Create alert
5. Alert saved to Elasticsearch alerts-* index
6. Alert published to RabbitMQ alerts exchange
7. Alert Actions worker consumes alert
8. Alert Actions enriches alert (GeoIP, threat intel)
9. Bot consumes alert and sends to Slack channel

4. Real-Time Collaboration Flow (Meteor)

User 1 Browser          User 2 Browser
    │                      │
    │                      │
    ▼                      ▼
┌─────────┐          ┌─────────┐
│ METEOR  │          │ METEOR  │
│ Client  │          │ Client  │
└────┬────┘          └────┬────┘
     │                     │
     │ WebSocket/DDP       │
     │ (Real-time)         │
     │                     │
     └──────────┬──────────┘
                │
                ▼
        ┌──────────────┐
        │   METEOR     │
        │   Server     │
        │  (Node.js)   │
        └──────┬───────┘
               │
               │ Read/Write
               ▼
        ┌──────────────┐
        │   MONGODB    │
        │  (Incidents) │
        └──────────────┘

Example: Two users working on same incident

1. User 1 opens incident #123 in Meteor UI
2. User 2 opens same incident #123
3. Both users subscribe to MongoDB collection: incidents
4. User 1 adds a comment
5. Meteor server updates MongoDB
6. MongoDB change triggers Meteor reactivity
7. Both User 1 and User 2 see the comment appear instantly (no page refresh)

---

Service Communication Patterns

1. HTTP/REST Communication

Used for:

• Meteor → REST API
• External systems → Loginput
• Kibana → Elasticsearch

Protocol: HTTP/HTTPS Format: JSON

Example:

// Meteor calling REST API
HTTP.post('http://rest:8081/blockip', {
    data: {ip: '1.2.3.4', reason: 'malicious'}
});

2. Message Queue Communication (RabbitMQ)

Used for:

• Loginput → MQ Workers
• Alerts → Alert Actions
• Alerts → Bot

Protocol: AMQP (Advanced Message Queuing Protocol) Format: JSON messages

Exchanges:

• eventtask: Events from loginput
• mozdef.event: Processed events (topic exchange)
• alerts: Generated alerts

Example:

# Loginput publishes to RabbitMQ
mqConn.ensure(mqproducer, mqproducer.publish, max_retries=10)
mqproducer.publish(
    eventDict,
    exchange=eventTaskExchange,
    routing_key=options.taskexchange
)

# MQ Worker consumes from RabbitMQ
taskConsumer(mqConn, eventTaskQueue, eventTopicExchange, es).run()

3. Direct Database Access

Used for:

• REST API → Elasticsearch (queries)
• REST API → MongoDB (incidents)
• MQ Workers → Elasticsearch (indexing)
• Alerts → Elasticsearch (queries)
• Meteor → MongoDB (real-time subscriptions)

Protocol: Native client libraries

• Elasticsearch: elasticsearch Python client
• MongoDB: pymongo (Python) or native (Meteor)

Example:

# REST API querying Elasticsearch
es = ElasticsearchClient(options.esservers)
search_query = SearchQuery(minutes=20)
search_query.add_must([TermMatch('category', 'authentication')])
results = es.query(search_query)

4. WebSocket/DDP (Meteor)

Used for:

• Browser → Meteor server (real-time updates)

Protocol: DDP (Distributed Data Protocol) over WebSocket Format: JSON

Example:

// Meteor automatically handles WebSocket
// When MongoDB changes, clients are notified
incidents.find({_id: incidentId}).observe({
    added: function(doc) {
        // Real-time update received
    }
});

---

How They Run Together

Docker Compose Orchestration

All services are orchestrated via docker-compose.yml:

services:
  nginx:          # Reverse proxy
  elasticsearch:  # Event storage
  rabbitmq:       # Message queue
  mongodb:        # Database
  kibana:         # Visualization
  
  # MozDef Services
  loginput:       # Event ingestion
  mq_worker:      # Event processing
  rest:           # REST API
  meteor:         # Web interface
  alerts:         # Alert engine
  alertactions:   # Alert processing
  bot:            # Notifications (optional)
  cron:           # Scheduled tasks

Startup Sequence

1. Infrastructure Services Start First:

   elasticsearch → rabbitmq → mongodb → kibana
   

2. Bootstrap Service:

   • Waits for Elasticsearch to be ready
   • Creates initial indices
   • Sets up Kibana dashboards

3. Backend Services Start:

   loginput → mq_worker → rest → alerts → alertactions
   

4. Frontend Services Start:

   meteor (depends on mongodb and rest)
   

5. Nginx Starts Last:

   • Depends on all services being ready
   • Routes traffic to appropriate services

Service Dependencies

nginx
  ├── meteor (port 80)
  ├── loginput (port 8080)
  ├── rest (port 8081)
  └── kibana (port 9090)

meteor
  ├── mongodb (user data, incidents)
  └── rest (API calls)

rest
  ├── elasticsearch (event queries)
  └── mongodb (incident storage)

loginput
  └── rabbitmq (publish events)

mq_worker
  ├── rabbitmq (consume events)
  └── elasticsearch (index events)

alerts
  ├── elasticsearch (query events)
  ├── rabbitmq (publish alerts)
  └── rest (sync schedules)

alertactions
  └── rabbitmq (consume alerts)

kibana
  └── elasticsearch (direct connection)

Health Checks

Each service has health checks to ensure proper startup:

healthcheck:
  test: ["CMD", "curl", "-f", "127.0.0.1:8080/status"]
  interval: 30s
  timeout: 3s
  retries: 10

Services wait for dependencies to be healthy before starting.

Configuration Integration

Environment Variables:

• Services read from environment variables
• Docker Compose sets these via environment: section
• Settings cascade: env vars → config files → defaults

Key Configuration Points:

• OPTIONS_METEOR_ROOTAPI: REST API URL (default: http://rest:8081)
• OPTIONS_METEOR_KIBANAURL: Kibana URL (default: http://localhost:9090/app/kibana)
• OPTIONS_MQPROTOCOL: Message queue protocol (amqp or sqs)

Network Communication

All services run in the same Docker network (default):

• Services can communicate via service names (e.g., http://rest:8081)
• Ports are exposed only where needed (80, 8080, 8081, 9090)
• Internal ports (Elasticsearch 9200, RabbitMQ 5672) are not exposed

---

Summary

Frontend

• Meteor: Real-time web UI (Node.js)
• Kibana: Log visualization (JavaScript)

Backend

• REST API: Backend API (Python)
• Loginput: Event ingestion (Python)
• MQ Workers: Event processing (Python)
• Alerts: Alert engine (Python/Celery)
• Alert Actions: Alert processing (Python)
• Bot: Notifications (Python)

Services

• Elasticsearch: Event storage
• RabbitMQ: Message queue
• MongoDB: Database
• Nginx: Reverse proxy

Integration Points

1. HTTP: Meteor ↔ REST API, External → Loginput
2. RabbitMQ: Loginput → MQ Workers, Alerts → Alert Actions
3. Elasticsearch: All services query/write events
4. MongoDB: Meteor stores incidents, REST API reads/writes
5. WebSocket: Browser ↔ Meteor (real-time)

How They Run

• Docker Compose orchestrates all services
• Services start in dependency order
• Health checks ensure proper startup
• All services communicate via Docker network

This architecture allows MozDef to:

• Scale horizontally (multiple MQ workers, loginput instances)
• Process events asynchronously (via RabbitMQ)
• Provide real-time collaboration (via Meteor)
• Handle high event volumes (300M+ events/day)