ARCHITECTURE_INTEGRATION_GUIDE.md

# MozDef Architecture: Frontend, Backend & Services Integration Guide

## Table of Contents
1. [Frontend vs Backend Overview](#frontend-vs-backend-overview)
2. [Services Breakdown](#services-breakdown)
3. [How Services Integrate](#how-services-integrate)
4. [Complete Data Flow](#complete-data-flow)
5. [Service Communication Patterns](#service-communication-patterns)
6. [How They Run Together](#how-they-run-together)

---

## Frontend vs Backend Overview

### Frontend Components (User-Facing)

**Frontend** = What users interact with directly

| Component | Technology | Purpose | Port |
|-----------|-----------|---------|------|
| **Meteor Web Interface** | Node.js/Meteor | Real-time web UI for incident handling | 80 (via Nginx) |
| **Kibana** | JavaScript/React | Log visualization and analysis | 9090 (via Nginx) |

**Key Characteristics:**
- **Meteor**: Runs on Node.js, provides real-time collaboration
- **Kibana**: Standalone visualization tool, connects directly to Elasticsearch
- Both are served through **Nginx** as a reverse proxy

### Backend Components (Processing & API)

**Backend** = Services that process data and provide APIs

| Component | Technology | Purpose | Port |
|-----------|-----------|---------|------|
| **REST API** | Python/Bottle.py | RESTful API for web interface | 8081 |
| **Loginput** | Python/Bottle.py | HTTP endpoint for event ingestion | 8080 |
| **MQ Workers** | Python | Process events from RabbitMQ queue | N/A (internal) |
| **Alerts Engine** | Python/Celery | Alert correlation and generation | N/A (internal) |
| **Alert Actions** | Python | Post-alert processing | N/A (internal) |
| **Bot** | Python | Slack/IRC notifications | N/A (internal) |

**Key Characteristics:**
- All backend services are **Python-based**
- Use **uWSGI** for process management
- Communicate via **RabbitMQ** message queues
- Query/write to **Elasticsearch** and **MongoDB**

---

## Services Breakdown

### Infrastructure Services (Supporting Services)

These are the foundational services that other components depend on:

| Service | Technology | Purpose | Port |
|---------|-----------|---------|------|
| **Elasticsearch** | Java | Event storage, indexing, search | 9200 (internal) |
| **RabbitMQ** | Erlang | Message queue for event processing | 5672 (internal), 15672 (admin) |
| **MongoDB** | C++ | Database for Meteor (incidents, users) | 3002 (internal) |
| **Nginx** | C | Reverse proxy and web server | 80, 8080, 8081, 9090 |

### Application Services (MozDef Components)

These are the MozDef-specific services:

#### **Ingest Services** (Data Ingestion)
- **Loginput**: Receives JSON events via HTTP
- **MQ Workers**: Process events from queue
- **Syslog**: Receives syslog messages

#### **Alert Services** (Security Alerting)
- **Alerts**: Celery-based alert engine
- **Alert Actions**: Post-alert processing
- **Bot**: Chat notifications

#### **Web Services** (User Interface)
- **REST API**: Backend API for web interface
- **Meteor**: Frontend web application
- **Kibana**: Log visualization (separate service)

---

## How Services Integrate

### Integration Architecture Diagram

```
┌─────────────────────────────────────────────────────────────────┐
│                         USER BROWSER                             │
└───────────────┬─────────────────────────────────────────────────┘
                │ HTTP/WebSocket
                ▼
┌─────────────────────────────────────────────────────────────────┐
│                         NGINX (Reverse Proxy)                     │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐        │
│  │ Port 80  │  │ Port 8080│  │ Port 8081│  │ Port 9090│        │
│  │ Meteor   │  │ Loginput │  │ REST API │  │ Kibana   │        │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘  └────┬─────┘        │
└───────┼─────────────┼──────────────┼─────────────┼──────────────┘
        │             │              │             │
        ▼             ▼              ▼             ▼
┌─────────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐
│   METEOR    │  │ LOGINPUT │  │ REST API │  │  KIBANA  │
│  (Node.js)  │  │ (Python) │  │ (Python) │  │          │
└──────┬──────┘  └────┬─────┘  └────┬─────┘  └────┬─────┘
       │              │              │              │
       │              │              │              │
       │              ▼              │              │
       │      ┌──────────────┐       │              │
       │      │   RABBITMQ   │       │              │
       │      │  (Message    │       │              │
       │      │   Queue)     │       │              │
       │      └──────┬───────┘       │              │
       │             │               │              │
       │             ▼               │              │
       │      ┌──────────────┐       │              │
       │      │  MQ WORKERS  │       │              │
       │      │   (Python)   │       │              │
       │      └──────┬───────┘       │              │
       │             │               │              │
       │             │               │              │
       └─────────────┼───────────────┼──────────────┘
                     │               │
                     ▼               ▼
              ┌──────────────┐  ┌──────────────┐
              │ ELASTICSEARCH│  │   MONGODB    │
              │  (Storage)   │  │  (Database)  │
              └──────┬───────┘  └──────┬───────┘
                     │                  │
                     │                  │
                     ▼                  │
              ┌──────────────┐          │
              │   ALERTS     │          │
              │  (Celery)    │          │
              └──────┬───────┘          │
                     │                  │
                     ▼                  │
              ┌──────────────┐          │
              │ ALERT ACTIONS│          │
              │   & BOT      │          │
              └──────────────┘          │
```

---

## Complete Data Flow

### 1. Event Ingestion Flow

```
External Log Shipper
    │
    │ JSON over HTTP POST
    ▼
┌─────────────────┐
│     NGINX       │  (Port 8080)
│   Reverse Proxy │
└────────┬────────┘
         │
         │ uWSGI socket
         ▼
┌─────────────────┐
│    LOGINPUT     │  (Python/Bottle.py)
│  - Receives JSON│
│  - Validates    │
│  - Adds metadata│
└────────┬────────┘
         │
         │ Publishes to RabbitMQ
         ▼
┌─────────────────┐
│    RABBITMQ     │  (Message Queue)
│  Exchange:      │
│  eventtask      │
└────────┬────────┘
         │
         │ Consumes from queue
         ▼
┌─────────────────┐
│   MQ WORKERS    │  (Python)
│  - Normalize    │
│  - Enrich       │
│  - GeoIP        │
│  - Watchlist    │
└────────┬────────┘
         │
         │ Index document
         ▼
┌─────────────────┐
│  ELASTICSEARCH  │
│  - Stores event │
│  - Indexes      │
│  - Searchable   │
└─────────────────┘
```

### 2. Frontend-Backend Communication Flow

```
User Browser
    │
    │ HTTP Request
    ▼
┌─────────────────┐
│     NGINX       │  (Port 80)
│   Reverse Proxy │
└────────┬────────┘
         │
         │ Proxy to Meteor
         ▼
┌─────────────────┐
│     METEOR      │  (Node.js)
│  Web Interface  │
│                 │
│  Client Code:   │
│  - UI Components│
│  - Reactivity   │
│                 │
│  Server Code:   │
│  - Methods      │
│  - HTTP calls   │
└────────┬────────┘
         │
         │ HTTP POST/GET
         │ (via mozdef.rootAPI)
         ▼
┌─────────────────┐
│    REST API     │  (Python/Bottle.py)
│  Port 8081      │
│                 │
│  Endpoints:     │
│  - /events      │
│  - /alerts      │
│  - /incidents   │
│  - /watchitem   │
│  - /blockip     │
└────────┬────────┘
         │
         │ Query/Write
         ├──────────────┐
         ▼              ▼
┌──────────────┐  ┌──────────────┐
│ ELASTICSEARCH│  │   MONGODB    │
│  (Events)    │  │ (Incidents)  │
└──────────────┘  └──────────────┘
```

**Example: User blocks an IP address**

1. User clicks "Block IP" in Meteor UI
2. Meteor client calls Meteor method: `Meteor.call('blockip', {ip: '1.2.3.4'})`
3. Meteor server method executes: `blockIP(formobj)`
4. Meteor server makes HTTP POST: `HTTP.post(mozdef.rootAPI + '/blockip', {data: formobj})`
5. REST API receives POST at `/blockip`
6. REST API processes request (may call plugins, update watchlist, etc.)
7. REST API returns response
8. Meteor receives response and updates UI reactively

### 3. Alert Processing Flow

```
┌─────────────────┐
│  ELASTICSEARCH  │
│  (Event Store)  │
└────────┬────────┘
         │
         │ Periodic Query
         │ (Every X minutes)
         ▼
┌─────────────────┐
│     ALERTS      │  (Python/Celery)
│  Alert Engine   │
│                 │
│  - Queries ES   │
│  - Correlates   │
│  - Aggregates   │
│  - Generates    │
└────────┬────────┘
         │
         │ Creates alert document
         ├──────────────────────┐
         ▼                      ▼
┌──────────────┐        ┌──────────────┐
│ ELASTICSEARCH│        │   RABBITMQ   │
│ (alerts-*)   │        │ (alerts)     │
└──────────────┘        └──────┬───────┘
                                │
                                │ Consume alert
                                ▼
                        ┌──────────────┐
                        │ ALERT ACTIONS │
                        │   (Python)    │
                        │               │
                        │ - Notify      │
                        │ - Enrich      │
                        │ - Respond     │
                        └──────┬───────┘
                               │
                               ▼
                        ┌──────────────┐
                        │     BOT      │
                        │  (Slack/IRC) │
                        │               │
                        │ Sends message│
                        └──────────────┘
```

**Example: SSH Brute Force Alert**

1. **Alerts Engine** (Celery) runs every 5 minutes
2. Queries Elasticsearch: "Find SSH failed login events in last 5 minutes"
3. Aggregates by source IP: "Count failures per IP"
4. If IP has > 10 failures: Create alert
5. Alert saved to Elasticsearch `alerts-*` index
6. Alert published to RabbitMQ `alerts` exchange
7. **Alert Actions** worker consumes alert
8. **Alert Actions** enriches alert (GeoIP, threat intel)
9. **Bot** consumes alert and sends to Slack channel

### 4. Real-Time Collaboration Flow (Meteor)

```
User 1 Browser          User 2 Browser
    │                      │
    │                      │
    ▼                      ▼
┌─────────┐          ┌─────────┐
│ METEOR  │          │ METEOR  │
│ Client  │          │ Client  │
└────┬────┘          └────┬────┘
     │                     │
     │ WebSocket/DDP       │
     │ (Real-time)         │
     │                     │
     └──────────┬──────────┘
                │
                ▼
        ┌──────────────┐
        │   METEOR     │
        │   Server     │
        │  (Node.js)   │
        └──────┬───────┘
               │
               │ Read/Write
               ▼
        ┌──────────────┐
        │   MONGODB    │
        │  (Incidents) │
        └──────────────┘
```

**Example: Two users working on same incident**

1. User 1 opens incident #123 in Meteor UI
2. User 2 opens same incident #123
3. Both users subscribe to MongoDB collection: `incidents`
4. User 1 adds a comment
5. Meteor server updates MongoDB
6. MongoDB change triggers Meteor reactivity
7. Both User 1 and User 2 see the comment appear **instantly** (no page refresh)

---

## Service Communication Patterns

### 1. HTTP/REST Communication

**Used for:**
- Meteor → REST API
- External systems → Loginput
- Kibana → Elasticsearch

**Protocol:** HTTP/HTTPS
**Format:** JSON

**Example:**
```javascript
// Meteor calling REST API
HTTP.post('http://rest:8081/blockip', {
    data: {ip: '1.2.3.4', reason: 'malicious'}
});
```

### 2. Message Queue Communication (RabbitMQ)

**Used for:**
- Loginput → MQ Workers
- Alerts → Alert Actions
- Alerts → Bot

**Protocol:** AMQP (Advanced Message Queuing Protocol)
**Format:** JSON messages

**Exchanges:**
- `eventtask`: Events from loginput
- `mozdef.event`: Processed events (topic exchange)
- `alerts`: Generated alerts

**Example:**
```python
# Loginput publishes to RabbitMQ
mqConn.ensure(mqproducer, mqproducer.publish, max_retries=10)
mqproducer.publish(
    eventDict,
    exchange=eventTaskExchange,
    routing_key=options.taskexchange
)

# MQ Worker consumes from RabbitMQ
taskConsumer(mqConn, eventTaskQueue, eventTopicExchange, es).run()
```

### 3. Direct Database Access

**Used for:**
- REST API → Elasticsearch (queries)
- REST API → MongoDB (incidents)
- MQ Workers → Elasticsearch (indexing)
- Alerts → Elasticsearch (queries)
- Meteor → MongoDB (real-time subscriptions)

**Protocol:** Native client libraries
- Elasticsearch: `elasticsearch` Python client
- MongoDB: `pymongo` (Python) or native (Meteor)

**Example:**
```python
# REST API querying Elasticsearch
es = ElasticsearchClient(options.esservers)
search_query = SearchQuery(minutes=20)
search_query.add_must([TermMatch('category', 'authentication')])
results = es.query(search_query)
```

### 4. WebSocket/DDP (Meteor)

**Used for:**
- Browser → Meteor server (real-time updates)

**Protocol:** DDP (Distributed Data Protocol) over WebSocket
**Format:** JSON

**Example:**
```javascript
// Meteor automatically handles WebSocket
// When MongoDB changes, clients are notified
incidents.find({_id: incidentId}).observe({
    added: function(doc) {
        // Real-time update received
    }
});
```

---

## How They Run Together

### Docker Compose Orchestration

All services are orchestrated via `docker-compose.yml`:

```yaml
services:
  nginx:          # Reverse proxy
  elasticsearch:  # Event storage
  rabbitmq:       # Message queue
  mongodb:        # Database
  kibana:         # Visualization
  
  # MozDef Services
  loginput:       # Event ingestion
  mq_worker:      # Event processing
  rest:           # REST API
  meteor:         # Web interface
  alerts:         # Alert engine
  alertactions:   # Alert processing
  bot:            # Notifications (optional)
  cron:           # Scheduled tasks
```

### Startup Sequence

1. **Infrastructure Services Start First:**
   ```
   elasticsearch → rabbitmq → mongodb → kibana
   ```

2. **Bootstrap Service:**
   - Waits for Elasticsearch to be ready
   - Creates initial indices
   - Sets up Kibana dashboards

3. **Backend Services Start:**
   ```
   loginput → mq_worker → rest → alerts → alertactions
   ```

4. **Frontend Services Start:**
   ```
   meteor (depends on mongodb and rest)
   ```

5. **Nginx Starts Last:**
   - Depends on all services being ready
   - Routes traffic to appropriate services

### Service Dependencies

```
nginx
  ├── meteor (port 80)
  ├── loginput (port 8080)
  ├── rest (port 8081)
  └── kibana (port 9090)

meteor
  ├── mongodb (user data, incidents)
  └── rest (API calls)

rest
  ├── elasticsearch (event queries)
  └── mongodb (incident storage)

loginput
  └── rabbitmq (publish events)

mq_worker
  ├── rabbitmq (consume events)
  └── elasticsearch (index events)

alerts
  ├── elasticsearch (query events)
  ├── rabbitmq (publish alerts)
  └── rest (sync schedules)

alertactions
  └── rabbitmq (consume alerts)

kibana
  └── elasticsearch (direct connection)
```

### Health Checks

Each service has health checks to ensure proper startup:

```yaml
healthcheck:
  test: ["CMD", "curl", "-f", "127.0.0.1:8080/status"]
  interval: 30s
  timeout: 3s
  retries: 10
```

Services wait for dependencies to be healthy before starting.

### Configuration Integration

**Environment Variables:**
- Services read from environment variables
- Docker Compose sets these via `environment:` section
- Settings cascade: env vars → config files → defaults

**Key Configuration Points:**
- `OPTIONS_METEOR_ROOTAPI`: REST API URL (default: `http://rest:8081`)
- `OPTIONS_METEOR_KIBANAURL`: Kibana URL (default: `http://localhost:9090/app/kibana`)
- `OPTIONS_MQPROTOCOL`: Message queue protocol (`amqp` or `sqs`)

### Network Communication

All services run in the same Docker network (`default`):
- Services can communicate via service names (e.g., `http://rest:8081`)
- Ports are exposed only where needed (80, 8080, 8081, 9090)
- Internal ports (Elasticsearch 9200, RabbitMQ 5672) are not exposed

---

## Summary

### Frontend
- **Meteor**: Real-time web UI (Node.js)
- **Kibana**: Log visualization (JavaScript)

### Backend
- **REST API**: Backend API (Python)
- **Loginput**: Event ingestion (Python)
- **MQ Workers**: Event processing (Python)
- **Alerts**: Alert engine (Python/Celery)
- **Alert Actions**: Alert processing (Python)
- **Bot**: Notifications (Python)

### Services
- **Elasticsearch**: Event storage
- **RabbitMQ**: Message queue
- **MongoDB**: Database
- **Nginx**: Reverse proxy

### Integration Points
1. **HTTP**: Meteor ↔ REST API, External → Loginput
2. **RabbitMQ**: Loginput → MQ Workers, Alerts → Alert Actions
3. **Elasticsearch**: All services query/write events
4. **MongoDB**: Meteor stores incidents, REST API reads/writes
5. **WebSocket**: Browser ↔ Meteor (real-time)

### How They Run
- Docker Compose orchestrates all services
- Services start in dependency order
- Health checks ensure proper startup
- All services communicate via Docker network

This architecture allows MozDef to:
- **Scale horizontally** (multiple MQ workers, loginput instances)
- **Process events asynchronously** (via RabbitMQ)
- **Provide real-time collaboration** (via Meteor)
- **Handle high event volumes** (300M+ events/day)