DETAILED_RUN_GUIDE.md

# MozDef: Complete Detailed Run Guide

## Table of Contents
1. [Prerequisites & System Requirements](#prerequisites--system-requirements)
2. [Complete Installation Process](#complete-installation-process)
3. [Component-by-Component Setup](#component-by-component-setup)
4. [Starting Services in Order](#starting-services-in-order)
5. [Verification & Testing](#verification--testing)
6. [Configuration Deep Dive](#configuration-deep-dive)
7. [Troubleshooting Each Component](#troubleshooting-each-component)
8. [Development Mode Setup](#development-mode-setup)

---

## Prerequisites & System Requirements

### System Requirements

**Minimum Requirements:**
- **CPU**: 4 cores (8 recommended)
- **RAM**: 8GB minimum (16GB recommended)
- **Disk**: 50GB free space (100GB+ for production)
- **OS**: Linux (CentOS 7, Ubuntu 18.04+, or Docker-compatible)

**Docker Requirements:**
- Docker Engine 20.10+
- Docker Compose 1.29+
- At least 4GB RAM allocated to Docker daemon
- Docker storage driver: `aufs` (recommended for Mac OS X)

### Software Prerequisites

**For Docker Installation (Recommended):**
```bash
# Check Docker version
docker --version
# Should be: Docker version 20.10.x or higher

# Check Docker Compose version
docker-compose --version
# Should be: docker-compose version 1.29.x or higher

# Check available memory
docker info | grep "Total Memory"
# Should show at least 4GB available
```

**For Manual Installation:**
- Python 3.6+
- Node.js 8.x (for Meteor)
- Elasticsearch 6.8
- RabbitMQ 3.7+
- MongoDB 3.6+
- Nginx
- Make utility

### Network Requirements

**Ports to Open:**
- `80` - Meteor web interface (HTTP)
- `8080` - Loginput endpoint
- `8081` - REST API (optional, internal)
- `9090` - Kibana
- `514` - Syslog (UDP/TCP)

**Internal Ports (Docker network only):**
- `9200` - Elasticsearch
- `5672` - RabbitMQ AMQP
- `15672` - RabbitMQ Management
- `3002` - MongoDB
- `5601` - Kibana (internal)
- `3000` - Meteor (internal)

---

## Complete Installation Process

### Method 1: Docker Installation (Recommended)

#### Step 1: Clone/Download MozDef

```bash
# If you already have MozDef
cd /root/MozDef

# Or clone from GitHub
git clone https://github.com/mozilla/MozDef.git
cd MozDef
```

#### Step 2: Verify Directory Structure

```bash
# Check key directories exist
ls -la
# Should see: alerts, bot, config, docker, loginput, meteor, mq, rest, etc.

# Check Makefile exists
ls -la Makefile
```

#### Step 3: Build Docker Images

**Option A: Build from Source (Takes 15-30 minutes)**
```bash
# Build all images from source
make build

# This builds:
# - mozdef_base (base Python environment)
# - mozdef_elasticsearch
# - mozdef_rabbitmq
# - mozdef_mongodb
# - mozdef_kibana
# - mozdef_nginx
# - mozdef_bootstrap
# - mozdef_loginput
# - mozdef_mq_worker
# - mozdef_rest
# - mozdef_meteor
# - mozdef_alerts
# - mozdef_alertactions
# - mozdef_cron
# - mozdef_syslog
```

**Option B: Pull Pre-built Images (Faster, 5-10 minutes)**
```bash
# Pull pre-built images from Docker Hub
make build BUILD_MODE=pull

# This downloads images from hub.docker.com/mozdef/*
```

**Option C: Build Without Cache (Clean Build)**
```bash
# Force rebuild everything from scratch
make build NO_CACHE=--no-cache
```

#### Step 4: Verify Images Built

```bash
# List all MozDef images
docker images | grep mozdef

# Should see:
# mozdef/mozdef_base
# mozdef/mozdef_elasticsearch
# mozdef/mozdef_rabbitmq
# mozdef/mozdef_mongodb
# mozdef/mozdef_kibana
# mozdef/mozdef_nginx
# mozdef/mozdef_bootstrap
# mozdef/mozdef_loginput
# mozdef/mozdef_mq_worker
# mozdef/mozdef_rest
# mozdef/mozdef_meteor
# mozdef/mozdef_alerts
# mozdef/mozdef_alertactions
# mozdef/mozdef_cron
# mozdef/mozdef_syslog
```

#### Step 5: Start All Services

```bash
# Start all services with docker-compose
make run

# This command:
# 1. Creates Docker network (mozdef_default)
# 2. Creates Docker volumes (elasticsearch, rabbitmq, mongodb, geolite_db)
# 3. Starts services in dependency order:
#    - Infrastructure: elasticsearch, rabbitmq, mongodb
#    - Bootstrap: Sets up Elasticsearch indices
#    - Backend: loginput, mq_worker, rest, alerts, alertactions
#    - Frontend: meteor, kibana
#    - Proxy: nginx
```

#### Step 6: Monitor Startup

```bash
# Watch all services starting
docker-compose -f docker/compose/docker-compose.yml -p mozdef ps

# Watch logs in real-time
docker-compose -f docker/compose/docker-compose.yml -p mozdef logs -f

# Check specific service logs
docker-compose -f docker/compose/docker-compose.yml -p mozdef logs -f elasticsearch
docker-compose -f docker/compose/docker-compose.yml -p mozdef logs -f bootstrap
```

**Expected Startup Time:**
- Infrastructure services: 1-2 minutes
- Bootstrap: 30-60 seconds
- Backend services: 1-2 minutes
- Frontend services: 2-3 minutes
- **Total: 5-8 minutes**

---

## Component-by-Component Setup

### 1. Elasticsearch Service

**Purpose:** Event storage, indexing, and search engine

**Configuration:**
- **Image:** `mozdef/mozdef_elasticsearch`
- **Port:** 9200 (internal), not exposed externally
- **Volume:** `elasticsearch:/var/lib/elasticsearch`
- **Health Check:** `curl -f http://127.0.0.1:9200`

**Start Individually:**
```bash
# Start only Elasticsearch
docker-compose -f docker/compose/docker-compose.yml -p mozdef up -d elasticsearch

# Check if running
docker-compose -f docker/compose/docker-compose.yml -p mozdef ps elasticsearch

# Check health
docker exec -it mozdef_elasticsearch_1 curl -f http://127.0.0.1:9200

# View logs
docker-compose -f docker/compose/docker-compose.yml -p mozdef logs elasticsearch
```

**Verify Elasticsearch:**
```bash
# Check cluster health
docker exec -it mozdef_elasticsearch_1 curl http://127.0.0.1:9200/_cluster/health?pretty

# Should return:
# {
#   "cluster_name" : "elasticsearch",
#   "status" : "green" or "yellow",
#   ...
# }

# List indices (should be empty initially)
docker exec -it mozdef_elasticsearch_1 curl http://127.0.0.1:9200/_cat/indices?v
```

**Troubleshooting:**
- **Not starting:** Check Docker memory allocation (needs 2GB+)
- **Yellow status:** Normal for single-node cluster
- **Out of memory:** Increase Docker memory limit

---

### 2. RabbitMQ Service

**Purpose:** Message queue for event processing

**Configuration:**
- **Image:** `mozdef/mozdef_rabbitmq`
- **Ports:** 5672 (AMQP, internal), 15672 (Management, internal)
- **Volume:** `rabbitmq:/var/lib/rabbitmq`
- **Default Credentials:** guest/guest
- **Health Check:** `curl -f http://127.0.0.1:15672`

**Start Individually:**
```bash
# Start RabbitMQ
docker-compose -f docker/compose/docker-compose.yml -p mozdef up -d rabbitmq

# Check status
docker-compose -f docker/compose/docker-compose.yml -p mozdef ps rabbitmq

# Access management UI (if port exposed)
# http://localhost:15672 (guest/guest)
```

**Verify RabbitMQ:**
```bash
# Check if management interface responds
docker exec -it mozdef_rabbitmq_1 curl -f http://127.0.0.1:15672

# List exchanges (should see eventtask, mozdef.event, alerts)
docker exec -it mozdef_rabbitmq_1 rabbitmqctl list_exchanges

# List queues
docker exec -it mozdef_rabbitmq_1 rabbitmqctl list_queues
```

**Troubleshooting:**
- **Connection refused:** Wait 30-60 seconds for full startup
- **Management UI not accessible:** Port not exposed (normal, internal only)

---

### 3. MongoDB Service

**Purpose:** Database for Meteor (incidents, users, settings)

**Configuration:**
- **Image:** `mozdef/mozdef_mongodb`
- **Port:** 3002 (internal)
- **Volume:** `mongodb:/var/lib/mongo`
- **Config:** `/etc/mongod.conf`
- **Health Check:** `curl -f http://127.0.0.1:3002`

**Start Individually:**
```bash
# Start MongoDB
docker-compose -f docker/compose/docker-compose.yml -p mozdef up -d mongodb

# Check status
docker-compose -f docker/compose/docker-compose.yml -p mozdef ps mongodb

# Connect to MongoDB shell
docker exec -it mozdef_mongodb_1 mongo --port 3002
```

**Verify MongoDB:**
```bash
# List databases
docker exec -it mozdef_mongodb_1 mongo --port 3002 --eval "db.adminCommand('listDatabases')"

# Should see: meteor, admin, local, config
```

**Troubleshooting:**
- **Not starting:** Check disk space (MongoDB needs space for journal)
- **Connection refused:** Wait for initialization (30 seconds)

---

### 4. Bootstrap Service

**Purpose:** Initial setup of Elasticsearch indices and Kibana dashboards

**Configuration:**
- **Image:** `mozdef/mozdef_bootstrap`
- **Runs Once:** Exits after setup completes
- **Dependencies:** base, elasticsearch
- **Script:** `initial_setup.py`

**What It Does:**
1. Waits for Elasticsearch to be ready
2. Creates Elasticsearch index templates
3. Creates initial indices (events-*, alerts-*)
4. Sets up Kibana index patterns
5. Imports Kibana dashboards (if available)

**Start Individually:**
```bash
# Start bootstrap (runs once and exits)
docker-compose -f docker/compose/docker-compose.yml -p mozdef up bootstrap

# Check logs
docker-compose -f docker/compose/docker-compose.yml -p mozdef logs bootstrap

# Should see:
# "Waiting for Elasticsearch..."
# "Elasticsearch is ready"
# "Creating index templates..."
# "Setting up Kibana..."
# "Bootstrap complete"
```

**Verify Bootstrap:**
```bash
# Check if indices were created
docker exec -it mozdef_elasticsearch_1 curl http://127.0.0.1:9200/_cat/indices?v

# Should see:
# events-YYYY.MM.DD
# alerts-YYYY.MM.DD
# (with today's date)
```

**Troubleshooting:**
- **Fails immediately:** Elasticsearch not ready (wait 1-2 minutes)
- **Index creation fails:** Check Elasticsearch logs

---

### 5. Loginput Service

**Purpose:** HTTP endpoint for receiving JSON events

**Configuration:**
- **Image:** `mozdef/mozdef_loginput`
- **Port:** 8080 (via Nginx)
- **Config File:** `loginput/index.conf`
- **Endpoints:**
  - `/events` - Single event POST
  - `/_bulk` - Bulk events POST
  - `/nxlog` - NXLog endpoint
  - `/status` - Health check

**Configuration File:** `loginput/index.conf`
```ini
[options]
mquser=guest
mqpassword=guest
mqserver=rabbitmq
mqport=5672
mqvhost=/
mqprotocol=amqp
taskexchange=eventtask
listen_host=0.0.0.0
listen_port=8080
```

**Start Individually:**
```bash
# Start loginput
docker-compose -f docker/compose/docker-compose.yml -p mozdef up -d loginput

# Check status
docker-compose -f docker/compose/docker-compose.yml -p mozdef ps loginput

# Check health
curl http://localhost:8080/status
# Should return: {"status":"ok","service":"loginput"}
```

**Verify Loginput:**
```bash
# Test endpoint
curl -X POST http://localhost:8080/events \
  -H "Content-Type: application/json" \
  -d '{
    "timestamp": "2024-01-15T10:00:00+00:00",
    "utctimestamp": "2024-01-15T10:00:00+00:00",
    "hostname": "test.example.com",
    "processname": "test.py",
    "processid": 1234,
    "severity": "INFO",
    "summary": "Test event",
    "category": "test",
    "source": "test",
    "tags": ["test"],
    "details": {}
  }'

# Should return: HTTP 200 OK
```

**View Logs:**
```bash
# Loginput logs
docker-compose -f docker/compose/docker-compose.yml -p mozdef logs loginput

# Should see:
# "Starting loginput service..."
# "Connected to RabbitMQ"
# "Listening on 0.0.0.0:8080"
```

**Troubleshooting:**
- **Connection refused:** Nginx not running or port conflict
- **RabbitMQ connection fails:** RabbitMQ not ready (wait 30 seconds)
- **500 errors:** Invalid JSON format (check event structure)

---

### 6. MQ Worker Service

**Purpose:** Process events from RabbitMQ queue and index to Elasticsearch

**Configuration:**
- **Image:** `mozdef/mozdef_mq_worker`
- **Config File:** `mq/esworker_eventtask.conf`
- **Dependencies:** elasticsearch, rabbitmq, loginput

**Configuration File:** `mq/esworker_eventtask.conf`
```ini
[options]
prefetch=150
esbulksize=150
esservers=http://elasticsearch:9200
mqack=False
mquser=guest
mqpassword=guest
mqserver=rabbitmq
mqport=5672
mqvhost=/
mqprotocol=amqp
taskexchange=eventtask
eventexchange=mozdef.event
```

**Start Individually:**
```bash
# Start MQ worker
docker-compose -f docker/compose/docker-compose.yml -p mozdef up -d mq_worker

# Check status
docker-compose -f docker/compose/docker-compose.yml -p mozdef ps mq_worker

# Check if worker is processing
docker-compose -f docker/compose/docker-compose.yml -p mozdef logs mq_worker
```

**Verify MQ Worker:**
```bash
# Send test event (see Loginput section)
# Then check if event appears in Elasticsearch
docker exec -it mozdef_elasticsearch_1 curl \
  "http://127.0.0.1:9200/events-*/_search?q=category:test&pretty"

# Should return your test event
```

**View Logs:**
```bash
# MQ Worker logs
docker-compose -f docker/compose/docker-compose.yml -p mozdef logs mq_worker

# Should see:
# "Connected to RabbitMQ"
# "Connected to Elasticsearch"
# "Consuming from queue: eventtask"
# "Processing event..."
# "Indexed event to Elasticsearch"
```

**Troubleshooting:**
- **Not consuming messages:** Check RabbitMQ connection
- **Elasticsearch errors:** Check Elasticsearch health
- **Events not appearing:** Check worker logs for errors

---

### 7. REST API Service

**Purpose:** RESTful API for web interface

**Configuration:**
- **Image:** `mozdef/mozdef_rest`
- **Port:** 8081 (internal, via Nginx)
- **Config File:** `rest/index.conf`
- **Dependencies:** elasticsearch, mongodb

**Configuration File:** `rest/index.conf`
```ini
[options]
esservers=http://elasticsearch:9200
mongohost=mongodb
mongoport=3002
listen_host=0.0.0.0
listen_port=8081
```

**Start Individually:**
```bash
# Start REST API
docker-compose -f docker/compose/docker-compose.yml -p mozdef up -d rest

# Check status
docker-compose -f docker/compose/docker-compose.yml -p mozdef ps rest

# Check health
curl http://localhost:8081/status
# Should return: {"status":"ok","service":"restapi"}
```

**Verify REST API:**
```bash
# Test endpoints
curl http://localhost:8081/veris
curl http://localhost:8081/kibanadashboards
curl http://localhost:8081/getwatchlist
```

**View Logs:**
```bash
docker-compose -f docker/compose/docker-compose.yml -p mozdef logs rest
```

**Troubleshooting:**
- **Connection refused:** Nginx not routing correctly
- **Elasticsearch errors:** Check Elasticsearch connection
- **MongoDB errors:** Check MongoDB connection

---

### 8. Meteor Service (Web Interface)

**Purpose:** Real-time web interface for incident handling

**Configuration:**
- **Image:** `mozdef/mozdef_meteor`
- **Port:** 3000 (internal), 80 (via Nginx)
- **Dependencies:** mongodb, rest
- **Settings:** `meteor/imports/settings.js`

**Environment Variables:**
```bash
MONGO_URL=mongodb://mongodb:3002/meteor
ROOT_URL=http://localhost
PORT=3000
```

**Start Individually:**
```bash
# Start Meteor
docker-compose -f docker/compose/docker-compose.yml -p mozdef up -d meteor

# Check status
docker-compose -f docker/compose/docker-compose.yml -p mozdef ps meteor

# Check health
curl http://localhost:3000
# Should return HTML (Meteor app)
```

**Verify Meteor:**
```bash
# Access web interface
# http://localhost (via Nginx) or http://localhost:3000 (direct)

# Check if Meteor is running
docker exec -it mozdef_meteor_1 curl http://127.0.0.1:3000
```

**View Logs:**
```bash
docker-compose -f docker/compose/docker-compose.yml -p mozdef logs meteor

# Should see:
# "MozDef starting"
# "Meteor server started"
# "Connected to MongoDB"
```

**Troubleshooting:**
- **Not accessible:** Check Nginx routing
- **MongoDB connection fails:** Wait for MongoDB to be ready
- **REST API errors:** Check REST API is running

---

### 9. Kibana Service

**Purpose:** Log visualization and analysis

**Configuration:**
- **Image:** `mozdef/mozdef_kibana`
- **Port:** 5601 (internal), 9090 (via Nginx)
- **Dependencies:** elasticsearch
- **Config:** Connects to `http://elasticsearch:9200`

**Start Individually:**
```bash
# Start Kibana
docker-compose -f docker/compose/docker-compose.yml -p mozdef up -d kibana

# Check status
docker-compose -f docker/compose/docker-compose.yml -p mozdef ps kibana

# Check health
curl http://localhost:9090/app/kibana
# Should return HTML
```

**Verify Kibana:**
```bash
# Access Kibana
# http://localhost:9090/app/kibana

# Check if Kibana API responds
curl http://localhost:9090/api/status
```

**View Logs:**
```bash
docker-compose -f docker/compose/docker-compose.yml -p mozdef logs kibana
```

**Troubleshooting:**
- **Not accessible:** Check Nginx routing
- **Elasticsearch connection fails:** Wait for Elasticsearch
- **Index patterns missing:** Run bootstrap again

---

### 10. Alerts Service

**Purpose:** Alert correlation engine (Celery-based)

**Configuration:**
- **Image:** `mozdef/mozdef_alerts`
- **Technology:** Celery (Python task scheduler)
- **Dependencies:** elasticsearch, rabbitmq, rest, bootstrap
- **Config:** `alerts/lib/config.py`

**Start Individually:**
```bash
# Start Alerts
docker-compose -f docker/compose/docker-compose.yml -p mozdef up -d alerts

# Check status
docker-compose -f docker/compose/docker-compose.yml -p mozdef ps alerts

# Check Celery health
docker exec -it mozdef_alerts_1 celery -A lib.tasks inspect ping
# Should return: pong
```

**Verify Alerts:**
```bash
# Check registered alerts
docker exec -it mozdef_alerts_1 celery -A lib.tasks inspect registered

# Check active tasks
docker exec -it mozdef_alerts_1 celery -A lib.tasks inspect active

# Check scheduled tasks
docker exec -it mozdef_alerts_1 celery -A lib.tasks inspect scheduled
```

**View Logs:**
```bash
docker-compose -f docker/compose/docker-compose.yml -p mozdef logs alerts

# Should see:
# "Loading alerts..."
# "Registered alert: bruteforce_ssh"
# "Registered alert: proxy_drop_executable"
# "Celery worker started"
# "Beat scheduler started"
```

**Troubleshooting:**
- **No alerts registered:** Check alert configuration files
- **Celery not starting:** Check RabbitMQ connection
- **Alerts not firing:** Check Elasticsearch queries

---

### 11. Alert Actions Service

**Purpose:** Post-alert processing and actions

**Configuration:**
- **Image:** `mozdef/mozdef_alertactions`
- **Config File:** `alerts/alert_actions_worker.conf`
- **Dependencies:** elasticsearch, rabbitmq, alerts

**Configuration File:** `alerts/alert_actions_worker.conf`
```ini
[options]
esservers=http://elasticsearch:9200
mquser=guest
mqpassword=guest
mqserver=rabbitmq
mqport=5672
mqvhost=/
mqprotocol=amqp
alertexchange=alerts
alertqueue=alerts
```

**Start Individually:**
```bash
# Start Alert Actions
docker-compose -f docker/compose/docker-compose.yml -p mozdef up -d alertactions

# Check status
docker-compose -f docker/compose/docker-compose.yml -p mozdef ps alertactions

# Check if worker is running
docker exec -it mozdef_alertactions_1 ps aux | grep python
```

**Verify Alert Actions:**
```bash
# Check if consuming from alerts queue
docker exec -it mozdef_rabbitmq_1 rabbitmqctl list_queues name consumers

# Should see "alerts" queue with 1 consumer
```

**View Logs:**
```bash
docker-compose -f docker/compose/docker-compose.yml -p mozdef logs alertactions
```

**Troubleshooting:**
- **Not consuming alerts:** Check RabbitMQ connection
- **No alerts processed:** Ensure alerts service is generating alerts

---

### 12. Nginx Service

**Purpose:** Reverse proxy routing traffic to services

**Configuration:**
- **Image:** `mozdef/mozdef_nginx`
- **Config File:** `config/nginx.conf`
- **Ports:** 80, 8080, 8081, 9090
- **Dependencies:** meteor, kibana, loginput, rest

**Start Individually:**
```bash
# Start Nginx (starts last, depends on all services)
docker-compose -f docker/compose/docker-compose.yml -p mozdef up -d nginx

# Check status
docker-compose -f docker/compose/docker-compose.yml -p mozdef ps nginx

# Test routing
curl http://localhost/status
curl http://localhost:8080/status
curl http://localhost:9090/app/kibana
```

**Verify Nginx:**
```bash
# Check Nginx config
docker exec -it mozdef_nginx_1 nginx -t

# Check access logs
docker-compose -f docker/compose/docker-compose.yml -p mozdef logs nginx
```

**Troubleshooting:**
- **502 Bad Gateway:** Upstream service not ready
- **Connection refused:** Service not started
- **404 Not Found:** Check routing configuration

---

### 13. Cron Service

**Purpose:** Scheduled tasks (index rotation, stats, etc.)

**Configuration:**
- **Image:** `mozdef/mozdef_cron`
- **Dependencies:** elasticsearch, rabbitmq, mongodb
- **Tasks:** Defined in `/opt/mozdef/envs/mozdef/cron/`

**Start Individually:**
```bash
# Start Cron
docker-compose -f docker/compose/docker-compose.yml -p mozdef up -d cron

# Check status
docker-compose -f docker/compose/docker-compose.yml -p mozdef ps cron

# Check cron jobs
docker exec -it mozdef_cron_1 crontab -l
```

**View Logs:**
```bash
docker-compose -f docker/compose/docker-compose.yml -p mozdef logs cron
```

---

## Starting Services in Order

### Correct Startup Sequence

Services must start in dependency order:

```bash
# 1. Infrastructure Services (no dependencies)
docker-compose -f docker/compose/docker-compose.yml -p mozdef up -d elasticsearch
docker-compose -f docker/compose/docker-compose.yml -p mozdef up -d rabbitmq
docker-compose -f docker/compose/docker-compose.yml -p mozdef up -d mongodb

# 2. Wait for infrastructure (30-60 seconds)
sleep 60

# 3. Bootstrap (sets up Elasticsearch)
docker-compose -f docker/compose/docker-compose.yml -p mozdef up bootstrap

# 4. Backend Services
docker-compose -f docker/compose/docker-compose.yml -p mozdef up -d loginput
docker-compose -f docker/compose/docker-compose.yml -p mozdef up -d mq_worker
docker-compose -f docker/compose/docker-compose.yml -p mozdef up -d rest

# 5. Frontend Services
docker-compose -f docker/compose/docker-compose.yml -p mozdef up -d kibana
docker-compose -f docker/compose/docker-compose.yml -p mozdef up -d meteor

# 6. Alert Services
docker-compose -f docker/compose/docker-compose.yml -p mozdef up -d alerts
docker-compose -f docker/compose/docker-compose.yml -p mozdef up -d alertactions

# 7. Proxy (starts last)
docker-compose -f docker/compose/docker-compose.yml -p mozdef up -d nginx

# 8. Optional Services
docker-compose -f docker/compose/docker-compose.yml -p mozdef up -d cron
docker-compose -f docker/compose/docker-compose.yml -p mozdef up -d syslog
```

### Using Make Command (Recommended)

```bash
# This handles all dependencies automatically
make run
```

---

## Verification & Testing

### 1. Check All Services Running

```bash
# List all services
docker-compose -f docker/compose/docker-compose.yml -p mozdef ps

# All services should show "Up" status
# Health checks should pass (no "unhealthy" status)
```

### 2. Test Event Ingestion

```bash
# Send test event
curl -X POST http://localhost:8080/events \
  -H "Content-Type: application/json" \
  -d '{
    "timestamp": "'$(date -u +"%Y-%m-%dT%H:%M:%S+00:00")'",
    "utctimestamp": "'$(date -u +"%Y-%m-%dT%H:%M:%S+00:00")'",
    "hostname": "test.example.com",
    "processname": "test.py",
    "processid": 1234,
    "severity": "INFO",
    "summary": "Test event from detailed guide",
    "category": "test",
    "source": "test",
    "tags": ["test", "verification"],
    "details": {
      "message": "This is a test event"
    }
  }'

# Should return: HTTP 200 OK
```

### 3. Verify Event in Elasticsearch

```bash
# Wait 5-10 seconds for processing
sleep 10

# Search for test event
docker exec -it mozdef_elasticsearch_1 curl \
  "http://127.0.0.1:9200/events-*/_search?q=category:test&pretty&size=1"

# Should return your test event
```

### 4. Verify Event in Kibana

```bash
# 1. Open Kibana: http://localhost:9090/app/kibana
# 2. Go to "Discover"
# 3. Select index pattern: "events-*"
# 4. Search for: category:test
# 5. Should see your test event
```

### 5. Test REST API

```bash
# Health check
curl http://localhost:8081/status

# Get watchlist
curl http://localhost:8081/getwatchlist

# Get VERIS stats
curl http://localhost:8081/veris
```

### 6. Test Meteor Web Interface

```bash
# 1. Open browser: http://localhost
# 2. Should see MozDef login page
# 3. Create account or login
# 4. Should see dashboard
```

### 7. Test Alert Generation

```bash
# Send multiple failed login events to trigger SSH brute force alert
for i in {1..15}; do
  curl -X POST http://localhost:8080/events \
    -H "Content-Type: application/json" \
    -d "{
      \"timestamp\": \"$(date -u +"%Y-%m-%dT%H:%M:%S+00:00")\",
      \"utctimestamp\": \"$(date -u +"%Y-%m-%dT%H:%M:%S+00:00")\",
      \"hostname\": \"test.example.com\",
      \"processname\": \"sshd\",
      \"processid\": 1234,
      \"severity\": \"WARNING\",
      \"summary\": \"Failed SSH login attempt\",
      \"category\": \"authentication\",
      \"source\": \"syslog\",
      \"tags\": [\"ssh\", \"authentication\", \"failure\"],
      \"details\": {
        \"username\": \"testuser\",
        \"sourceipaddress\": \"192.168.1.100\",
        \"success\": false
      }
    }"
  sleep 1
done

# Wait 5 minutes for alert to fire
# Check alerts in Elasticsearch
docker exec -it mozdef_elasticsearch_1 curl \
  "http://127.0.0.1:9200/alerts-*/_search?pretty&size=5"
```

---

## Configuration Deep Dive

### Environment Variables

Set via Docker Compose or environment file:

```bash
# Create custom environment file
cat > my-mozdef.env <<EOF
OPTIONS_METEOR_ROOTURL=http://localhost
OPTIONS_METEOR_PORT=80
OPTIONS_METEOR_ROOTAPI=http://rest:8081
OPTIONS_METEOR_KIBANAURL=http://localhost:9090/app/kibana
OPTIONS_METEOR_ENABLECLIENTACCOUNTCREATION=true
OPTIONS_METEOR_AUTHENTICATIONTYPE=meteor-password
OPTIONS_MQPROTOCOL=amqp
EOF

# Run with custom env file
make run-env-mozdef -e ENV=my-mozdef.env
```

### Key Configuration Files

| Service | Config File | Location |
|---------|------------|----------|
| Loginput | `index.conf` | `loginput/index.conf` |
| MQ Worker | `esworker_eventtask.conf` | `mq/esworker_eventtask.conf` |
| REST API | `index.conf` | `rest/index.conf` |
| Alerts | `lib/config.py` | `alerts/lib/config.py` |
| Alert Actions | `alert_actions_worker.conf` | `alerts/alert_actions_worker.conf` |
| Nginx | `nginx.conf` | `config/nginx.conf` |
| Meteor | `settings.js` | `meteor/imports/settings.js` |

---

## Troubleshooting Each Component

### Elasticsearch Issues

**Problem: Not starting**
```bash
# Check logs
docker-compose -f docker/compose/docker-compose.yml -p mozdef logs elasticsearch

# Common issues:
# - Out of memory: Increase Docker memory
# - Disk space: Check available disk
# - Permissions: Check volume permissions
```

**Problem: Yellow/Red cluster status**
```bash
# Single node cluster shows yellow (normal)
# Red means unassigned shards - check logs
docker-compose -f docker/compose/docker-compose.yml -p mozdef logs elasticsearch
```

### RabbitMQ Issues

**Problem: Connection refused**
```bash
# Wait for RabbitMQ to fully start (30-60 seconds)
docker-compose -f docker/compose/docker-compose.yml -p mozdef logs rabbitmq

# Check if management interface is up
docker exec -it mozdef_rabbitmq_1 curl http://127.0.0.1:15672
```

### Loginput Issues

**Problem: 500 errors on event POST**
```bash
# Check event JSON format
# Must include all mandatory fields

# Check logs
docker-compose -f docker/compose/docker-compose.yml -p mozdef logs loginput

# Common issues:
# - Invalid JSON
# - Missing mandatory fields
# - RabbitMQ connection failed
```

### MQ Worker Issues

**Problem: Events not appearing in Elasticsearch**
```bash
# Check worker logs
docker-compose -f docker/compose/docker-compose.yml -p mozdef logs mq_worker

# Check RabbitMQ queue
docker exec -it mozdef_rabbitmq_1 rabbitmqctl list_queues

# Check Elasticsearch connection
docker exec -it mq_worker curl http://elasticsearch:9200
```

### Meteor Issues

**Problem: Web interface not loading**
```bash
# Check Meteor logs
docker-compose -f docker/compose/docker-compose.yml -p mozdef logs meteor

# Check MongoDB connection
docker exec -it mozdef_meteor_1 mongo --host mongodb --port 3002

# Check REST API connection
docker exec -it mozdef_meteor_1 curl http://rest:8081/status
```

### Alerts Issues

**Problem: Alerts not firing**
```bash
# Check Celery status
docker exec -it mozdef_alerts_1 celery -A lib.tasks inspect active

# Check registered alerts
docker exec -it mozdef_alerts_1 celery -A lib.tasks inspect registered

# Check alert logs
docker-compose -f docker/compose/docker-compose.yml -p mozdef logs alerts

# Verify Elasticsearch queries work
docker exec -it mozdef_alerts_1 python -c "
from mozdef_util.elasticsearch_client import ElasticsearchClient
from mozdef_util.query_models import SearchQuery, TermMatch
es = ElasticsearchClient(['http://elasticsearch:9200'])
query = SearchQuery(minutes=5)
query.add_must([TermMatch('category', 'authentication')])
results = es.query(query)
print(f'Found {len(results)} events')
"
```

---

## Development Mode Setup

### Run Individual Services for Development

```bash
# Start only infrastructure
docker-compose -f docker/compose/docker-compose.yml -p mozdef up -d elasticsearch rabbitmq mongodb

# Run service in foreground (see logs)
docker-compose -f docker/compose/docker-compose.yml -p mozdef up loginput

# Rebuild specific service
docker-compose -f docker/compose/docker-compose.yml -p mozdef build loginput
docker-compose -f docker/compose/docker-compose.yml -p mozdef up -d loginput
```

### Development Meteor (Live Reload)

```bash
# Run Meteor in development mode
make run-dev-meteor

# This mounts meteor directory for live code changes
```

### Run Tests

```bash
# Build test environment
make build-tests

# Run all tests
make tests

# Run specific test
make run-tests TEST_CASE=tests/alerts/test_bruteforce_ssh.py
```

---

## Complete Startup Checklist

- [ ] Docker and Docker Compose installed
- [ ] At least 4GB RAM allocated to Docker
- [ ] MozDef directory cloned/downloaded
- [ ] Docker images built (`make build`)
- [ ] All services started (`make run`)
- [ ] Elasticsearch healthy (green/yellow status)
- [ ] RabbitMQ management accessible
- [ ] MongoDB running
- [ ] Bootstrap completed successfully
- [ ] Loginput responding (`curl http://localhost:8080/status`)
- [ ] REST API responding (`curl http://localhost:8081/status`)
- [ ] Meteor web interface accessible (`http://localhost`)
- [ ] Kibana accessible (`http://localhost:9090/app/kibana`)
- [ ] Test event sent and appears in Elasticsearch
- [ ] Test event visible in Kibana
- [ ] Alerts service running (Celery ping successful)

---

## Quick Reference Commands

```bash
# Start everything
make run

# Stop everything
make stop

# View all logs
docker-compose -f docker/compose/docker-compose.yml -p mozdef logs -f

# View specific service logs
docker-compose -f docker/compose/docker-compose.yml -p mozdef logs -f [service-name]

# Restart specific service
docker-compose -f docker/compose/docker-compose.yml -p mozdef restart [service-name]

# Rebuild specific service
docker-compose -f docker/compose/docker-compose.yml -p mozdef build [service-name]

# Clean everything
make clean

# Check service status
docker-compose -f docker/compose/docker-compose.yml -p mozdef ps

# Execute command in container
docker exec -it mozdef_[service]_1 [command]

# Access service shell
docker exec -it mozdef_[service]_1 /bin/bash
```

---

This guide provides complete step-by-step instructions for running every component of MozDef. Follow it sequentially for a successful deployment!