Hugging Face's logo

ineso22
/
Mozdef

Model card Files
xet
 Community
Mozdef / ONLINE_OFFLINE_MILESTONES.md
ineso22's picture
ineso22
Upload folder using huggingface_hub
7c89ed7 verified
preview code
|
raw
history blame contribute delete
9.62 kB

MozDef Online → Offline Setup Milestones (Frontend / Backend / Infra)

This file gives you a checklist-style plan for:

  • Bringing MozDef up online (with internet)
  • Packing everything
  • Running it offline

Each milestone is split into infrastructure, backend, and frontend where it makes sense, so you can see exactly what layer you’re validating.

Milestone 1 – Hosts & Tools Ready

1.1 Online host (build + test)

  • Do
    • Install Docker + Docker Compose.
    • Ensure ~50 GB free disk.
  • Check
    docker --version
docker-compose --version
docker run --rm hello-world
df -h              # check free space
uname -m           # should be x86_64

1.2 Offline host (final runtime)

  • Do
    • Install Docker + Docker Compose from local media / mirror.
  • Check
    docker --version
docker-compose --version
uname -m           # x86_64

Milestone 2 – Online: Get Source & Build Images

This milestone builds all layers inside Docker (isolated from host OS).

2.1 Get MozDef source

  • Do
    cd /root
git clone https://github.com/mozilla/MozDef.git   # or use your existing /root/MozDef
cd MozDef
  • Check
    ls
# expect: alerts bot config docker docs loginput meteor mq rest ... Makefile

2.2 Build all images (infra + backend + frontend)

  • Do
    cd /root/MozDef
make build
  • Check
    docker images | grep mozdef
# expect at least:
#  mozdef/mozdef_base
#  mozdef/mozdef_elasticsearch
#  mozdef/mozdef_rabbitmq
#  mozdef/mozdef_mongodb
#  mozdef/mozdef_kibana
#  mozdef/mozdef_nginx
#  mozdef/mozdef_bootstrap
#  mozdef/mozdef_loginput
#  mozdef/mozdef_mq_worker
#  mozdef/mozdef_rest
#  mozdef/mozdef_meteor
#  mozdef/mozdef_alerts
#  mozdef/mozdef_alertactions
#  mozdef/mozdef_cron
#  mozdef/mozdef_syslog

2.3 Confirm runtime isolation (critical OS/Node concern)

  • Do
    # host versions (for info only)
node --version || echo "no host node"
python3 --version || echo "no host python"

# container runtimes (REAL runtime MozDef will use)
docker run --rm mozdef/mozdef_meteor node --version      # expect v8.11.4
docker run --rm mozdef/mozdef_base python3 --version     # expect Python 3.6.x
  • Goal
    • Prove that Node 8 and Python 3.6 live inside containers, independent of CentOS/RHEL 8 host packages.

Milestone 3 – Online: Full Stack Test (Infra → Backend → Frontend)

3.1 Start everything

  • Do
    cd /root/MozDef
make run        # starts all infra + backend + frontend
sleep 300       # wait 5–8 minutes
  • Check
    docker-compose -f docker/compose/docker-compose.yml -p mozdef ps
# all services should show "Up" (no "unhealthy")

3.2 Infra checks (shared for backend + frontend)

  • Elasticsearch
    docker exec -it mozdef_elasticsearch_1 curl \
  http://127.0.0.1:9200/_cluster/health?pretty
# status: "green" or "yellow"
  • RabbitMQ
    docker exec -it mozdef_rabbitmq_1 curl \
  http://127.0.0.1:15672 || echo "mgmt UI ok if 200/redirect"
  • MongoDB
    docker exec -it mozdef_mongodb_1 \
  mongo --port 3002 --eval "db.adminCommand('ping')"

3.3 Backend checks

  • loginput (ingest API)
    curl http://localhost:8080/status
# {"status":"ok","service":"loginput"}
  • rest (REST API for Meteor)
    curl http://localhost:8081/status
# {"status":"ok","service":"restapi"}

3.4 Frontend checks

  • Meteor UI
    curl -I http://localhost
# HTTP/1.1 200 OK
  • Kibana
    curl -I http://localhost:9090/app/kibana
# HTTP/1.1 200 OK

3.5 End‑to‑end event test (frontend → backend → Elasticsearch)

  • Do
    curl -X POST http://localhost:8080/events \
  -H "Content-Type: application/json" \
  -d '{
    "timestamp": "2024-01-15T10:00:00+00:00",
    "utctimestamp": "2024-01-15T10:00:00+00:00",
    "hostname": "test.example.com",
    "processname": "test.py",
    "processid": 1234,
    "severity": "INFO",
    "summary": "Online pre-pack test event",
    "category": "test",
    "source": "test",
    "tags": ["test"],
    "details": {}
  }'
sleep 10
  • Check
    docker exec -it mozdef_elasticsearch_1 curl \
  "http://127.0.0.1:9200/events-*/_search?q=category:test&size=1&pretty"
# expect your test event in hits

3.6 Stop stack (ready to freeze for offline)

  • Do
    cd /root/MozDef
make stop
docker-compose -f docker/compose/docker-compose.yml -p mozdef ps
# no containers "Up"

Milestone 4 – Online: Package for Offline

4.1 Package Docker images (all layers)

  • Do (single tar for all) 
    mkdir -p ~/mozdef-offline-package/docker-images
cd /root/MozDef

docker save $(docker images mozdef/* --format "{{.Repository}}:{{.Tag}}") \
  -o ~/mozdef-offline-package/docker-images/mozdef-all-images.tar
  • Check
    ls -lh ~/mozdef-offline-package/docker-images/mozdef-all-images.tar
# ~10–20 GB

4.2 (Optional safety) cache Python / npm

  • Python (backend libs)
    mkdir -p ~/mozdef-offline-package/python-packages
cd /root/MozDef

python3 -m venv /tmp/mozdef-download-env
source /tmp/mozdef-download-env/bin/activate

pip download -r requirements.txt \
  -d ~/mozdef-offline-package/python-packages \
  --platform linux_x86_64 --python-version 36 --only-binary=:all: || true
  • npm (frontend libs) – only if you expect to rebuild Meteor offline:
    mkdir -p ~/mozdef-offline-package/npm-packages
cd /root/MozDef/meteor

if [ ! -f package-lock.json ]; then
  npm install --package-lock-only
fi

npm ci --cache ~/mozdef-offline-package/npm-packages/.npm \
  --prefer-offline=false || npm install --cache ~/mozdef-offline-package/npm-packages/.npm

4.3 Package source code

  • Do
    mkdir -p ~/mozdef-offline-package/source-code
tar -czf ~/mozdef-offline-package/source-code/MozDef-source.tar.gz /root/MozDef

4.4 Create final archive to move

  • Do
    cd ~/mozdef-offline-package
tar -czf ../mozdef-offline-package-$(date +%Y%m%d).tar.gz .
du -sh ../mozdef-offline-package-*.tar.gz

Milestone 5 – Transfer to Offline

  • Option: USB / disk

    cp ../mozdef-offline-package-*.tar.gz /mnt/usb/

  • Option: split if too big

    cd ~
split -b 4G mozdef-offline-package-*.tar.gz mozdef-part-
# rejoin on offline machine:
# cat mozdef-part-* > mozdef-offline-package.tar.gz

Milestone 6 – Offline: Load Images & Run

6.1 Extract package

  • Do
    mkdir -p ~/mozdef-offline-install
cd ~/mozdef-offline-install

tar -xzf /path/to/mozdef-offline-package-*.tar.gz
cd mozdef-offline-package

6.2 Load Docker images (infra + backend + frontend)

  • Do
    cd docker-images

docker load -i mozdef-all-images.tar

docker images | grep mozdef
# should list all mozdef/* images

6.3 Install source code

  • Do
    cd ~/mozdef-offline-install/mozdef-offline-package
mkdir -p /opt/mozdef

tar -xzf source-code/MozDef-source.tar.gz -C /opt/mozdef
cd /opt/mozdef/MozDef

6.4 Quick isolation sanity check (same as online)

  • Do
    node --version || echo "host node (irrelevant)"
docker run --rm mozdef/mozdef_meteor node --version     # expect v8.11.4

python3 --version || echo "host python (irrelevant)"
docker run --rm mozdef/mozdef_base python3 --version    # expect Python 3.6.x

Milestone 7 – Offline: Start Stack & Validate

7.1 Start all services offline

  • Do
    cd /opt/mozdef/MozDef
make run
sleep 300
  • Check
    docker-compose -f docker/compose/docker-compose.yml -p mozdef ps
# all services "Up"

7.2 Infra checks (same as online)

  • Elasticsearch / RabbitMQ / MongoDB – same commands as Milestone 3.2.

7.3 Backend checks

  • loginput
    curl http://localhost:8080/status
  • rest
    curl http://localhost:8081/status

7.4 Frontend checks

  • Meteor UI
    curl -I http://localhost
  • Kibana
    curl -I http://localhost:9090/app/kibana

7.5 Offline end‑to‑end event test

  • Do
    curl -X POST http://localhost:8080/events \
  -H "Content-Type: application/json" \
  -d '{
    "timestamp": "2024-01-15T10:00:00+00:00",
    "utctimestamp": "2024-01-15T10:00:00+00:00",
    "hostname": "offline.example.com",
    "processname": "test.py",
    "processid": 1234,
    "severity": "INFO",
    "summary": "Offline test event",
    "category": "offline-test",
    "source": "offline",
    "tags": ["offline","test"],
    "details": {}
  }'
sleep 10
  • Check
    docker exec -it mozdef_elasticsearch_1 curl \
  "http://127.0.0.1:9200/events-*/_search?q=category:offline-test&size=1&pretty"
# expect your offline event

How to Use This Milestone File

  • Go milestone by milestone.
  • After each Check section passes, you know that layer (infra/backend/frontend) is good.
  • The detailed command reference for packaging, manifests, and scripts lives in OFFLINE_DEPLOYMENT_GUIDE.md.
    This file is your progress checklist; the guide is the full cookbook.