Hugging Face's logo

ineso22
/
Mozdef

Model card Files
xet
 Community
Mozdef / PROJECT_PACKAGING_GUIDE.md
ineso22's picture
ineso22
Upload folder using huggingface_hub
7c89ed7 verified
preview code
|
raw
history blame contribute delete
9.32 kB

MozDef - Project Packaging & Transfer Guide

📦 Project Size & Location

Project Structure 

/root/MozDef/
├── Source Code (application files)
├── docker/ (Docker configurations)
│   ├── compose/ (Docker Compose files)
│   └── Dockerfiles (for each service)
├── Documentation (*.md files)
└── Configuration files

📊 Project Size Analysis

1. Source Code Size

Location: /root/MozDef

Size Breakdown:

  • Source code: ~50-200 MB (varies)
  • Documentation: ~1-5 MB
  • Configuration files: ~1-10 MB

Check size:

du -sh /root/MozDef
du -sh /root/MozDef/* | sort -h

2. Docker Images Size

Location: Docker's storage directory (usually /var/lib/docker/)

Total Images: 17 MozDef images

Estimated Size:

  • Base images: ~500 MB - 2 GB each
  • Application images: ~200 MB - 1 GB each
  • Total: ~5-15 GB (depends on base images)

Check Docker images:

docker images mozdef/* --format "{{.Repository}}:{{.Tag}} {{.Size}}"
docker system df

3. Docker Volumes (Data)

Location: /var/lib/docker/volumes/

Volumes:

  • mozdef_elasticsearch (event data)
  • mozdef_mongodb (Meteor data)
  • mozdef_rabbitmq (queue data)
  • mozdef_geolite_db (GeoIP data)

Size: Varies based on data (can be GBs for production)

Check volumes:

docker volume ls | grep mozdef
docker system df -v

📁 Key File Locations

Docker Files 

/root/MozDef/docker/compose/
├── docker-compose.yml          # Main compose file
├── */Dockerfile                # Service Dockerfiles
└── */files/                    # Configuration files

Source Code 

/root/MozDef/
├── loginput/                   # Loginput service
├── rest/                       # REST API service
├── mq/                         # MQ Worker
├── alerts/                     # Alerts service
├── meteor/                     # Meteor frontend
└── config/                     # Configuration files

Docker Images (Runtime)

Location: Docker's internal storage

  • Default: /var/lib/docker/
  • Check: docker info | grep "Docker Root Dir"

📦 Creating Archive for Transfer

Method 1: Complete Archive (Recommended)

Step 1: Save Docker Images 

# Create archive directory
ARCHIVE_DIR="$HOME/mozdef-archive-$(date +%Y%m%d)"
mkdir -p "$ARCHIVE_DIR/docker-images"

# Save all MozDef images
docker save $(docker images mozdef/* --format "{{.Repository}}:{{.Tag}}") \
  -o "$ARCHIVE_DIR/docker-images/mozdef-all-images.tar"

# Check size
ls -lh "$ARCHIVE_DIR/docker-images/mozdef-all-images.tar"

Step 2: Archive Source Code 

# Archive source code
tar -czf "$ARCHIVE_DIR/source-code/MozDef-source.tar.gz" \
  -C /root MozDef \
  --exclude='MozDef/.git' \
  --exclude='MozDef/node_modules' \
  --exclude='MozDef/.meteor/local' \
  --exclude='MozDef/__pycache__'

# Check size
ls -lh "$ARCHIVE_DIR/source-code/MozDef-source.tar.gz"

Step 3: Save Docker Volumes (Optional - for data) 

# Save Elasticsearch data
docker run --rm \
  -v mozdef_elasticsearch:/data \
  -v "$ARCHIVE_DIR":/backup \
  alpine tar czf /backup/elasticsearch-data.tar.gz -C /data .

# Save MongoDB data
docker run --rm \
  -v mozdef_mongodb:/data \
  -v "$ARCHIVE_DIR":/backup \
  alpine tar czf /backup/mongodb-data.tar.gz -C /data .

Step 4: Copy Documentation 

# Copy all documentation
cp /root/MozDef/*.md "$ARCHIVE_DIR/documentation/" 2>/dev/null

Step 5: Create Final Archive 

# Create single archive
cd "$(dirname "$ARCHIVE_DIR")"
tar -czf "mozdef-complete-$(date +%Y%m%d).tar.gz" "$(basename "$ARCHIVE_DIR")"

# Check final size
du -sh "mozdef-complete-$(date +%Y%m%d).tar.gz"

Method 2: Split Archive (for large files)

If archive is too large, split it:

# Split into 2GB chunks
split -b 2G mozdef-complete.tar.gz mozdef-complete.tar.gz.part

# Reassemble on target:
# cat mozdef-complete.tar.gz.part* > mozdef-complete.tar.gz

🚀 Transferring to Another Device

Option 1: USB Drive / External Storage 

# 1. Mount USB drive
sudo mkdir -p /mnt/usb
sudo mount /dev/sdX1 /mnt/usb

# 2. Copy archive
cp mozdef-complete-*.tar.gz /mnt/usb/

# 3. Unmount
sudo umount /mnt/usb

Option 2: Network Transfer (SCP) 

# From source machine
scp mozdef-complete-*.tar.gz user@target-machine:/path/to/destination/

# Or using rsync (better for large files)
rsync -avz --progress mozdef-complete-*.tar.gz user@target-machine:/path/

Option 3: Network Share (NFS/SMB) 

# Mount network share
sudo mount -t nfs server:/share /mnt/nfs

# Copy archive
cp mozdef-complete-*.tar.gz /mnt/nfs/

Option 4: Cloud Storage (if allowed) 

# Upload to cloud storage
# Example with AWS S3:
aws s3 cp mozdef-complete-*.tar.gz s3://bucket-name/

# Download on target:
aws s3 cp s3://bucket-name/mozdef-complete-*.tar.gz ./

📥 Restoring on Target Device

Step 1: Extract Archive 

# Extract archive
tar -xzf mozdef-complete-*.tar.gz
cd mozdef-archive-*/

Step 2: Load Docker Images 

# Load Docker images
docker load -i docker-images/mozdef-all-images.tar

# Verify images loaded
docker images mozdef/*

Step 3: Extract Source Code 

# Extract source code
tar -xzf source-code/MozDef-source.tar.gz -C /root/

# Verify
ls -la /root/MozDef

Step 4: Restore Volumes (if needed) 

# Create volumes first
docker volume create mozdef_elasticsearch
docker volume create mozdef_mongodb
docker volume create mozdef_rabbitmq
docker volume create mozdef_geolite_db

# Restore data
docker run --rm \
  -v mozdef_elasticsearch:/data \
  -v "$(pwd)":/backup \
  alpine tar xzf /backup/elasticsearch-data.tar.gz -C /data

Step 5: Start Services 

# Navigate to project
cd /root/MozDef

# Start services
docker-compose -f docker/compose/docker-compose.yml -p mozdef up -d

# Verify
docker-compose -f docker/compose/docker-compose.yml -p mozdef ps

📋 Quick Archive Script

Save this script as archive_mozdef.sh:

#!/bin/bash
# MozDef Complete Archive Script

ARCHIVE_DIR="$HOME/mozdef-archive-$(date +%Y%m%d)"
mkdir -p "$ARCHIVE_DIR"/{docker-images,source-code,documentation}

echo "=== Creating MozDef Archive ==="
echo ""

# 1. Save Docker Images
echo "1/4 Saving Docker images..."
docker save $(docker images mozdef/* --format "{{.Repository}}:{{.Tag}}") \
  -o "$ARCHIVE_DIR/docker-images/mozdef-all-images.tar"
echo "   ✅ Images saved: $(du -sh "$ARCHIVE_DIR/docker-images/mozdef-all-images.tar" | awk '{print $1}')"

# 2. Archive Source Code
echo "2/4 Archiving source code..."
tar -czf "$ARCHIVE_DIR/source-code/MozDef-source.tar.gz" \
  -C /root MozDef \
  --exclude='MozDef/.git' \
  --exclude='MozDef/node_modules' \
  --exclude='MozDef/.meteor/local'
echo "   ✅ Source archived: $(du -sh "$ARCHIVE_DIR/source-code/MozDef-source.tar.gz" | awk '{print $1}')"

# 3. Copy Documentation
echo "3/4 Copying documentation..."
cp /root/MozDef/*.md "$ARCHIVE_DIR/documentation/" 2>/dev/null
echo "   ✅ Documentation copied"

# 4. Create Final Archive
echo "4/4 Creating final archive..."
cd "$(dirname "$ARCHIVE_DIR")"
tar -czf "mozdef-complete-$(date +%Y%m%d).tar.gz" "$(basename "$ARCHIVE_DIR")"
echo "   ✅ Final archive: $(du -sh "mozdef-complete-$(date +%Y%m%d).tar.gz" | awk '{print $1}')"

echo ""
echo "=== Archive Complete ==="
echo "Location: $(pwd)/mozdef-complete-$(date +%Y%m%d).tar.gz"

Usage:

chmod +x archive_mozdef.sh
./archive_mozdef.sh

📊 Estimated Archive Sizes

Minimal Archive (Images + Source)

  • Docker Images: ~5-15 GB
  • Source Code: ~100-500 MB
  • Total: ~5-16 GB

Complete Archive (with data)

  • Docker Images: ~5-15 GB
  • Source Code: ~100-500 MB
  • Volumes (data): Varies (can be GBs)
  • Total: Depends on data size

✅ Pre-Transfer Checklist

  • All Docker images saved
  • Source code archived
  • Documentation included
  • Volumes backed up (if needed)
  • Archive size verified
  • Archive integrity checked (optional: md5sum)

🔍 Verification Commands

Check Archive Contents 

# List archive contents
tar -tzf mozdef-complete-*.tar.gz | head -20

# Check archive integrity
tar -tzf mozdef-complete-*.tar.gz > /dev/null && echo "Archive OK"

Verify Docker Images 

# After loading
docker images mozdef/* | wc -l
# Should show 17 images

Verify Source Code 

# After extracting
ls -la /root/MozDef
# Should show all project files

📝 Summary

Project Location: /root/MozDef

Key Components:

  1. Source Code: /root/MozDef (~100-500 MB)
  2. Docker Images: Docker storage (~5-15 GB)
  3. Docker Files: /root/MozDef/docker/
  4. Volumes (Data): /var/lib/docker/volumes/ (varies)

Archive Creation:

  • Save images: docker save
  • Archive source: tar -czf
  • Create final: Single tar.gz file

Transfer Methods:

  • USB drive
  • Network (SCP/rsync)
  • Network share
  • Cloud storage (if allowed)

Restore Process:

  1. Extract archive
  2. Load Docker images
  3. Extract source code
  4. Start services

Last Updated: $(date)