Mozdef / PROJECT_PACKAGING_GUIDE.md
ineso22's picture
Upload folder using huggingface_hub
7c89ed7 verified
# MozDef - Project Packaging & Transfer Guide
## πŸ“¦ Project Size & Location
### Project Structure
```
/root/MozDef/
β”œβ”€β”€ Source Code (application files)
β”œβ”€β”€ docker/ (Docker configurations)
β”‚ β”œβ”€β”€ compose/ (Docker Compose files)
β”‚ └── Dockerfiles (for each service)
β”œβ”€β”€ Documentation (*.md files)
└── Configuration files
```
---
## πŸ“Š Project Size Analysis
### 1. Source Code Size
**Location:** `/root/MozDef`
**Size Breakdown:**
- Source code: ~50-200 MB (varies)
- Documentation: ~1-5 MB
- Configuration files: ~1-10 MB
**Check size:**
```bash
du -sh /root/MozDef
du -sh /root/MozDef/* | sort -h
```
### 2. Docker Images Size
**Location:** Docker's storage directory (usually `/var/lib/docker/`)
**Total Images:** 17 MozDef images
**Estimated Size:**
- Base images: ~500 MB - 2 GB each
- Application images: ~200 MB - 1 GB each
- **Total: ~5-15 GB** (depends on base images)
**Check Docker images:**
```bash
docker images mozdef/* --format "{{.Repository}}:{{.Tag}} {{.Size}}"
docker system df
```
### 3. Docker Volumes (Data)
**Location:** `/var/lib/docker/volumes/`
**Volumes:**
- `mozdef_elasticsearch` (event data)
- `mozdef_mongodb` (Meteor data)
- `mozdef_rabbitmq` (queue data)
- `mozdef_geolite_db` (GeoIP data)
**Size:** Varies based on data (can be GBs for production)
**Check volumes:**
```bash
docker volume ls | grep mozdef
docker system df -v
```
---
## πŸ“ Key File Locations
### Docker Files
```
/root/MozDef/docker/compose/
β”œβ”€β”€ docker-compose.yml # Main compose file
β”œβ”€β”€ */Dockerfile # Service Dockerfiles
└── */files/ # Configuration files
```
### Source Code
```
/root/MozDef/
β”œβ”€β”€ loginput/ # Loginput service
β”œβ”€β”€ rest/ # REST API service
β”œβ”€β”€ mq/ # MQ Worker
β”œβ”€β”€ alerts/ # Alerts service
β”œβ”€β”€ meteor/ # Meteor frontend
└── config/ # Configuration files
```
### Docker Images (Runtime)
**Location:** Docker's internal storage
- Default: `/var/lib/docker/`
- Check: `docker info | grep "Docker Root Dir"`
---
## πŸ“¦ Creating Archive for Transfer
### Method 1: Complete Archive (Recommended)
#### Step 1: Save Docker Images
```bash
# Create archive directory
ARCHIVE_DIR="$HOME/mozdef-archive-$(date +%Y%m%d)"
mkdir -p "$ARCHIVE_DIR/docker-images"
# Save all MozDef images
docker save $(docker images mozdef/* --format "{{.Repository}}:{{.Tag}}") \
-o "$ARCHIVE_DIR/docker-images/mozdef-all-images.tar"
# Check size
ls -lh "$ARCHIVE_DIR/docker-images/mozdef-all-images.tar"
```
#### Step 2: Archive Source Code
```bash
# Archive source code
tar -czf "$ARCHIVE_DIR/source-code/MozDef-source.tar.gz" \
-C /root MozDef \
--exclude='MozDef/.git' \
--exclude='MozDef/node_modules' \
--exclude='MozDef/.meteor/local' \
--exclude='MozDef/__pycache__'
# Check size
ls -lh "$ARCHIVE_DIR/source-code/MozDef-source.tar.gz"
```
#### Step 3: Save Docker Volumes (Optional - for data)
```bash
# Save Elasticsearch data
docker run --rm \
-v mozdef_elasticsearch:/data \
-v "$ARCHIVE_DIR":/backup \
alpine tar czf /backup/elasticsearch-data.tar.gz -C /data .
# Save MongoDB data
docker run --rm \
-v mozdef_mongodb:/data \
-v "$ARCHIVE_DIR":/backup \
alpine tar czf /backup/mongodb-data.tar.gz -C /data .
```
#### Step 4: Copy Documentation
```bash
# Copy all documentation
cp /root/MozDef/*.md "$ARCHIVE_DIR/documentation/" 2>/dev/null
```
#### Step 5: Create Final Archive
```bash
# Create single archive
cd "$(dirname "$ARCHIVE_DIR")"
tar -czf "mozdef-complete-$(date +%Y%m%d).tar.gz" "$(basename "$ARCHIVE_DIR")"
# Check final size
du -sh "mozdef-complete-$(date +%Y%m%d).tar.gz"
```
### Method 2: Split Archive (for large files)
If archive is too large, split it:
```bash
# Split into 2GB chunks
split -b 2G mozdef-complete.tar.gz mozdef-complete.tar.gz.part
# Reassemble on target:
# cat mozdef-complete.tar.gz.part* > mozdef-complete.tar.gz
```
---
## πŸš€ Transferring to Another Device
### Option 1: USB Drive / External Storage
```bash
# 1. Mount USB drive
sudo mkdir -p /mnt/usb
sudo mount /dev/sdX1 /mnt/usb
# 2. Copy archive
cp mozdef-complete-*.tar.gz /mnt/usb/
# 3. Unmount
sudo umount /mnt/usb
```
### Option 2: Network Transfer (SCP)
```bash
# From source machine
scp mozdef-complete-*.tar.gz user@target-machine:/path/to/destination/
# Or using rsync (better for large files)
rsync -avz --progress mozdef-complete-*.tar.gz user@target-machine:/path/
```
### Option 3: Network Share (NFS/SMB)
```bash
# Mount network share
sudo mount -t nfs server:/share /mnt/nfs
# Copy archive
cp mozdef-complete-*.tar.gz /mnt/nfs/
```
### Option 4: Cloud Storage (if allowed)
```bash
# Upload to cloud storage
# Example with AWS S3:
aws s3 cp mozdef-complete-*.tar.gz s3://bucket-name/
# Download on target:
aws s3 cp s3://bucket-name/mozdef-complete-*.tar.gz ./
```
---
## πŸ“₯ Restoring on Target Device
### Step 1: Extract Archive
```bash
# Extract archive
tar -xzf mozdef-complete-*.tar.gz
cd mozdef-archive-*/
```
### Step 2: Load Docker Images
```bash
# Load Docker images
docker load -i docker-images/mozdef-all-images.tar
# Verify images loaded
docker images mozdef/*
```
### Step 3: Extract Source Code
```bash
# Extract source code
tar -xzf source-code/MozDef-source.tar.gz -C /root/
# Verify
ls -la /root/MozDef
```
### Step 4: Restore Volumes (if needed)
```bash
# Create volumes first
docker volume create mozdef_elasticsearch
docker volume create mozdef_mongodb
docker volume create mozdef_rabbitmq
docker volume create mozdef_geolite_db
# Restore data
docker run --rm \
-v mozdef_elasticsearch:/data \
-v "$(pwd)":/backup \
alpine tar xzf /backup/elasticsearch-data.tar.gz -C /data
```
### Step 5: Start Services
```bash
# Navigate to project
cd /root/MozDef
# Start services
docker-compose -f docker/compose/docker-compose.yml -p mozdef up -d
# Verify
docker-compose -f docker/compose/docker-compose.yml -p mozdef ps
```
---
## πŸ“‹ Quick Archive Script
Save this script as `archive_mozdef.sh`:
```bash
#!/bin/bash
# MozDef Complete Archive Script
ARCHIVE_DIR="$HOME/mozdef-archive-$(date +%Y%m%d)"
mkdir -p "$ARCHIVE_DIR"/{docker-images,source-code,documentation}
echo "=== Creating MozDef Archive ==="
echo ""
# 1. Save Docker Images
echo "1/4 Saving Docker images..."
docker save $(docker images mozdef/* --format "{{.Repository}}:{{.Tag}}") \
-o "$ARCHIVE_DIR/docker-images/mozdef-all-images.tar"
echo " βœ… Images saved: $(du -sh "$ARCHIVE_DIR/docker-images/mozdef-all-images.tar" | awk '{print $1}')"
# 2. Archive Source Code
echo "2/4 Archiving source code..."
tar -czf "$ARCHIVE_DIR/source-code/MozDef-source.tar.gz" \
-C /root MozDef \
--exclude='MozDef/.git' \
--exclude='MozDef/node_modules' \
--exclude='MozDef/.meteor/local'
echo " βœ… Source archived: $(du -sh "$ARCHIVE_DIR/source-code/MozDef-source.tar.gz" | awk '{print $1}')"
# 3. Copy Documentation
echo "3/4 Copying documentation..."
cp /root/MozDef/*.md "$ARCHIVE_DIR/documentation/" 2>/dev/null
echo " βœ… Documentation copied"
# 4. Create Final Archive
echo "4/4 Creating final archive..."
cd "$(dirname "$ARCHIVE_DIR")"
tar -czf "mozdef-complete-$(date +%Y%m%d).tar.gz" "$(basename "$ARCHIVE_DIR")"
echo " βœ… Final archive: $(du -sh "mozdef-complete-$(date +%Y%m%d).tar.gz" | awk '{print $1}')"
echo ""
echo "=== Archive Complete ==="
echo "Location: $(pwd)/mozdef-complete-$(date +%Y%m%d).tar.gz"
```
**Usage:**
```bash
chmod +x archive_mozdef.sh
./archive_mozdef.sh
```
---
## πŸ“Š Estimated Archive Sizes
### Minimal Archive (Images + Source)
- **Docker Images:** ~5-15 GB
- **Source Code:** ~100-500 MB
- **Total:** ~5-16 GB
### Complete Archive (with data)
- **Docker Images:** ~5-15 GB
- **Source Code:** ~100-500 MB
- **Volumes (data):** Varies (can be GBs)
- **Total:** Depends on data size
---
## βœ… Pre-Transfer Checklist
- [ ] All Docker images saved
- [ ] Source code archived
- [ ] Documentation included
- [ ] Volumes backed up (if needed)
- [ ] Archive size verified
- [ ] Archive integrity checked (optional: `md5sum`)
---
## πŸ” Verification Commands
### Check Archive Contents
```bash
# List archive contents
tar -tzf mozdef-complete-*.tar.gz | head -20
# Check archive integrity
tar -tzf mozdef-complete-*.tar.gz > /dev/null && echo "Archive OK"
```
### Verify Docker Images
```bash
# After loading
docker images mozdef/* | wc -l
# Should show 17 images
```
### Verify Source Code
```bash
# After extracting
ls -la /root/MozDef
# Should show all project files
```
---
## πŸ“ Summary
**Project Location:** `/root/MozDef`
**Key Components:**
1. **Source Code:** `/root/MozDef` (~100-500 MB)
2. **Docker Images:** Docker storage (~5-15 GB)
3. **Docker Files:** `/root/MozDef/docker/`
4. **Volumes (Data):** `/var/lib/docker/volumes/` (varies)
**Archive Creation:**
- Save images: `docker save`
- Archive source: `tar -czf`
- Create final: Single tar.gz file
**Transfer Methods:**
- USB drive
- Network (SCP/rsync)
- Network share
- Cloud storage (if allowed)
**Restore Process:**
1. Extract archive
2. Load Docker images
3. Extract source code
4. Start services
---
**Last Updated:** $(date)