docs/RESTIC_USAGE.md

Restic Data Backup and Restore Usage Guide

Overview

Restic is a fast, secure, and efficient backup tool that is now fully integrated into the HuggingFace environment, supporting automatic backup and restore functionality.

Quick Start

1. Basic Configuration

Copy the configuration template and modify as needed:

# Configuration file is already available in container
# Edit configuration file directly
nano /home/user/config/restic.conf

Basic configuration:

# Enable Restic service
RESTIC_ENABLED="true"

# Set backup paths
RESTIC_BACKUP_PATHS="/home/user/data:/home/user/config"

# Auto-restore on HuggingFace restart
RESTIC_AUTO_RESTORE="true"
RESTIC_RESTORE_MODE="replace"
RESTIC_FORCE_RESTORE="true"

2. Start Service

# Start Restic service
./scripts/start/restic-start.sh

# Or auto-start in Docker container
# Add Restic startup script in docker-entrypoint.sh

Core Features

Automatic Backup

• Scheduled Backup: Automatic backup every hour by default
• Startup Backup: Automatic backup when service starts
• Incremental Backup: Only backup changed data to save space
• Deduplication and Compression: Automatic deduplication and compression for optimized storage

Automatic Restore

• Startup Restore: Automatic data restore when HuggingFace restarts
• Precise Restore: Only restore specified directories without affecting other files
• Multiple Modes:
  • safe: Safe mode, restore to temporary directory
  • direct: Direct mode, restore directly to target location
  • replace: Replace mode, clear then restore (recommended for HF restart)

Data Management

• Snapshot Management: Keep multiple historical versions
• Auto Cleanup: Automatically clean old snapshots according to retention policy
• Integrity Check: Regularly check backup integrity

Usage Scenarios

Scenario 1: Basic HuggingFace Environment Backup

# Configuration file example
RESTIC_ENABLED="true"
RESTIC_BACKUP_PATHS="/home/user/data:/home/user/config"
RESTIC_AUTO_RESTORE="true"
RESTIC_RESTORE_MODE="replace"
RESTIC_BACKUP_INTERVAL="3600"  # 1 hour
RESTIC_KEEP_DAILY="7"          # Keep 7 days

Suitable for: Most HuggingFace deployment scenarios

Scenario 2: Production Environment High Availability

# Use remote S3 storage
RESTIC_REPOSITORY_PATH="s3:s3.amazonaws.com/my-hf-backup"
RESTIC_BACKUP_INTERVAL="1800"  # 30 minutes
RESTIC_KEEP_DAILY="30"         # Keep 30 days
RESTIC_VERIFY_BACKUP="true"    # Verify backup

Suitable for: Critical production environments

Scenario 3: Development and Testing Environment

# Lower frequency backup
RESTIC_BACKUP_INTERVAL="7200"  # 2 hours
RESTIC_AUTO_RESTORE="false"    # Manual restore
RESTIC_KEEP_DAILY="3"          # Keep 3 days

Suitable for: Development and testing environments

Command Line Tools

Restore Script

Use the dedicated restore script:

# Show help
./scripts/utils/restic-restore.sh --help

# List available snapshots
./scripts/utils/restic-restore.sh --list

# Safe mode restore (recommended)
./scripts/utils/restic-restore.sh --mode safe

# Direct restore (fast)
./scripts/utils/restic-restore.sh --mode direct --force

# Replace mode restore (recommended for HF restart)
./scripts/utils/restic-restore.sh --mode replace --force

# Only restore config files
./scripts/utils/restic-restore.sh --include "/home/user/config"

# Restore specific snapshot
./scripts/utils/restic-restore.sh a1b2c3d4

Direct Restic Commands

# Set environment variables
export RESTIC_REPOSITORY="/home/user/data/restic-repo"
export RESTIC_PASSWORD="$(cat /home/user/config/restic-password)"

# View snapshots
restic snapshots

# Manual backup
restic backup /home/user/data --tag manual

# Manual restore
restic restore latest --target /tmp/restore

# Check repository
restic check

# Clean old snapshots
restic forget --keep-daily 7 --prune

Configuration Options Details

Required Configuration

RESTIC_ENABLED="true"                        # Enable service
RESTIC_REPOSITORY_PATH="/path/to/repo"       # Repository path
RESTIC_BACKUP_PATHS="/path1:/path2"          # Backup paths (colon-separated)

Backup Configuration

RESTIC_BACKUP_INTERVAL="3600"               # Backup interval (seconds)
RESTIC_BACKUP_TAG="automated,hf"            # Backup tags
RESTIC_BACKUP_ON_STARTUP="true"             # Backup on startup
RESTIC_EXCLUDE_PATTERNS="*.tmp,*.log"       # Exclude patterns

Restore Configuration

RESTIC_AUTO_RESTORE="true"                  # Auto restore
RESTIC_RESTORE_MODE="replace"               # Restore mode
RESTIC_FORCE_RESTORE="true"                 # Force restore
RESTIC_RESTORE_INCLUDE_PATHS="/home/user"   # Only restore specified paths

Retention Policy

RESTIC_KEEP_HOURLY="24"                     # Keep hourly backups for 24 hours
RESTIC_KEEP_DAILY="7"                       # Keep daily backups for 7 days
RESTIC_KEEP_WEEKLY="4"                      # Keep weekly backups for 4 weeks
RESTIC_KEEP_MONTHLY="12"                    # Keep monthly backups for 12 months

Performance Optimization

RESTIC_COMPRESSION="auto"                   # Compression mode: off, auto, max
RESTIC_BACKEND_CONNECTIONS="5"              # Backend connections
RESTIC_PACK_SIZE="32"                       # Pack size in MiB
RESTIC_READ_CONCURRENCY="4"                 # Read concurrency
RESTIC_NO_SCAN="false"                      # Disable backup progress estimation

Advanced Options

RESTIC_EXTRA_VERIFY="true"                  # Enable extra data verification
RESTIC_CHECK_INTERVAL="259200"              # Repository check interval (3 days)
RESTIC_AUTO_REPAIR="true"                   # Auto-repair index if needed
RESTIC_CACHE_SIZE="1000"                    # Cache size in MB
RESTIC_LOG_LEVEL="INFO"                     # Log level: DEBUG, INFO, WARN, ERROR

Best Practices

For HuggingFace Environment

1. Use Replace Mode for Auto-restore: Set RESTIC_RESTORE_MODE="replace" for HF restart scenarios
2. Enable Force Restore: Set RESTIC_FORCE_RESTORE="true" to avoid prompts
3. Optimize Backup Frequency: Use 1-2 hours interval for active development
4. Monitor Storage Usage: Adjust retention policy based on available space

Security Considerations

1. Password Management: Use strong auto-generated passwords
2. Repository Access: Restrict access to backup repository
3. Data Verification: Enable extra verification for critical data
4. Regular Testing: Test restore procedures regularly

Performance Tips

1. Use Compression: Enable auto compression for better storage efficiency
2. Adjust Concurrency: Increase read concurrency for fast storage
3. Cache Optimization: Set appropriate cache size based on available memory
4. Network Optimization: Increase backend connections for remote repositories

Troubleshooting

Common Issues

Issue	Solution
Service not starting	Check RESTIC_ENABLED="true" and configuration file
Backup fails	Verify repository path and permissions
Restore fails	Check snapshot availability and target path permissions
High memory usage	Reduce pack size and backend connections
Slow performance	Enable no-scan mode and adjust concurrency

Debug Commands

# Check service status
ps aux | grep restic

# View logs
tail -f /home/user/log/restic.log

# Test repository
restic check --read-data

# View backup statistics
restic stats

# Check cache
restic cache --help

Performance Monitoring

# Monitor backup progress
restic backup --verbose /path/to/data

# Check repository size
restic stats --mode raw-data

# Analyze snapshot differences
restic diff snapshot1 snapshot2

Migration Guide

From Persistence to Restic

1. Disable persistence service: PERSISTENCE_ENABLED="false"
2. Enable restic service: RESTIC_ENABLED="true"
3. Configure backup paths to include persistence data
4. Test backup and restore functionality
5. Update startup scripts if needed

Backup Format Compatibility

Restic repositories are not compatible with other backup formats. For migration:

1. Create full backup with old system
2. Initialize new restic repository
3. Perform initial backup with restic
4. Verify data integrity
5. Switch to restic service

FAQ

Q: Can I use both persistence and restic services? A: No, they are mutually exclusive. Choose one based on your needs.

Q: How much space do backups consume? A: Restic uses deduplication and compression, typically reducing space by 50-80%.

Q: Can I backup to cloud storage? A: Yes, restic supports S3, Azure, Google Cloud, and other backends.

Q: How to recover from corrupted repository? A: Run restic check --read-data and restic repair index to fix issues.

Q: Is encryption enabled by default? A: Yes, restic encrypts all data with AES-256.