docs/RESTIC_USAGE.md

# Restic Data Backup and Restore Usage Guide

## Overview

Restic is a fast, secure, and efficient backup tool that is now fully integrated into the HuggingFace environment, supporting automatic backup and restore functionality.

## Quick Start

### 1. Basic Configuration

Copy the configuration template and modify as needed:

```bash
# Configuration file is already available in container
# Edit configuration file directly
nano /home/user/config/restic.conf
```

Basic configuration:

```bash
# Enable Restic service
RESTIC_ENABLED="true"

# Set backup paths
RESTIC_BACKUP_PATHS="/home/user/data:/home/user/config"

# Auto-restore on HuggingFace restart
RESTIC_AUTO_RESTORE="true"
RESTIC_RESTORE_MODE="replace"
RESTIC_FORCE_RESTORE="true"
```

### 2. Start Service

```bash
# Start Restic service
./scripts/start/restic-start.sh

# Or auto-start in Docker container
# Add Restic startup script in docker-entrypoint.sh
```

## Core Features

### Automatic Backup

- **Scheduled Backup**: Automatic backup every hour by default
- **Startup Backup**: Automatic backup when service starts
- **Incremental Backup**: Only backup changed data to save space
- **Deduplication and Compression**: Automatic deduplication and compression for optimized storage

### Automatic Restore

- **Startup Restore**: Automatic data restore when HuggingFace restarts
- **Precise Restore**: Only restore specified directories without affecting other files
- **Multiple Modes**:
  - `safe`: Safe mode, restore to temporary directory
  - `direct`: Direct mode, restore directly to target location
  - `replace`: Replace mode, clear then restore (recommended for HF restart)

### Data Management

- **Snapshot Management**: Keep multiple historical versions
- **Auto Cleanup**: Automatically clean old snapshots according to retention policy
- **Integrity Check**: Regularly check backup integrity

## Usage Scenarios

### Scenario 1: Basic HuggingFace Environment Backup

```bash
# Configuration file example
RESTIC_ENABLED="true"
RESTIC_BACKUP_PATHS="/home/user/data:/home/user/config"
RESTIC_AUTO_RESTORE="true"
RESTIC_RESTORE_MODE="replace"
RESTIC_BACKUP_INTERVAL="3600"  # 1 hour
RESTIC_KEEP_DAILY="7"          # Keep 7 days
```

**Suitable for**: Most HuggingFace deployment scenarios

### Scenario 2: Production Environment High Availability

```bash
# Use remote S3 storage
RESTIC_REPOSITORY_PATH="s3:s3.amazonaws.com/my-hf-backup"
RESTIC_BACKUP_INTERVAL="1800"  # 30 minutes
RESTIC_KEEP_DAILY="30"         # Keep 30 days
RESTIC_VERIFY_BACKUP="true"    # Verify backup
```

**Suitable for**: Critical production environments

### Scenario 3: Development and Testing Environment

```bash
# Lower frequency backup
RESTIC_BACKUP_INTERVAL="7200"  # 2 hours
RESTIC_AUTO_RESTORE="false"    # Manual restore
RESTIC_KEEP_DAILY="3"          # Keep 3 days
```

**Suitable for**: Development and testing environments

## Command Line Tools

### Restore Script

Use the dedicated restore script:

```bash
# Show help
./scripts/utils/restic-restore.sh --help

# List available snapshots
./scripts/utils/restic-restore.sh --list

# Safe mode restore (recommended)
./scripts/utils/restic-restore.sh --mode safe

# Direct restore (fast)
./scripts/utils/restic-restore.sh --mode direct --force

# Replace mode restore (recommended for HF restart)
./scripts/utils/restic-restore.sh --mode replace --force

# Only restore config files
./scripts/utils/restic-restore.sh --include "/home/user/config"

# Restore specific snapshot
./scripts/utils/restic-restore.sh a1b2c3d4
```

### Direct Restic Commands

```bash
# Set environment variables
export RESTIC_REPOSITORY="/home/user/data/restic-repo"
export RESTIC_PASSWORD="$(cat /home/user/config/restic-password)"

# View snapshots
restic snapshots

# Manual backup
restic backup /home/user/data --tag manual

# Manual restore
restic restore latest --target /tmp/restore

# Check repository
restic check

# Clean old snapshots
restic forget --keep-daily 7 --prune
```

## Configuration Options Details

### Required Configuration

```bash
RESTIC_ENABLED="true"                        # Enable service
RESTIC_REPOSITORY_PATH="/path/to/repo"       # Repository path
RESTIC_BACKUP_PATHS="/path1:/path2"          # Backup paths (colon-separated)
```

### Backup Configuration

```bash
RESTIC_BACKUP_INTERVAL="3600"               # Backup interval (seconds)
RESTIC_BACKUP_TAG="automated,hf"            # Backup tags
RESTIC_BACKUP_ON_STARTUP="true"             # Backup on startup
RESTIC_EXCLUDE_PATTERNS="*.tmp,*.log"       # Exclude patterns
```

### Restore Configuration

```bash
RESTIC_AUTO_RESTORE="true"                  # Auto restore
RESTIC_RESTORE_MODE="replace"               # Restore mode
RESTIC_FORCE_RESTORE="true"                 # Force restore
RESTIC_RESTORE_INCLUDE_PATHS="/home/user"   # Only restore specified paths
```

### Retention Policy

```bash
RESTIC_KEEP_HOURLY="24"                     # Keep hourly backups for 24 hours
RESTIC_KEEP_DAILY="7"                       # Keep daily backups for 7 days
RESTIC_KEEP_WEEKLY="4"                      # Keep weekly backups for 4 weeks
RESTIC_KEEP_MONTHLY="12"                    # Keep monthly backups for 12 months
```

### Performance Optimization

```bash
RESTIC_COMPRESSION="auto"                   # Compression mode: off, auto, max
RESTIC_BACKEND_CONNECTIONS="5"              # Backend connections
RESTIC_PACK_SIZE="32"                       # Pack size in MiB
RESTIC_READ_CONCURRENCY="4"                 # Read concurrency
RESTIC_NO_SCAN="false"                      # Disable backup progress estimation
```

### Advanced Options

```bash
RESTIC_EXTRA_VERIFY="true"                  # Enable extra data verification
RESTIC_CHECK_INTERVAL="259200"              # Repository check interval (3 days)
RESTIC_AUTO_REPAIR="true"                   # Auto-repair index if needed
RESTIC_CACHE_SIZE="1000"                    # Cache size in MB
RESTIC_LOG_LEVEL="INFO"                     # Log level: DEBUG, INFO, WARN, ERROR
```

## Best Practices

### For HuggingFace Environment

1. **Use Replace Mode for Auto-restore**: Set `RESTIC_RESTORE_MODE="replace"` for HF restart scenarios
2. **Enable Force Restore**: Set `RESTIC_FORCE_RESTORE="true"` to avoid prompts
3. **Optimize Backup Frequency**: Use 1-2 hours interval for active development
4. **Monitor Storage Usage**: Adjust retention policy based on available space

### Security Considerations

1. **Password Management**: Use strong auto-generated passwords
2. **Repository Access**: Restrict access to backup repository
3. **Data Verification**: Enable extra verification for critical data
4. **Regular Testing**: Test restore procedures regularly

### Performance Tips

1. **Use Compression**: Enable auto compression for better storage efficiency
2. **Adjust Concurrency**: Increase read concurrency for fast storage
3. **Cache Optimization**: Set appropriate cache size based on available memory
4. **Network Optimization**: Increase backend connections for remote repositories

## Troubleshooting

### Common Issues

| Issue | Solution |
|-------|----------|
| Service not starting | Check `RESTIC_ENABLED="true"` and configuration file |
| Backup fails | Verify repository path and permissions |
| Restore fails | Check snapshot availability and target path permissions |
| High memory usage | Reduce pack size and backend connections |
| Slow performance | Enable no-scan mode and adjust concurrency |

### Debug Commands

```bash
# Check service status
ps aux | grep restic

# View logs
tail -f /home/user/log/restic.log

# Test repository
restic check --read-data

# View backup statistics
restic stats

# Check cache
restic cache --help
```

### Performance Monitoring

```bash
# Monitor backup progress
restic backup --verbose /path/to/data

# Check repository size
restic stats --mode raw-data

# Analyze snapshot differences
restic diff snapshot1 snapshot2
```

## Migration Guide

### From Persistence to Restic

1. Disable persistence service: `PERSISTENCE_ENABLED="false"`
2. Enable restic service: `RESTIC_ENABLED="true"`
3. Configure backup paths to include persistence data
4. Test backup and restore functionality
5. Update startup scripts if needed

### Backup Format Compatibility

Restic repositories are not compatible with other backup formats. For migration:

1. Create full backup with old system
2. Initialize new restic repository
3. Perform initial backup with restic
4. Verify data integrity
5. Switch to restic service

## FAQ

**Q: Can I use both persistence and restic services?**
A: No, they are mutually exclusive. Choose one based on your needs.

**Q: How much space do backups consume?**
A: Restic uses deduplication and compression, typically reducing space by 50-80%.

**Q: Can I backup to cloud storage?**
A: Yes, restic supports S3, Azure, Google Cloud, and other backends.

**Q: How to recover from corrupted repository?**
A: Run `restic check --read-data` and `restic repair index` to fix issues.

**Q: Is encryption enabled by default?**
A: Yes, restic encrypts all data with AES-256.