Enhance job initialization and progress tracking; update .gitignore to exclude cache directories

Refactor Redis connection handling and update Dockerfile for environment variable support