Hugging Face's logo

Spaces:
printf-sourav
/
CICD_DEBUGGER
Configuration error

App Files Community
CICD_DEBUGGER / README.md
printf-sourav's picture
printf-sourav
AI completed
276a6a0
preview code
|
Raw
History Blame Contribute Delete
5.27 kB

CI/CD Pipeline Debugger Environment (OpenEnv)

1. Project Goal

This repository implements an AI training and evaluation environment where an agent learns to debug broken CI/CD pipelines automatically.

The environment targets real-world DevOps failure patterns, including:

  • YAML syntax and structure issues
  • Incorrect build/test commands (for example, npm tset -> npm test)
  • Dependency and setup failures
  • Multi-stage pipeline execution errors

This is designed as an RL-style interaction loop:

Observe -> Think -> Act -> Get Reward -> Repeat

2. Why This Matters

CI/CD failures are common, repetitive, and often multi-step to resolve. This project turns that workflow into a structured learning environment where agents:

  • Read failure context
  • Reason about root causes
  • Propose and apply fixes
  • Get shaped rewards for robust behavior

3. System Architecture

High-level flow:

Agent (LLM) -> Action -> Environment.step() -> Reward/Evaluation -> Next step

Core integration path:

Model -> Action -> Environment.step() -> RewardCalculator

RewardCalculator integrates:

  • DeterministicGrader
  • LLMJudge
  • HiddenTestRunner
  • AntiHackingDetector

4. Core Modules

4.1 Quality Judge

  • File: env/graders/llm_judge.py
  • Purpose: quality-aware scoring of fixes
  • Output keys: correctness, minimalism, quality (all in [0,1])
  • Guarantees:
    • strict JSON parsing attempt
    • robust fallback parsing for messy output
    • no-crash behavior (safe zero scores on failure)

4.2 Deterministic Grader

  • File: env/graders/deterministic.py
  • Purpose: reproducible correctness scoring (0-1)
  • Checks:
    • YAML validity
    • command and fix correctness
    • similarity and issue resolution
  • Rules:
    • deterministic only
    • same input, same score

4.3 Anti-Hacking Detector

  • File: env/anti_hacking.py
  • Purpose: detect reward-hacking and shortcut behavior
  • Penalty detectors:
    • stage skipping (if: false, when: never)
    • fake success (echo tests passed, unsafe exit 0 patterns)
    • pipeline breakage between versions
    • excessive edits
    • timeout abuse via too many steps

4.4 Hidden Tests

  • File: env/hidden_tests.py
  • Purpose: test fix robustness, not just exact-match overfitting
  • Method:
    • deterministic variant generation (OS, versions, env shifts)
    • evaluate pass rate across variants

4.5 Reward Shaping

  • File: env/rewards.py
  • Purpose: step-level learning signal
  • Components:
    • progress rewards (logs, analysis, fix proposal)
    • execution rewards (pipeline run, tests pass)
    • quality rewards (deterministic + hidden tests + LLM judge)
    • anti-hacking penalties

5. Inference and Evaluation

5.1 Prompt and Model Layers

  • inference/prompts.py: stable prompt templates and fallback action heuristics
  • inference/model_wrapper.py: OpenAI-client action generation, candidate generation, and safe fallback

5.2 Metrics and Artifacts

  • inference/metrics.py: reward, success-rate, and failure reason tracking
  • inference/visualize.py: reward curve and metrics artifact export

5.3 Submission-Critical Runtime

  • File: inference.py (root)
  • Responsibilities:
    • initialize model and environment
    • run step loop
    • calculate rewards
    • emit strict stdout contract
    • always emit END line

Required output format:

  • [START] task=... env=... model=...
  • [STEP] step= action=... reward=0.00 done=<true|false> error=<msg|null>
  • [END] success=<true|false> steps= rewards=<r1,r2,...>

Rules enforced:

  • single-line logs only
  • reward values with 2 decimals
  • lowercase booleans
  • no extra runtime log noise

6. Task Coverage

The project includes 13 CI-fix tasks spanning:

  • easy: syntax and typo fixes
  • medium: dependency/env/cache/permissions issues
  • hard: matrix logic, conditional flow, orchestration-level failures

7. Setup 

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Environment variables:

export API_BASE_URL="https://router.huggingface.co/v1"
export MODEL_NAME="Qwen/Qwen2.5-72B-Instruct"
export HF_TOKEN="<your_token>"
export LOCAL_IMAGE_NAME="<your_env_image_name>"

8. Run Inference

Offline/local mode:

python inference.py --offline --force-local-env --max-steps 8 --policy-mode imp --trajectories 4

Model-backed mode:

python inference.py --max-steps 8 --policy-mode imp --trajectories 4

Policy modes:

  • sft: deterministic heuristic policy
  • direct: single model action per step
  • imp: multi-candidate generation and ranking

9. Tests

Run all tests:

python -m unittest discover -s tests -v

Coverage includes:

  • LLM judge
  • deterministic grader
  • anti-hacking detectors
  • hidden tests
  • reward system
  • end-to-end inference output format

10. Validation and Submission

OpenEnv validation:

openenv validate

Pre-submission script:

./validate-submission.sh <your_hf_space_url>

Docker run:

docker build -t cicd-debugger-env .
docker run --rm -e OFFLINE_INFERENCE=1 cicd-debugger-env

11. One-line Presentation Summary

We built an OpenEnv-compliant reinforcement learning environment where AI agents learn to debug real CI/CD pipelines using multi-step reasoning, hybrid grading, anti-hacking safeguards, and robust reward shaping.