BATCH_CLI_IMPLEMENTATION.md

CLI Batch Processing Implementation Summary

Overview

Implemented a complete CLI batch processing mode for AI Agent Guards that shares code with the UI. This allows scanning multiple session files in batch, generating markdown reports, and integrating with CI/CD pipelines.

Date: 2026-01-29

---

✅ What Was Implemented

1. Core Shared Logic (multi_agent_demo/core/)

NEW FILES:

• __init__.py - Module exports
• scanner_runner.py - Shared scanner execution logic

Purpose: Extract scanner execution into reusable functions that both UI and CLI can use.

Key Functions:

• run_scanners_on_session() - Run scanners on single session
• aggregate_results() - Aggregate statistics across multiple sessions

2. Report Generation (multi_agent_demo/reports/)

NEW FILES:

• __init__.py - Module exports
• markdown_generator.py - Markdown report formatting

Purpose: Generate copyable markdown reports for CLI output.

Features:

• Overall statistics (total, safe, unsafe)
• Per-scanner breakdown (blocks, warnings, safe)
• Detailed results for sessions with issues
• Smart filtering (only shows problematic sessions)

3. CLI Entry Point (multi_agent_demo/cli.py)

NEW FILE: cli.py

Purpose: Command-line interface for batch processing.

Features:

• Argument parsing (-d directory, -s scanners, -o output, --show-safe)
• Progress bar with color-coded status
• Batch session processing with error handling
• Console and file output
• ANSI color support for terminal

Usage:

python -m multi_agent_demo.cli -d ./sessions
python -m multi_agent_demo.cli -d ./sessions -s AlignmentCheck FactsChecker
python -m multi_agent_demo.cli -d ./sessions -o report.md

4. Testing Scripts

NEW FILES:

• test_batch_cli.py - Manual test (creates sample sessions)
• test_batch_cli_automated.py - Automated CI/CD test

Purpose: Verify CLI works correctly.

5. Documentation

NEW FILES:

• BATCH_CLI_GUIDE.md - Comprehensive usage guide

UPDATED FILES:

• README.md - Added CLI sections
  • Updated Overview (3 modes: UI, CLI, Deviations)
  • Added "Batch Processing CLI" feature section
  • Added CLI usage examples
  • Added core modules documentation
• CLAUDE.md - Added CLI commands
  • Updated running commands
  • Updated module structure

---

📊 Architecture

Code Sharing Between UI and CLI

┌─────────────────────────────────────────────┐
│            User Interfaces                  │
├──────────────────┬──────────────────────────┤
│   app.py (UI)    │   cli.py (CLI)          │
└────────┬─────────┴──────────┬───────────────┘
         │                    │
         │  ┌─────────────────┴───────────┐
         │  │  core/scanner_runner.py     │
         │  │  - run_scanners_on_session()│
         └──┤  - aggregate_results()      │
            └─────────────┬───────────────┘
                          │
            ┌─────────────┴───────────────┐
            │     firewall.py             │
            │  - run_scanner_tests()      │
            └─────────────┬───────────────┘
                          │
            ┌─────────────┴───────────────┐
            │     scanners/               │
            │  - PromptGuard              │
            │  - AlignmentCheck           │
            │  - FactsChecker             │
            │  - DataDisclosureGuard      │
            └─────────────────────────────┘

Benefits:

• Single source of truth for scanner logic
• Changes automatically apply to both UI and CLI
• Consistent results across modes
• Easier maintenance

---

🎯 Use Cases

1. CI/CD Integration

# In GitHub Actions
python -m multi_agent_demo.cli -d ./test_sessions -o scan_report.md

2. Regression Testing

# Before deployment
python -m multi_agent_demo.cli -d ./baseline_sessions -o before.md
# After changes
python -m multi_agent_demo.cli -d ./baseline_sessions -o after.md
diff before.md after.md

3. Large-Scale Analysis

# Process production logs
python -m multi_agent_demo.cli -d ./prod_sessions -o analysis.md

4. Compliance Reporting

# Generate monthly report
python -m multi_agent_demo.cli -d ./sessions/2026-01 -o report.md --show-safe

---

📋 File Changes Summary

Created Files (9 new files)

multi_agent_demo/
├── core/
│   ├── __init__.py                    # NEW
│   └── scanner_runner.py              # NEW
├── reports/
│   ├── __init__.py                    # NEW
│   └── markdown_generator.py          # NEW
└── cli.py                             # NEW

# Root directory
├── test_batch_cli.py                  # NEW
├── test_batch_cli_automated.py        # NEW
├── BATCH_CLI_GUIDE.md                 # NEW
└── BATCH_CLI_IMPLEMENTATION.md        # NEW (this file)

Updated Files (2 files)

├── README.md                          # UPDATED
│   - Added CLI overview
│   - Added batch processing features
│   - Added CLI usage examples
│   - Added core modules documentation
│
└── CLAUDE.md                          # UPDATED
    - Added CLI commands
    - Updated module structure

---

🧪 Testing

Manual Test

# Create test sessions
python test_batch_cli.py

# Run CLI on test sessions
python -m multi_agent_demo.cli -d /tmp/cli_test_sessions_XXXXX

Automated Test

# Run automated test
python test_batch_cli_automated.py

# Expected: ✅ ALL CHECKS PASSED

CI/CD Integration

Add to .github/workflows/test.yml:

- name: Test CLI batch processing
  run: python test_batch_cli_automated.py

---

📖 Example Output

Console

================================================================================
🛡️  AI AGENT GUARDS - BATCH SCANNER
================================================================================

📂 Scanning directory: ./sessions
✅ Found 51 session file(s)

🔍 Enabled scanners: PromptGuard, AlignmentCheck, FactsChecker

⚙️  Processing sessions...

[████████████████████████████████████████] 100% | 51/51 | 🟢 session_051.json

✅ Processing complete!

================================================================================
📊 SUMMARY
================================================================================

Total Sessions: 51
Safe Sessions: 35 ✅
Sessions with Issues: 16 ⚠️

Total Blocks: 6 🚫
Total Warnings: 43 ⚠️
Total Safe: 198 ✅

Markdown Report

# 🛡️ AI Agent Guards - Batch Scan Report

## 📊 Overall Statistics

- **Total Sessions Scanned:** 51
- **Safe Sessions:** 35 ✅
- **Sessions with Issues:** 16 ⚠️

**Accumulated Counts:**
- 🚫 **Blocks:** 6
- ⚠️ **Warnings:** 43
- ✅ **Safe:** 198

## 🔍 Results by Scanner

### AlignmentCheck

| Metric | Count |
|--------|-------|
| 🚫 Blocks | 3 |
| ⚠️ Warnings | 0 |
| ✅ Safe | 48 |

## 📋 Detailed Results per Session

_Only showing sessions with issues._

### Session 5: `session_005.json`

**Overall Decision:** 🔴 BLOCK

**Scanner Results:**

- **AlignmentCheck:** 🔴 BLOCK
  - Total: 4 | Safe: 2 | Warnings: 0 | Blocks: 2

---

🔧 Configuration

Session JSON Format

{
  "session_id": "session_001",
  "purpose": "Banking assistant",
  "messages": [
    {"type": "user", "content": "What's my balance?"},
    {"type": "assistant", "content": "Your balance is $1,250."}
  ]
}

CLI Arguments

Argument	Required	Description	Example
-d, --directory	Yes	Directory with JSON files	-d ./sessions
-s, --scanners	No	Scanners to run (default: all)	-s AlignmentCheck FactsChecker
-o, --output	No	Output file (default: console)	-o report.md
--show-safe	No	Show safe session details	--show-safe

Available Scanners

• PromptGuard - Malicious prompts/injections (BLOCK/WARNING/SAFE)
• AlignmentCheck - Goal hijacking/drift (BLOCK/SAFE)
• FactsChecker - Contradictions/ungrounded (BLOCK/WARNING/SAFE)
• DataDisclosureGuard - PII disclosure (BLOCK/WARNING/SAFE)

---

🚀 Next Steps

For Users

1. Try the CLI:

   python test_batch_cli.py  # Create samples
   python -m multi_agent_demo.cli -d /tmp/cli_test_sessions_XXXXX
   

2. Read the guide: See BATCH_CLI_GUIDE.md

3. Integrate with CI/CD: Add to your pipeline for automated scanning

For Developers

1. Test the implementation:

   python test_batch_cli_automated.py
   

2. Extend functionality:

   • Add new scanners → automatically available in CLI
   • Modify report format → edit markdown_generator.py
   • Add custom aggregations → edit scanner_runner.py

3. Update CI/CD:

   • Add test_batch_cli_automated.py to GitHub Actions

---

✨ Key Features

1. Shared Code: UI and CLI use same scanner logic
2. Smart Filtering: Only shows sessions with issues
3. Progress Display: Real-time colored progress bar
4. Markdown Reports: Copyable format for documentation
5. Flexible Scanner Selection: Choose which scanners to run
6. Batch Processing: Process hundreds of sessions at once
7. CI/CD Ready: Exit codes and file output for automation
8. Detailed Statistics: Per-scanner and per-session breakdowns

---

📚 Documentation

• BATCH_CLI_GUIDE.md - Complete usage guide
• README.md - Main documentation
• CLAUDE.md - Development guide

---

🎉 Summary

What was requested:

• CLI batch processing for multiple session files
• Shared code between UI and CLI
• Progress display
• Markdown reports with statistics
• CI/CD testing
• README updates

What was delivered:

• ✅ Complete CLI implementation with all requested features
• ✅ Shared core logic (scanner_runner.py)
• ✅ Markdown report generator
• ✅ Progress bar with color-coded status
• ✅ Manual and automated tests
• ✅ Comprehensive documentation (README, CLAUDE.md, BATCH_CLI_GUIDE.md)
• ✅ Smart filtering (only shows sessions with issues)
• ✅ Flexible scanner selection
• ✅ File and console output

Bonus features:

• ANSI color support for better terminal UX
• Recursive directory scanning
• Error handling and graceful failures
• Example use cases and integration patterns
• Programmatic usage examples