BATCH_CLI_GUIDE.md

CLI Batch Processing Guide

Overview

The AI Agent Guards platform now supports batch processing mode via a command-line interface (CLI). This allows you to scan multiple session files at once, get aggregated statistics, and generate markdown reports.

Key Benefits:

• Process hundreds of sessions in one command
• CI/CD integration for automated testing
• Markdown reports for documentation
• Shared code with UI ensures consistency
• Smart filtering: only shows sessions with issues

---

Quick Start

1. Basic Usage

Scan all JSON files in a directory:

python -m multi_agent_demo.cli -d ./sessions

2. Select Specific Scanners

python -m multi_agent_demo.cli -d ./sessions -s AlignmentCheck FactsChecker

3. Save Report to File

python -m multi_agent_demo.cli -d ./sessions -o report.md

4. Include Safe Sessions in Report

By default, only sessions with issues are detailed. To show all:

python -m multi_agent_demo.cli -d ./sessions --show-safe

---

Available Scanners

Scanner	What It Detects	Decisions
PromptGuard	Malicious prompts and injections	BLOCK, WARNING, SAFE
AlignmentCheck	Goal hijacking and behavioral drift	BLOCK, SAFE (no warnings)
FactsChecker	Contradictions and ungrounded claims	BLOCK, WARNING, SAFE
DataDisclosureGuard	PII disclosure issues	BLOCK, WARNING, SAFE

Note: If you see "Total Warnings: 0", it's normal when running only AlignmentCheck, which doesn't produce warnings (only BLOCK or SAFE).

---

Session JSON Format

The CLI supports two formats:

Format 1: Langfuse Export Format (OpenOps)

{
  "scenario_name": "environment_prod_0450c00c",
  "agent_purpose": "You are the OpenOps Agent, an AI assistant...",
  "messages": [
    {"type": "user", "content": "create a workflow that monitors price..."},
    {"type": "assistant", "content": "I'll help you create that workflow..."}
  ],
  "exported_at": "2026-01-26T19:06:23.108931+00:00",
  "format_version": "1.0"
}

Required Fields:

• agent_purpose - Agent's purpose/role description
• messages - Array of message objects with type and content

Optional Fields:

• scenario_name - Scenario/session identifier
• exported_at - Export timestamp
• format_version - Format version

Format 2: Simple Format

{
  "session_id": "session_001",
  "purpose": "Banking assistant that helps users check balances",
  "messages": [
    {"type": "user", "content": "What's my account balance?"},
    {"type": "assistant", "content": "Your current balance is $1,250.00"}
  ]
}

Required Fields:

• purpose - Agent's purpose/role description
• messages - Array of message objects with type and content

Optional Fields:

• session_id - Session identifier
• agent_name - Agent name
• agent_role - Agent role

---

Note: The CLI automatically detects which format you're using. It checks for agent_purpose first (Langfuse format), then falls back to purpose (simple format).

---

Output Example

Console Output

================================================================================
🛡️  AI AGENT GUARDS - BATCH SCANNER
================================================================================

📂 Scanning directory: ./sessions
✅ Found 51 session file(s)

🔍 Enabled scanners: PromptGuard, AlignmentCheck, FactsChecker, DataDisclosureGuard

⚙️  Processing sessions...

[████████████████████████████████████████] 100% | 51/51 | 🟢 session_051.json

✅ Processing complete!

📊 Aggregating results...
📝 Generating report...

================================================================================
📊 SUMMARY
================================================================================

Total Sessions: 51
Safe Sessions: 35 ✅
Sessions with Issues: 16 ⚠️

Total Blocks: 6 🚫
Total Warnings: 43 ⚠️
Total Safe: 198 ✅

================================================================================

Markdown Report Structure

# 🛡️ AI Agent Guards - Batch Scan Report

---

## 📊 Overall Statistics

- **Total Sessions Scanned:** 51
- **Safe Sessions:** 35 ✅
- **Sessions with Issues:** 16 ⚠️

**Accumulated Counts:**
- 🚫 **Blocks:** 6
- ⚠️ **Warnings:** 43
- ✅ **Safe:** 198

---

## 🔍 Results by Scanner

### AlignmentCheck

| Metric | Count |
|--------|-------|
| 🚫 Blocks | 3 |
| ⚠️ Warnings | 0 |
| ✅ Safe | 48 |

### FactsChecker

| Metric | Count |
|--------|-------|
| 🚫 Blocks | 3 |
| ⚠️ Warnings | 43 |
| ✅ Safe | 5 |

---

## 📋 Detailed Results per Session

_Note: Only showing sessions with issues. Safe sessions are omitted for brevity._

### Session 5: `session_005_goal_hijacking.json`

**Overall Decision:** 🔴 BLOCK

**Scanner Results:**

- **AlignmentCheck:** 🔴 BLOCK
  - Total: 4 | Safe: 2 | Warnings: 0 | Blocks: 2
  - _Reason:_ Agent redirected conversation from stated purpose...

- **FactsChecker:** 🟡 WARNING
  - Total: 4 | Safe: 2 | Warnings: 2 | Blocks: 0
  - _Reason:_ Detected ungrounded claims in messages 3 and 4...

---

NEW: Google Sheets Integration 🎉

The report now includes two formats for easy data analysis:

1. Sessions Summary Table - Markdown table with all sessions and per-scanner results

   • Format: DECISION (total: safe/warning/block)
   • Example: SAFE (3: 3/0/0) = 3 messages, all safe
   • Includes "Overall" column showing worst decision

2. Copy-Paste Format (CSV) - Comma-separated values for direct paste into Google Sheets

   • Format: DECISION (safe/warning/block)
   • Shorter format, perfect for spreadsheet analysis
   • Just copy and paste - columns align automatically!

For complete guide on using the reports in Google Sheets, see: 📖 BATCH_CLI_REPORT_FORMAT.md

Quick example:

Session	AlignmentCheck	PromptGuard	FactsChecker	Overall
session1.json	SAFE (3/0/0)	SAFE (2/0/0)	WARNING (3/2/0)	WARNING
session2.json	BLOCK (1/0/2)	SAFE (2/0/0)	SAFE (5/0/0)	BLOCK

Copy → Paste into Google Sheets → Done! ✨

---

Use Cases

1. CI/CD Integration

Add to your GitHub Actions workflow:

- name: Scan agent sessions
  run: |
    python -m multi_agent_demo.cli -d ./test_sessions -o scan_report.md

- name: Upload report
  uses: actions/upload-artifact@v3
  with:
    name: security-scan-report
    path: scan_report.md

2. Regression Testing

Before deploying changes:

# Scan baseline sessions
python -m multi_agent_demo.cli -d ./baseline_sessions -o baseline.md

# Make changes to agent

# Scan again and compare
python -m multi_agent_demo.cli -d ./baseline_sessions -o updated.md
diff baseline.md updated.md

3. Large-Scale Analysis

Process production logs:

# Export sessions from production to JSON
python export_sessions.py --output ./prod_sessions

# Scan all sessions
python -m multi_agent_demo.cli -d ./prod_sessions -o prod_analysis.md

# Review report for patterns
cat prod_analysis.md | grep "BLOCK"

4. Compliance Reporting

Generate reports for audits:

# Scan last month's sessions
python -m multi_agent_demo.cli \
  -d ./sessions/2026-01 \
  -o compliance_report_jan_2026.md \
  --show-safe

# Include in compliance documentation

---

Testing the CLI

Manual Test

Create test sessions:

python test_batch_cli.py

This creates a temp directory with sample sessions and shows you the command to run.

Automated Test

Run the automated test:

python test_batch_cli_automated.py

This creates sessions, runs the CLI, and validates the output automatically.

---

Architecture

The CLI shares code with the UI to ensure consistency:

multi_agent_demo/
├── cli.py                      # CLI entry point
├── app.py                      # UI entry point
├── core/                       # Shared logic
│   └── scanner_runner.py      # Scanner execution (used by both CLI and UI)
├── reports/                    # CLI-specific
│   └── markdown_generator.py  # Markdown report generation
├── firewall.py                # Scanner orchestration (shared)
└── scanners/                   # Scanner implementations (shared)

Benefits of shared code:

• Changes to scanner logic automatically apply to both CLI and UI
• Same validation rules everywhere
• Consistent results across modes
• Single source of truth

---

Advanced Options

Custom Session Format

If your sessions have a different structure, create a wrapper:

from multi_agent_demo.core import run_scanners_on_session

# Load your custom format
custom_session = load_my_session("session.json")

# Convert to expected format
session_data = {
    "purpose": custom_session["agent_purpose"],
    "messages": [
        {"type": msg["role"], "content": msg["text"]}
        for msg in custom_session["conversation"]
    ]
}

# Run scanners
result = run_scanners_on_session(
    session_data=session_data,
    enabled_scanners=["AlignmentCheck", "FactsChecker"]
)

Programmatic Usage

Use the CLI logic in your own scripts:

from multi_agent_demo.core import run_scanners_on_session, aggregate_results
from multi_agent_demo.reports import generate_markdown_report

# Load sessions
sessions = [load_json(f) for f in session_files]

# Run scanners
results = [
    run_scanners_on_session(session, ["AlignmentCheck"])
    for session in sessions
]

# Aggregate
stats = aggregate_results(results)

# Generate report
report = generate_markdown_report(results, session_files, stats)
print(report)

---

Troubleshooting

Error: "No JSON files found"

Check:

• Directory path is correct
• JSON files have .json extension
• Files are not in subdirectories (CLI scans recursively with **/*.json)

Error: "Module not found"

Run from project root:

cd /path/to/mutli-agent-demo
python -m multi_agent_demo.cli -d ./sessions

Error: "TOGETHER_API_KEY not found"

Set environment variable:

export TOGETHER_API_KEY=your_key_here
python -m multi_agent_demo.cli -d ./sessions

Or create .env file:

TOGETHER_API_KEY=your_key_here
OPENAI_API_KEY=your_key_here

Slow Performance

Use fewer scanners:

# Fast: Only AlignmentCheck
python -m multi_agent_demo.cli -d ./sessions -s AlignmentCheck

# Slower: All scanners
python -m multi_agent_demo.cli -d ./sessions

---

FAQ

Q: Can I use the CLI without the UI? A: Yes! The CLI is standalone. Just ensure dependencies are installed.

Q: Does the CLI support the same scanners as the UI? A: Yes, exactly the same scanners with the same logic.

Q: Can I integrate with other CI/CD tools? A: Yes! The CLI is a standard Python script with exit codes:

• 0 = success
• 1 = error

Q: How do I get only the statistics without the full report? A: Redirect stderr to see just the summary:

python -m multi_agent_demo.cli -d ./sessions 2>&1 | tail -20

Q: Can I scan a single file? A: Yes, put it in a directory:

mkdir temp_scan
cp session.json temp_scan/
python -m multi_agent_demo.cli -d temp_scan

---

Next Steps

1. Try it: Run python test_batch_cli.py to create sample sessions
2. Integrate: Add to your CI/CD pipeline
3. Customize: Adjust report format in markdown_generator.py
4. Scale: Process production logs for analysis

For more details, see:

• README.md - Full documentation
• CLAUDE.md - Development guide
• INSTALL.md - Installation guide