Claude Code commited on
Commit
502fbd9
·
1 Parent(s): 5f4bb14

Claude Code: Create a scheduled job for Cain's autonomous maintenance. The job should

Browse files
.openclaw/cron/README.md ADDED
@@ -0,0 +1,67 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # OpenClaw Cron System
2
+
3
+ Cain's autonomous maintenance and health monitoring system.
4
+
5
+ ## Overview
6
+
7
+ The cron system provides self-healing capabilities for Cain by automatically monitoring health status and recovering from errors.
8
+
9
+ ## Job Configuration
10
+
11
+ Located in `.openclaw/cron/jobs.json`, following the OpenClaw cron schema:
12
+
13
+ ```json
14
+ {
15
+ "jobs": [
16
+ {
17
+ "id": "health-check",
18
+ "schedule": "*/30 * * * *",
19
+ "enabled": true,
20
+ "description": "Monitor Cain's health and auto-recover from errors",
21
+ "tool": "hf_space_status",
22
+ "on_failure": {
23
+ "tool": "hf_restart_space",
24
+ "condition": "status in ['RUNTIME_ERROR', 'BUILDING'] and duration_minutes > 10"
25
+ }
26
+ }
27
+ ]
28
+ }
29
+ ```
30
+
31
+ ## Current Jobs
32
+
33
+ ### health-check
34
+ - **Schedule**: Every 30 minutes
35
+ - **Purpose**: Monitor Cain's Hugging Face Space status
36
+ - **Tool**: `hf_space_status`
37
+ - **Auto-recovery**: Restarts space if in RUNTIME_ERROR or BUILDING state for >10 minutes
38
+ - **Logs**: `.openclaw/logs/health-check.jsonl`
39
+
40
+ ## Log Format
41
+
42
+ Health checks are logged in JSONL format:
43
+
44
+ ```json
45
+ {"timestamp":"2026-03-14T00:00:00Z","job_id":"health-check","status":"RUNNING","stage":"RUNNING","detail":"HuggingClaw is running","action":"check"}
46
+ {"timestamp":"2026-03-14T00:30:00Z","job_id":"health-check","status":"RUNTIME_ERROR","action":"recovery_triggered"}
47
+ {"timestamp":"2026-03-14T00:31:00Z","job_id":"health-check","status":"RUNNING","action":"recovery_success"}
48
+ ```
49
+
50
+ ## Adding New Jobs
51
+
52
+ 1. Edit `.openclaw/cron/jobs.json`
53
+ 2. Add a new job object to the `jobs` array
54
+ 3. Set `enabled: true` to activate
55
+ 4. Restart the cron daemon
56
+
57
+ ## Monitoring
58
+
59
+ View recent health checks:
60
+ ```bash
61
+ tail -20 .openclaw/logs/health-check.jsonl
62
+ ```
63
+
64
+ Count failures in last 24 hours:
65
+ ```bash
66
+ jq -r 'select(.action=="recovery_triggered")' .openclaw/logs/health-checks.jsonl | wc -l
67
+ ```
.openclaw/cron/crontab.example ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # OpenClaw Cron System - Crontab Reference
2
+ # ==========================================
3
+ #
4
+ # To install this cron job, run:
5
+ # .openclaw/cron/setup.sh
6
+ #
7
+ # Or manually add this line to your crontab (crontab -e):
8
+
9
+ */30 * * * * cd /tmp/claude-workspace && /tmp/claude-workspace/.openclaw/cron/executor.py >> /tmp/openclaw-cron.log 2>&1
10
+
11
+ # Cron schedule format: */30 * * * *
12
+ # - */30: Every 30 minutes
13
+ # - *: Every hour
14
+ # - *: Every day
15
+ # - *: Every month
16
+ # - *: Every day of week
17
+
18
+ # To view cron logs:
19
+ # tail -f /tmp/openclaw-cron.log
20
+
21
+ # To view health check logs:
22
+ # .openclaw/cron/logs.sh
23
+
24
+ # To uninstall, edit crontab (crontab -e) and remove the OpenClaw line
.openclaw/cron/executor.py ADDED
@@ -0,0 +1,205 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """
3
+ OpenClaw Cron Executor
4
+ Runs autonomous maintenance jobs for Cain
5
+ """
6
+ import json
7
+ import os
8
+ import sys
9
+ from datetime import datetime, timedelta
10
+ from pathlib import Path
11
+ import time
12
+ import subprocess
13
+
14
+ try:
15
+ from huggingface_hub import HfApi
16
+ except ImportError:
17
+ print("Error: huggingface_hub not installed. Run: pip install huggingface_hub")
18
+ sys.exit(1)
19
+
20
+ JOBS_FILE = Path(__file__).parent / "jobs.json"
21
+ LOGS_DIR = Path(__file__).parent.parent / "logs"
22
+
23
+ # Space configuration
24
+ SPACE_ID = "tao-shen/HuggingClaw-Cain"
25
+
26
+
27
+ def get_hf_api():
28
+ """Get HuggingFace API instance"""
29
+ token = os.environ.get("HF_TOKEN")
30
+ if not token:
31
+ # Try reading from .env file
32
+ env_file = Path(__file__).parent.parent.parent / ".env"
33
+ if env_file.exists():
34
+ with open(env_file) as f:
35
+ for line in f:
36
+ if line.startswith("HF_TOKEN="):
37
+ token = line.strip().split("=", 1)[1].strip('"\'')
38
+ break
39
+ return HfApi(token=token)
40
+
41
+
42
+ def hf_space_status(space_id: str = SPACE_ID) -> dict:
43
+ """Check HuggingFace Space status"""
44
+ api = get_hf_api()
45
+ try:
46
+ info = api.space_info(repo_id=space_id)
47
+ # Get runtime info - the structure may vary
48
+ runtime = getattr(info, 'runtime', None)
49
+ if runtime:
50
+ stage = getattr(runtime, 'stage', 'UNKNOWN')
51
+ status = getattr(runtime, 'status', 'UNKNOWN')
52
+ detail = f"Stage: {stage}, Status: {status}"
53
+ else:
54
+ stage = getattr(info, 'stage', 'UNKNOWN')
55
+ status = 'UNKNOWN'
56
+ detail = str(info)
57
+
58
+ return {
59
+ "stage": stage,
60
+ "status": status,
61
+ "detail": detail,
62
+ "success": True,
63
+ "space_info": str(info)
64
+ }
65
+ except Exception as e:
66
+ return {
67
+ "stage": "UNKNOWN",
68
+ "status": "ERROR",
69
+ "detail": str(e),
70
+ "success": False
71
+ }
72
+
73
+
74
+ def hf_restart_space(space_id: str = SPACE_ID) -> dict:
75
+ """Restart a HuggingFace Space"""
76
+ api = get_hf_api()
77
+ try:
78
+ api.restart_space(repo_id=space_id)
79
+ return {
80
+ "success": True,
81
+ "detail": f"Restart initiated for {space_id}"
82
+ }
83
+ except Exception as e:
84
+ return {
85
+ "success": False,
86
+ "detail": str(e)
87
+ }
88
+
89
+
90
+ def log_event(job_id: str, event_data: dict):
91
+ """Log an event to the appropriate log file"""
92
+ log_file = LOGS_DIR / f"{job_id}.jsonl"
93
+ log_file.parent.mkdir(parents=True, exist_ok=True)
94
+
95
+ event = {
96
+ "timestamp": datetime.utcnow().isoformat() + "Z",
97
+ "job_id": job_id,
98
+ **event_data
99
+ }
100
+
101
+ with open(log_file, "a") as f:
102
+ f.write(json.dumps(event) + "\n")
103
+
104
+
105
+ def load_jobs() -> list:
106
+ """Load jobs from configuration file"""
107
+ with open(JOBS_FILE) as f:
108
+ config = json.load(f)
109
+ return config.get("jobs", [])
110
+
111
+
112
+ def check_condition(status: str, condition: str) -> bool:
113
+ """Evaluate a failure condition"""
114
+ if "status in [" in condition:
115
+ allowed_statuses = condition.split("status in [")[1].split("]")[0]
116
+ allowed_statuses = [s.strip().strip("'\"") for s in allowed_statuses.split(",")]
117
+ return status in allowed_statuses
118
+ return False
119
+
120
+
121
+ def run_job(job: dict):
122
+ """Execute a single job"""
123
+ job_id = job["id"]
124
+
125
+ if not job.get("enabled", True):
126
+ return
127
+
128
+ print(f"[{datetime.utcnow().isoformat()}] Running job: {job_id}")
129
+
130
+ try:
131
+ # Get health status
132
+ result = hf_space_status(space_id="tao-shen/HuggingClaw-Cain")
133
+
134
+ status = result.get("stage", "UNKNOWN")
135
+ detail = result.get("detail", "")
136
+
137
+ log_event(job_id, {
138
+ "status": status,
139
+ "stage": result.get("stage"),
140
+ "detail": detail,
141
+ "action": "check"
142
+ })
143
+
144
+ print(f" Status: {status} - {detail}")
145
+
146
+ # Check failure condition
147
+ if "on_failure" in job:
148
+ failure_config = job["on_failure"]
149
+ condition = failure_config.get("condition", "")
150
+
151
+ if check_condition(status, condition):
152
+ print(f" Failure condition met: {condition}")
153
+ log_event(job_id, {
154
+ "status": status,
155
+ "action": "recovery_triggered",
156
+ "condition": condition
157
+ })
158
+
159
+ # Execute recovery action
160
+ if failure_config.get("tool") == "hf_restart_space":
161
+ print(" Attempting recovery via hf_restart_space...")
162
+ recovery_result = hf_restart_space(space_id="tao-shen/HuggingClaw-Cain")
163
+
164
+ log_event(job_id, {
165
+ "action": "recovery_attempt",
166
+ "success": recovery_result.get("success", False),
167
+ "detail": recovery_result.get("detail", "")
168
+ })
169
+
170
+ if recovery_result.get("success"):
171
+ print(" Recovery initiated successfully")
172
+ else:
173
+ print(f" Recovery failed: {recovery_result.get('detail')}")
174
+
175
+ except Exception as e:
176
+ print(f" Error: {e}")
177
+ log_event(job_id, {
178
+ "action": "error",
179
+ "error": str(e)
180
+ })
181
+
182
+
183
+ def main():
184
+ """Main entry point"""
185
+ if len(sys.argv) > 1 and sys.argv[1] == "--list":
186
+ jobs = load_jobs()
187
+ print("OpenClaw Cron Jobs:")
188
+ print("-" * 50)
189
+ for job in jobs:
190
+ status = "enabled" if job.get("enabled", True) else "disabled"
191
+ print(f" {job['id']}: {job.get('schedule', 'N/A')} [{status}]")
192
+ print(f" {job.get('description', '')}")
193
+ return
194
+
195
+ jobs = load_jobs()
196
+ print(f"OpenClaw Cron Executor - {len(jobs)} jobs loaded")
197
+
198
+ for job in jobs:
199
+ run_job(job)
200
+
201
+ print("Done.")
202
+
203
+
204
+ if __name__ == "__main__":
205
+ main()
.openclaw/cron/jobs.json ADDED
@@ -0,0 +1,31 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "jobs": [
3
+ {
4
+ "id": "health-check",
5
+ "schedule": "*/30 * * * *",
6
+ "enabled": true,
7
+ "description": "Monitor Cain's health and auto-recover from errors",
8
+ "tool": "hf_space_status",
9
+ "params": {
10
+ "space_id": "tao-shen/HuggingClaw-Cain"
11
+ },
12
+ "on_failure": {
13
+ "tool": "hf_restart_space",
14
+ "condition": "status in ['RUNTIME_ERROR', 'BUILDING'] and duration_minutes > 10",
15
+ "params": {
16
+ "space_id": "tao-shen/HuggingClaw-Cain"
17
+ }
18
+ },
19
+ "logging": {
20
+ "file": ".openclaw/logs/health-check.jsonl",
21
+ "format": "jsonl",
22
+ "retention_days": 30
23
+ }
24
+ }
25
+ ],
26
+ "metadata": {
27
+ "version": "1.0",
28
+ "created_at": "2026-03-14T00:00:00Z",
29
+ "description": "OpenClaw Cron Job Configuration for Cain Autonomous Maintenance"
30
+ }
31
+ }
.openclaw/cron/logs.sh ADDED
@@ -0,0 +1,82 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/bin/bash
2
+ # View OpenClaw Cron Logs
3
+ # Utility to view health check history
4
+
5
+ LOG_FILE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/../logs/health-check.jsonl"
6
+
7
+ usage() {
8
+ echo "Usage: $0 [OPTIONS]"
9
+ echo ""
10
+ echo "Options:"
11
+ echo " -n, --lines N Show last N entries (default: 20)"
12
+ echo " -f, --follow Follow log output (like tail -f)"
13
+ echo " -s, --summary Show summary statistics"
14
+ echo " -h, --help Show this help message"
15
+ exit 1
16
+ }
17
+
18
+ lines=20
19
+ follow=false
20
+ summary=false
21
+
22
+ while [[ $# -gt 0 ]]; do
23
+ case $1 in
24
+ -n|--lines)
25
+ lines="$2"
26
+ shift 2
27
+ ;;
28
+ -f|--follow)
29
+ follow=true
30
+ shift
31
+ ;;
32
+ -s|--summary)
33
+ summary=true
34
+ shift
35
+ ;;
36
+ -h|--help)
37
+ usage
38
+ ;;
39
+ *)
40
+ echo "Unknown option: $1"
41
+ usage
42
+ ;;
43
+ esac
44
+ done
45
+
46
+ if [ "$summary" = true ]; then
47
+ echo "OpenClaw Health Check Summary"
48
+ echo "=============================="
49
+ echo ""
50
+
51
+ if [ ! -f "$LOG_FILE" ] || [ ! -s "$LOG_FILE" ]; then
52
+ echo "No logs found."
53
+ exit 0
54
+ fi
55
+
56
+ total=$(wc -l < "$LOG_FILE")
57
+ echo "Total entries: $total"
58
+ echo ""
59
+
60
+ echo "Action breakdown:"
61
+ jq -r '.action' "$LOG_FILE" 2>/dev/null | sort | uniq -c | sort -rn || echo " (Unable to parse)"
62
+ echo ""
63
+
64
+ echo "Status breakdown:"
65
+ jq -r '.status // "N/A"' "$LOG_FILE" 2>/dev/null | sort | uniq -c | sort -rn || echo " (Unable to parse)"
66
+ echo ""
67
+
68
+ echo "Recent failures:"
69
+ jq -r 'select(.action == "recovery_triggered" or .action == "error") | "\(.timestamp) - \(.status) - \(.action)"' "$LOG_FILE" 2>/dev/null | tail -5 || echo " None"
70
+
71
+ elif [ "$follow" = true ]; then
72
+ tail -f "$LOG_FILE" | while read -r line; do
73
+ echo "$line" | jq -r '[.timestamp, .action, .status // "N/A", .detail // ""] | @tsv' 2>/dev/null || echo "$line"
74
+ done
75
+ else
76
+ echo "OpenClaw Health Checks (last $lines entries)"
77
+ echo "============================================"
78
+ echo ""
79
+ tail -n "$lines" "$LOG_FILE" | while read -r line; do
80
+ echo "$line" | jq -r '[.timestamp, .action, .status // "N/A", .detail // ""] | @tsv' 2>/dev/null || echo "$line"
81
+ done
82
+ fi
.openclaw/cron/setup.sh ADDED
@@ -0,0 +1,51 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/bin/bash
2
+ # OpenClaw Cron Setup Script
3
+ # Installs the health check cron job for Cain
4
+
5
+ set -e
6
+
7
+ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
8
+ PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
9
+ EXECUTOR="$SCRIPT_DIR/executor.py"
10
+
11
+ echo "OpenClaw Cron Setup for Cain"
12
+ echo "=============================="
13
+
14
+ # Check if executor exists
15
+ if [ ! -f "$EXECUTOR" ]; then
16
+ echo "Error: Executor not found at $EXECUTOR"
17
+ exit 1
18
+ fi
19
+
20
+ # Make sure executor is executable
21
+ chmod +x "$EXECUTOR"
22
+
23
+ # Create crontab entry
24
+ CRON_ENTRY="*/30 * * * * cd $PROJECT_ROOT && $EXECUTOR >> /tmp/openclaw-cron.log 2>&1"
25
+
26
+ echo "Cron entry to be installed:"
27
+ echo "$CRON_ENTRY"
28
+ echo ""
29
+
30
+ # Check if entry already exists
31
+ if crontab -l 2>/dev/null | grep -q "openclaw-cron"; then
32
+ echo "Cron entry already exists. Updating..."
33
+ # Remove old entry and add new one
34
+ crontab -l 2>/dev/null | grep -v "openclaw-cron" | crontab -
35
+ fi
36
+
37
+ # Install new cron entry
38
+ (crontab -l 2>/dev/null; echo "$CRON_ENTRY") | crontab -
39
+
40
+ echo "Cron job installed successfully!"
41
+ echo ""
42
+ echo "Cain will now perform health checks every 30 minutes."
43
+ echo "Logs will be saved to:"
44
+ echo " - $PROJECT_ROOT/.openclaw/logs/health-checks.jsonl (job execution logs)"
45
+ echo " - /tmp/openclaw-cron.log (cron daemon logs)"
46
+ echo ""
47
+ echo "To view recent health checks:"
48
+ echo " tail -20 $PROJECT_ROOT/.openclaw/logs/health-checks.jsonl"
49
+ echo ""
50
+ echo "To manually run a health check:"
51
+ echo " $EXECUTOR"
.openclaw/logs/.gitignore ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ # OpenClaw Cron Logs
2
+ *.jsonl
3
+ *.log