Claude Code: Review Cain's current environment configuration and prepare a detailed c

HF_DATASET_CONFIGURATION_REPORT.md

@@ -0,0 +1,275 @@
+# HuggingFace Dataset Persistence Configuration Report
+**For Cain (tao-shen/HuggingClaw-Cain)**
+**Generated:** 2026-03-14
+
+---
+
+## Executive Summary
+
+Cain is currently **MISSING CRITICAL ENVIRONMENT VARIABLES** for data persistence. Without these, all conversations, settings, and credentials will be **LOST** when the Space restarts.
+
+**Current Status:** ⚠️ **NOT CONFIGURED** - Data persistence is DISABLED
+
+---
+
+## Required Environment Variables
+
+| Variable | Purpose | Required | Current Value | Expected Value |
+|----------|---------|----------|---------------|----------------|
+| `HF_TOKEN` | HuggingFace access token with WRITE permission | **YES** | ❌ NOT SET | `hf_***` (your token) |
+| `OPENCLAW_DATASET_REPO` | Target Dataset repository ID | AUTO* | ❌ NOT SET | `tao-shen/HuggingClaw-Cain-data` |
+| `AUTO_CREATE_DATASET` | Auto-create Dataset repo if missing | NO | ❌ NOT SET | `true` (recommended) |
+
+\* **Note:** `OPENCLAW_DATASET_REPO` is automatically derived from `SPACE_ID` if not set:
+- Space ID: `tao-shen/HuggingClaw-Cain`
+- Auto-derived Dataset: `tao-shen/HuggingClaw-Cain-data`
+
+---
+
+## Detailed Variable Descriptions
+
+### 1. `HF_TOKEN` (REQUIRED)
+
+**Purpose:** Authentication token for HuggingFace API with write permissions to create/access datasets.
+
+**How to Generate:**
+1. Visit: https://huggingface.co/settings/tokens
+2. Click "Create new token"
+3. Select type: **Write** (must have write permissions for dataset operations)
+4. Copy the generated token (starts with `hf_`)
+
+**Format:** `hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx`
+
+---
+
+### 2. `OPENCLAW_DATASET_REPO` (OPTIONAL - Auto-derived)
+
+**Purpose:** Specifies which Dataset repository to use for backing up `~/.openclaw` data.
+
+**Behavior:**
+- **If NOT set:** Automatically derived from `SPACE_ID` as `{SPACE_ID}-data`
+  - Current SPACE_ID: `tao-shen/HuggingClaw-Cain`
+  - Auto-derived: `tao-shen/HuggingClaw-Cain-data`
+- **If set manually:** Uses the specified repository ID
+
+**Format:** `username/dataset-name`
+
+**Example:** `tao-shen/HuggingClaw-Cain-data`
+
+---
+
+### 3. `AUTO_CREATE_DATASET` (OPTIONAL - Recommended)
+
+**Purpose:** Controls whether HuggingClaw automatically creates the Dataset repository on first startup.
+
+**Values:**
+- `true` - Auto-create private Dataset repo if it doesn't exist
+- `false` (default) - Require manual Dataset creation
+
+**Recommendation:** Set to `true` for automatic setup
+
+---
+
+## Step-by-Step Configuration Instructions
+
+### Step 1: Generate HF_TOKEN
+
+1. Go to https://huggingface.co/settings/tokens
+2. Click **"New token"**
+3. **Token type:** Select **"Write"** (required for dataset operations)
+4. **Token name:** Enter `HuggingClaw-Cain` (or any descriptive name)
+5. Click **"Generate token"**
+6. **Copy the token** - it will look like `hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx`
+
+⚠️ **IMPORTANT:** The token is only shown once. Save it securely.
+
+---
+
+### Step 2: Configure Space Settings
+
+1. Go to your Space: https://huggingface.co/spaces/tao-shen/HuggingClaw-Cain/settings
+2. Scroll to **"Repository Secrets"** section
+3. Add the following secrets:
+
+| Name | Value | Secret |
+|------|-------|--------|
+| `HF_TOKEN` | `hf_***` (your token from Step 1) | ✅ Yes |
+| `AUTO_CREATE_DATASET` | `true` | ❌ No |
+
+4. Click **"Save"** for each secret added
+
+**Optional Variable (if you want custom Dataset name):**
+| Name | Value | Secret |
+|------|-------|--------|
+| `OPENCLAW_DATASET_REPO` | `tao-shen/HuggingClaw-Cain-data` | ❌ No |
+
+**Note:** If you don't set `OPENCLAW_DATASET_REPO`, it will be auto-derived from SPACE_ID.
+
+---
+
+### Step 3: Restart the Space
+
+After adding environment variables:
+
+1. Go to **Settings** tab
+2. Scroll to **"Factory reset"** section
+3. Click **"Factory reset and reboot"** (this ensures clean restart with new env vars)
+4. Wait ~2-3 minutes for the Space to restart
+5. Monitor logs to verify Dataset setup
+
+---
+
+### Step 4: Verify Configuration
+
+After restart, check the logs for:
+
+```
+[SYNC] OPENCLAW_DATASET_REPO not set — auto-derived from SPACE_ID: tao-shen/HuggingClaw-Cain-data
+[SYNC] Dataset repo NOT found: tao-shen/HuggingClaw-Cain-data — creating...
+[SYNC] ✓ Dataset repo created: tao-shen/HuggingClaw-Cain-data
+[SYNC] ▶ Restoring ~/.openclaw from dataset tao-shen/HuggingClaw-Cain-data ...
+```
+
+Or check the Dataset exists at:
+https://huggingface.co/datasets/tao-shen/HuggingClaw-Cain-data
+
+---
+
+## Optional Environment Variables
+
+### `SYNC_INTERVAL`
+
+**Purpose:** Controls how often (in seconds) data is synced to the Dataset.
+
+**Default:** `60` (60 seconds)
+
+**Format:** Integer (seconds)
+
+**Example:** `SYNC_INTERVAL=120` (sync every 2 minutes)
+
+**Usage:** Only adjust if you need less frequent API calls. Lower values = safer but more API usage.
+
+---
+
+## Data Persistence Behavior
+
+### What Gets Backed Up
+
+The entire `~/.openclaw` directory is synced to the Dataset:
+
+- ✅ Conversations and chat history
+- ✅ Settings and configuration
+- ��� API credentials and keys
+- ✅ Agent plugins and configurations
+- ✅ Workspace files and memories
+
+### What Gets Excluded
+
+- ❌ `*.log` files (regenerated on startup)
+- ❌ `*.lock` files (stale after restart)
+- ❌ `*.tmp` and `*.pid` files
+- ❌ `__pycache__` directories
+
+### Sync Schedule
+
+1. **Startup:** Download latest data from Dataset
+2. **Periodic:** Upload changes every `SYNC_INTERVAL` seconds (default: 60s)
+3. **Shutdown:** Final upload before Space stops
+
+---
+
+## Troubleshooting
+
+### Error: "HF_TOKEN not set"
+
+**Solution:** Add `HF_TOKEN` as a Repository Secret in Space settings.
+
+### Error: "Dataset repo NOT found: xxx"
+
+**Solution:** Either:
+1. Set `AUTO_CREATE_DATASET=true` to auto-create
+2. Create the Dataset manually at https://huggingface.co/new-dataset
+
+### Error: "Could not derive username from HF_TOKEN"
+
+**Solution:** Verify your `HF_TOKEN` has read permissions. Check the token type is "Write".
+
+### Data not persisting after restart
+
+**Solution:** Check logs for sync errors:
+```
+[SYNC] ▶ Uploading ~/.openclaw → dataset ...
+[SYNC] ✓ Upload completed
+```
+
+If you see errors, verify:
+1. `HF_TOKEN` has Write permissions
+2. Dataset repository exists and is accessible
+3. No API quota issues
+
+---
+
+## Quick Reference: Minimal Working Configuration
+
+For automatic setup (recommended), add these as Repository Secrets:
+
+```bash
+HF_TOKEN=hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  # (Secret)
+AUTO_CREATE_DATASET=true
+```
+
+The Dataset ID will be automatically set to: `tao-shen/HuggingClaw-Cain-data`
+
+---
+
+## Current Cain Status
+
+| Component | Status |
+|-----------|--------|
+| Space Status | ✅ RUNNING |
+| Health | ✅ ALIVE (idle) |
+| HF_TOKEN | ❌ NOT SET |
+| OPENCLAW_DATASET_REPO | ❌ NOT SET (will auto-derive) |
+| AUTO_CREATE_DATASET | ❌ NOT SET (defaults to false) |
+| Data Persistence | ⚠️ DISABLED |
+
+---
+
+## Files Reference
+
+| File | Purpose |
+|------|---------|
+| `scripts/sync_hf.py` | Main persistence script |
+| `.env.example` | Environment variable reference |
+| `README.md` | Space documentation |
+
+---
+
+## Code References
+
+**sync_hf.py key lines:**
+
+```python
+# Line 61 - HF_TOKEN retrieval
+HF_TOKEN = os.environ.get("HF_TOKEN")
+
+# Line 100 - AUTO_CREATE_DATASET
+AUTO_CREATE_DATASET = os.environ.get("AUTO_CREATE_DATASET", "false").lower() in ("true", "1", "yes")
+
+# Lines 105-109 - Auto-derive Dataset from SPACE_ID
+HF_REPO_ID = os.environ.get("OPENCLAW_DATASET_REPO", "")
+if not HF_REPO_ID and SPACE_ID:
+    HF_REPO_ID = f"{SPACE_ID}-data"
+```
+
+---
+
+## Contact & Support
+
+For issues or questions:
+1. Check the main HuggingClaw repository
+2. Review sync logs in Cain's workspace: `~/.openclaw/workspace/sync.log`
+
+---
+
+**Report End**