Update README.md

README.md

@@ -8,7 +8,7 @@ dependencies: gcc, libcurl, bc, cJSON
 persistence: shadowclaw.bin
 ---
 
-# Shadowclaw v1.3 (Testing)
+# Shadowclaw v3.4 (Testing)
 
 <div style="max-width: 100%; overflow-x: auto; background: #f6f8fa; padding: 4px; border-radius: 4px;">
   <pre style="font-family: 'Courier New', monospace; font-size: clamp(4px, 2vw, 10px); line-height: 1.2; margin: 0;">
@@ -35,246 +35,179 @@ Low-power edge nodes: self-hosted persistent AI, no cloud.
 
 ---
 
-## Repository Structure
-
-```
-shadowclaw/
-├── shadowclaw.c      # main program: arena, tools, LLM glue, event loop
-├── Makefile          # simple build file
-├── cJSON.c           # lightweigt JSON parser (from cJSON library)
-└── cJSON.h           # cJSON header
-```
-
----
-
 ## Features
 
-- **Growable shadow arena** – all objects stored in one contiguous block with hidden headers.
-- **Persistent memory** – saves and loads full arena to/from `shadowclaw.bin`.
-- **Tool use** – shell, file I/O, HTTP GET, math (via `bc`).
-- **Ollama integration** – talk to any model running locally.
-- **Blob storage** – system prompts, user messages, assistant replies, tool calls, results.
-- **Tiny footprint** – stripped binary ≈ 100–200 KiB.
+- **🧠 Local LLM integration** – works with any Ollama model (default: `tinyllama:1.1b`).
+- **🔧 Built‑in tools** – `file_read`, `file_write`, `http_get`, `math`, `list_dir`, `shell` (disabled by default), and more.
+- **⏰ Cron jobs** – schedule recurring tasks using `@every N[s/m/h]`, `@hourly`, `@daily`, `@weekly`.
+- **🌐 Webhooks** – trigger HTTP POST calls on tool execution or cron events.
+- **🎓 Dynamic skills** – create reusable multi‑step workflows without recompiling.
+- **💾 Core memory** – persistent key‑value storage (JSON) that survives across sessions.
+- **📜 Soul file** – Markdown export of all memories (conversation, skills, crons, webhooks, core memory).
+- **🎨 Colored TUI** – optional GNU readline support for line editing and history.
+- **⚡ Thread‑safe** – cron jobs run in a separate thread, tool calls are queued.
+- **🛡️ Security** – path sandboxing, domain allowlist, shell opt‑in, dry‑run mode.
 
 ---
 
-## Requirements
-
-- Linux (developed on Debian, should work on most Unix‑likes)
-- `gcc`, `make`, `libcurl4-openssl-dev`, `bc`
-- [Ollama](https://ollama.com/) running locally (default `http://localhost:11434`) with a model pulled (e.g. `llama3.2`)
+## 📦 Requirements
 
-Install build dependencies on Debian/Ubuntu:
-
-```bash
-sudo apt update
-sudo apt install build-essential libcurl4-openssl-dev bc
-```
+- **Linux / macOS / WSL** (tested on Ubuntu 22.04, Kali)
+- **Ollama** (running locally) – optional, the agent can run in `--no-llm` mode
+- **Dependencies**:
+  - `libcurl` (HTTP requests)
+  - `libpthread` (threading)
+  - `libreadline` (optional, for TUI enhancements)
+  - `gcc` or `clang` with C99 support
 
 ---
 
-## Define your model:
-
-- **Change Ollama endpoint/model**: edit `ollama_endpoint` and `ollama_model` in `shadowclaw.c`.
-
-Find this line (507) in the shadowclaw.c file:
-
-  ```bash
-// --------------------------------------------------------------------
-//  Main (with slash commands from v1.2.2)
-// --------------------------------------------------------------------
-int main(int argc, char **argv) {
-    const char *state_file = "shadowclaw.bin";
-    const char *ollama_endpoint = "http://localhost:11434";
-    const char *ollama_model = "qwen2.5:0.5b";  // change as needed
-  ```
-
-Adjust the model endpoint in "ollama_model = "qwen2.5:0.5b";" to meet your model.
+## Installation + Launch
 
-- **Add new tools**: extend the `tools` array in `shadowclaw.c` with a name and a function that takes a `const char*` argument and returns a `char*` (must be `malloc`ed, the caller will `free` it).
+*Put all files into a single folder on your system*
 
----
-
-## How It Works
-
-### Shadow Arena
-
-All dynamic data lives in a single `realloc`‑ed block.  
-A `ShadowHeader` is stored **immediately before** the user‑visible pointer.  
-This header holds capacity, used length, a magic number, and a dirty flag.
-
-```
-+-------------------+---------------------+
-|  ShadowHeader     |  payload (blobs)    |
-+-------------------+---------------------+
-^
-`- pointer returned to user
-```
-
-### Blob Format
-
-Each item (system prompt, user message, tool result, etc.) is stored as:
-
-```
-+------------+------+------+-----------------+
-| BlobHeader | kind |  id  | payload (bytes) |
-+------------+------+------+-----------------+
+```bash
+cd ~/shadowclaw (The folder you put the files in)
+make clean && make
+./start.sh
 ```
 
-- `BlobHeader` contains the payload size, kind, and a 64‑bit ID.
-- The payload is arbitrary data (null‑terminated strings for most kinds).
-
-### Persistence
+## Setup your local Ollama model
 
--The whole arena (header + payload) is written to `shadowclaw.bin` with `fwrite`. 
+*Shadowclaw is set to use qwen2.5:0.5b as a default, to change this:*
 
--On startup, if the file exists and has a valid magic number, it is loaded back.
-
--All conversations and tool results are automatically saved to `shadowclaw.bin` and reloaded on restart.
-
----
-
-# Updated 3/3/2026: Shadowclaw v1.3
-
-### Built‑in Slash Commands
-
-Type any of these commands directly at the `>` prompt – they are handled without invoking the LLM.
-
-| Command   | Description |
-|-----------|-------------|
-| `/help`   | Show this help message. |
-| `/tools`  | List all available tools (shell, file I/O, HTTP, math, `list_dir`). |
-| `/state`  | Display current arena memory statistics (capacity, used bytes, dirty flag). |
-| `/clear`  | Clear the conversation history while retaining the system prompt. |
-| `/chat`   | Remind you that you are already in chat mode (the default behaviour). |
-| `/exit`   | Quit Shadowclaw. |
-
-## Build
+Find line 633 in the shadowclaw.c file:
 
 ```bash
-cd /Home/User/Shadowclaw        # The folder you have the files located.
+static const char *ollama_endpoint = "http://localhost:11434";
+static const char *ollama_model = "qwen2.5:0.5b"; (Change this to desired model)
+static long llm_connect_timeout = 15;
 ```
 
-```bash
-make clean && make
-```
-
-## Run
+Also line 16 in the start.sh file:
 
 ```bash
-./shadowclaw
+OLLAMA_ENDPOINT="${OLLAMA_ENDPOINT:-http://localhost:11434}"
+OLLAMA_MODEL="${OLLAMA_MODEL:-qwen2.5:0.5b}" (Change this to desired model)
 ```
 
-Use the slash commands above to get started – even without Ollama running, you can explore the built‑in features.
+---
 
-When your agent is running you should be able to use /help and see (example):
+### First start
 
-```bash
-┌──(kali㉿user)-[~/shadowclaw]
-└─$ ./shadowclaw
-ShadowClaw ready. Type your message (Ctrl-D to exit)
-/help
-Shadowclaw commands:
-  /help       Show this help
-  /tools      List available tools
-  /state      Show arena memory stats
-  /clear      Clear conversation history (keeps system prompt)
-  /chat       Remind you that chat mode is active
-  /exit       Exit Shadowclaw
-```
+- The agent creates `shadowclaw.bin` (binary state) and a folder `shadowclaw_data/` containing `shadowsoul.md`.
+- If Ollama is not reachable, it automatically falls back to `--no-llm` mode.
+- A default heartbeat cron job (`@every 120s`) is added automatically to keep the soul file updated.
 
-## Notes
+---
 
--If Ollama is not running, you’ll see LLM call failed.
+## Interactive Commands
 
--Tool arguments can be a JSON array – they will be joined with spaces (useful if your model outputs "args":["arg1","arg2"]).
+Shadowclaw understands both natural language (sent to the LLM) and slash commands.
 
--All conversations and tool results are saved in shadowclaw.bin and reloaded on restart.
+| Command | Description |
+|---------|-------------|
+| `/help` | Show help and list all commands. |
+| `/tools` | List available built‑in tools. |
+| `/state` | Show arena memory usage and soul file stats. |
+| `/clear` | Erase conversation history (keeps system prompt and core memory). |
+| `/exit` | Quit the agent. |
+| `/loop <schedule> <tool> [args]` | Schedule a recurring task. Examples:<br>`/loop 30m http_get https://example.com`<br>`/loop daily math "1+1"` |
+| `/crons` | List all scheduled cron jobs. |
+| `/webhooks` | Show registered webhooks. |
+| `/skills` | List dynamic skills. |
+| `/compact` | Manually compact the arena (remove deleted blobs). |
+| `/soul` | Display information about `shadowsoul.md`. |
 
 ---
 
-## How Tool Arguments Work
-
-Shadowclaw processes tool calls by parsing JSON, where the args value is passed as a string to the function, executing it, and appending the result as a kind 5 blob for the next prompt. It supports a fallback where the model outputs args as an array (e.g., [arg1, arg2]), allowing for flexible input handling when standard JSON encoding fails. 
-
--Tool Call Flow: The agent parses the JSON block, executes the tool, and appends the result.
-
--Result Visibility: The tool output is visible to the model in the next prompt.
+## 🛠️ Tools
+
+Tools are invoked by the LLM during the “plan” phase. Each tool is described in the LLM prompt with its parameters and an example.
+
+| Tool | Description | Example args |
+|------|-------------|---------------|
+| `file_read` | Read a file (max 10 MB, path must be inside CWD). | `notes.txt` |
+| `file_write` | Write content to a file (overwrites). | `output.txt Hello world` |
+| `http_get` | HTTP GET to an allowed domain (see `allowed_domains` in source). | `https://example.com/data` |
+| `math` | Evaluate arithmetic expression. | `(2+3)*4` |
+| `list_dir` | List directory contents. | `.` or `/home/user` |
+| `webhook_add` | Register a webhook (JSON: `{"url":"...","event":"..."}`). | `{"url":"http://...","event":"tool:http_get"}` |
+| `cron_add` | Add a cron job (JSON: `{"schedule":"...","tool":"...","args":"..."}`). | `{"schedule":"@every 30m","tool":"math","args":"1+1"}` |
+| `cron_list` | List all cron jobs. | (none) |
+| `cron_remove` | Remove cron jobs containing a substring in their JSON representation. | `@every` |
+| `skill_add` | Create a dynamic skill (JSON with `name`, `desc`, `steps` array, optionally `interpreter_command`). | See below. |
+| `skill_run` | Run a skill by name. | `weather London` |
+| `list_skills` | List all available skills. | (none) |
+| `update_core_memory` | Merge JSON object into core memory. | `{"user_name":"Alice","preferences":{"theme":"dark"}}` |
+| `recall` | Search conversation history for a keyword. | `project` |
+| `heartbeat` | Internal (used by cron). | (none) |
+
+> **Security:** The `shell` tool is compiled out by default. To enable it, add `-DENABLE_SHELL_TOOL` to `CFLAGS` and understand the risks.
 
--Fallback Mechanism: If JSON parsing fails, Shadowclaw interprets args as an array.
+---
 
--Alternative Tooling: In some scenarios, tool calls can be handled as a single string to avoid parsing errors. 
+## Dynamic Skills
 
-This system helps in handling malformed JSON, which occasionally occurs with certain LLM providers, ensuring the tool call succeeds.
+Skills are sequences of tool calls stored in the arena as `BLOB_KIND_SKILL`. Example creation:
 
 ```json
-{"tool":"write_file","args":["notes.txt","Hello world"]}
+{
+  "name": "weather",
+  "desc": "Get weather for a city",
+  "steps": [
+    {"tool": "http_get", "args": "https://wttr.in/{0}"},
+    {"tool": "file_write", "args": "/tmp/weather.txt {result}"}
+  ]
+}
 ```
 
-Shadowclaw automatically **joins the array elements with spaces**, so the tool receives a single string `"notes.txt Hello world"`. This makes the agent robust to different model behaviours to ensure:
-
-- **Simplicity** – most tools only need a single string argument (a command, a filename, a URL).  
-- **Flexibility** – if a tool needs multiple pieces of information (like `write_file` needing a filename **and** content), we use a simple delimiter (newline). The LLM can learn this pattern from the system prompt.  
-- **Lightweight Design** – no complex argument schemas, no extra JSON nesting. The entire tool call is tiny.
-- **Args are always a single string** to keep things simple.  
-- **Multiple values are encoded with delimiters** (like newline) – the tool and LLM agree on the format.  
-- **Shadowclaw’s niche** is minimalism: a single binary with persistent memory, running anywhere, with just enough tools to be genuinely useful.  
+Placeholders supported:
+- `{args}` – the whole argument string passed to `skill_run`
+- `{0}`, `{1}`, … – positional arguments (split by spaces)
+- `{result}` – output of the previous step
 
-You can easily add new tools by editing the `tools` array – each new function opens up more automation possibilities for your unique environment.
+Skills can also delegate to an external interpreter command (e.g., a Python script) via the optional `interpreter_command` field.
 
 ---
 
-## Niche Use Cases
-
-Shadowclaw is designed for **minimal, self‑contained AI agents** running on resource‑constrained or air‑gapped systems.  
-Its toolset is deliberately small but powerful enough to automate many tasks without spawning heavy processes.
+## Soul File
 
-| Tool | What it does | Example args | Unique Niche Use |
-|------|--------------|--------------|------------------|
-| `shell` | Executes any shell command | `"ls -la /home"` | Automate system maintenance, run scripts, control services on a headless Raspberry Pi or embedded Linux device. |
-| `read_file` | Reads a file from disk | `"/etc/passwd"` | Inspect configuration files, read logs, retrieve data for the LLM to analyse – all without a web interface. |
-| `write_file` | Writes content to a file (args: `filename\ncontent`) | `"notes.txt\nBuy milk"` | Create notes, write configuration files, save results – perfect for offline data logging. |
-| `http_get` | Fetches a URL (via libcurl) | `"https://api.example.com/data"` | Retrieve weather, fetch RSS feeds, call local REST APIs – even on devices with no browser, just a network stack. |
-| `math` | Evaluates an expression using `bc` | `"2 + 2 * 5"` | Let the LLM do arithmetic, unit conversions, or simple calculations without relying on external tools. |
-| `list_dir` | Lists directory contents | `"/home/user"` | Explore filesystem, find documents, check available storage – all natively, without forking a shell. |
+All persistent memories are written to `shadowclaw_data/shadowsoul.md` in Markdown format. It contains:
 
----
-
-## Niche Use Case Examples:
+- `## Core Memory` – JSON key‑value store.
+- `## Skills` – list of registered skills (JSON).
+- `## Cron Jobs` – all scheduled jobs.
+- `## Webhooks` – registered webhooks.
+- `## Conversation Log` – user, assistant, tool calls, and results.
 
-### 1. Raspberry Pi Zero / IoT Sensor Node
-- **Tool used:** `shell` + `write_file`  
-- **Flow:** LLM asks to read a temperature sensor via a shell script, then logs the value to a file.  
-- **Why Shadowclaw?** Runs in <200KB RAM, no Python, no bloat. Persistent memory keeps the last readings.
+The file is updated every 5 writes (write‑behind) and immediately after important events.
 
-### 2. Air‑Gapped Engineering Workstation
-- **Tool used:** `http_get` + `math` + `read_file`  
-- **Flow:** Engineer asks for a component value calculation; LLM fetches a local datasheet via HTTP, reads a config file, does the math, and writes a report.  
-- **Why Shadowclaw?** No cloud, no internet required. All data stays on the machine.
+---
 
-### 3. Embedded Router Automation
-- **Tool used:** `shell` + `list_dir`  
-- **Flow:** Network admin asks for a list of active interfaces; LLM runs `ifconfig` and parses the output, then suggests a config change.  
-- **Why Shadowclaw?** Tiny binary fits in router storage (often just a few MB). Uses curl for local API calls.
+## ⚙️ Configuration via Environment Variables
 
-### 4. Low‑Power Field Logger
-- **Tool used:** `write_file` + `math`  
-- **Flow:** A solar‑powered device collects environmental data; Shadowclaw can process and store it locally, then answer queries about trends.  
-- **Why Shadowclaw?** No database needed – just a binary and a single file (`shadowclaw.bin`) for persistence.
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `SHADOWCLAW_CONNECT_TIMEOUT` | 10 | Seconds to wait for Ollama connection. |
+| `SHADOWCLAW_TOTAL_TIMEOUT` | 120 | Total LLM request timeout (increased on retries). |
+| `SHADOWCLAW_RETRY_ATTEMPTS` | 3 | Number of retries with exponential backoff. |
 
 ---
 
-## Credits
+## 📁 Project Structure
 
-- **Tsoding** – for the “insane shadow data trick” (the header‑before‑data arena idea).
-- **cJSON** – Dave Gamble and contributors – the minimal JSON parser used.
-- **Ollama** – HTTP requests via curl.
-- **Vibe Code** - xAI Grok 4.20 and Deepseek AI
-- **Openclaw**
-- **webXOS**
----
+```
+shadowclaw/
+├── shadowclaw.c          # Main program, arena, tools, cron, LLM
+├── interpreter.c         # Local command interpreter (used in --no-llm mode)
+├── interpreter.h         # Header for interpreter
+├── cJSON.c / cJSON.h     # JSON library
+├── Makefile              # Build instructions
+├── start.sh              # Helper startup script (checks dependencies)
+└── README.md             # This file
+```
 
-## License
+## 📄 License
 
-This project is released under the open sourced MIT License.  
-cJSON is also MIT licensed.
+MIT License.