Design.md

🚀 PygmyClaw Multi-Agent System — Architecture & Progress

---

📌 Overview

This project is evolving into a multi-agent AI command center capable of:

• Code generation & execution
• Task routing (code / research / image)
• Persistent sessions
• Memory + history tracking
• Hugging Face dataset logging
• Autonomous workflows (future)

---

🎯 Vision

A unified AI system that can think, act, remember, and improve over time

---

🧠 System Architecture

High-Level Flow

User Prompt
    ↓
Agent Router
    ↓
Agent Executor (LLM)
    ↓
Output Parser
    ↓
Execution Layer (Code / Tools)
    ↓
Session Manager (Save State)
    ↓
HF Upload (Persistence)
    ↓
Final Response

---

🧱 Core Components

1. session_manager.py

Responsibility:

• Manage session lifecycle

• Store:

  • workspace.json
  • history.json

Key Methods:

create_session()
load_session(session_id)
append_history(data)
update_workspace(key, value)
get_history()

Structure:

/workspace/hf/sessions/
    └── sess_xxxx/
         ├── workspace.json
         └── history.json

---

2. agent_router.py

Responsibility:

Route user intent → correct agent type

Logic:

"code"     → coding agent
"research" → research agent
"image"    → image agent
default    → command agent

Example:

Input	Routed To
"write python code"	code
"who is elon musk"	research
"generate image"	image

---

3. agent_executor.py

Responsibility:

Wrapper around PygmyClaw LLM

Features:

• Injects system prompts
• Controls behavior per agent type

Example:

if agent_type == "code":
    return """
    - Only return Python code
    - No JSON
    - Must be executable
    """

---

4. run_llm() (Core Brain)

Responsibilities:

• Call executor
• Log output
• Save to session
• Upload to HF

Flow:

Prompt → LLM → Response
           ↓
    Save history
           ↓
    Upload session

---

5. run_agent()

Responsibilities:

• Route task
• Call LLM
• Extract code
• Execute if needed

Flow:

Prompt
  ↓
Router → agent_type
  ↓
LLM
  ↓
Code Extract
  ↓
Execute (if code)

---

6. Code Execution Layer

Current:

def run_code(code):
    write → temp file
    execute → python3 file

Issues solved:

• Syntax errors handled
• Execution isolated
• Logging added

---

7. Hugging Face Upload

Dataset:

rahul7star/pyclaw2

Structure:

sessions/
   └── sess_xxx/
        ├── workspace.json
        └── history.json

Upload Logic:

upload_session(session_id)

---

🔄 Current System Flow (Detailed)

User Input
   ↓
route_task()
   ↓
AgentExecutor.run()
   ↓
LLM Output
   ↓
extract_code()
   ↓
run_code() (if needed)
   ↓
SessionManager.append_history()
   ↓
upload_session()
   ↓
Return result

---

✅ What We Have Built So Far

✔ Core System

• ✅ Multi-agent routing
• ✅ LLM execution wrapper
• ✅ Code extraction & execution
• ✅ Session persistence
• ✅ HF dataset integration
• ✅ Command system (REPL)

---

✔ Stability Fixes

• Fixed session initialization errors
• Fixed missing session_id
• Fixed upload path issues
• Fixed PygmyClaw API mismatches
• Fixed shell execution fallback
• Fixed malformed LLM outputs (partial)

---

✔ Working Features

• Generate & run Python code
• Save execution history
• Persist sessions across runs
• Upload sessions to HF
• Basic autonomous agent loop

---

⚠️ Known Limitations

• ❌ LLM sometimes returns invalid code
• ❌ No retry/fix loop yet
• ❌ Tools not fully functional
• ❌ No argument parsing for tools
• ❌ No model switching (yet)

---

🧠 Design Philosophy

1. Local-First Execution

• Prefer running code locally over LLM calls

2. Structured Memory

• Everything stored in session files

3. Deterministic Flow

• Avoid unpredictable LLM outputs when possible

4. Modular Agents

• Each agent has a clear responsibility

---

🚀 Next Steps (Roadmap)

---

🔥 Phase 1 (Current — Stabilization)

• Improve code validation before execution
• Add retry loop for failed code
• Improve logging & debugging

---

🔥 Phase 2 (Next)

Multi-Model Routing

Code → Code LLM
Research → Reasoning LLM
Image → Diffusion model

---

🔥 Phase 3

Tool System (Deferred)

• Convert code → reusable tools
• Tool execution engine
• Tool selection logic

---

🔥 Phase 4

Autonomous Intelligence

• Self-improving agent
• Task decomposition
• Planning + execution loop

---

🧭 Future Architecture (Target)

                ┌──────────────┐
                │   USER UI    │
                └──────┬───────┘
                       ↓
                ┌──────────────┐
                │   ROUTER     │
                └──────┬───────┘
                       ↓
        ┌──────────────┼──────────────┐
        ↓              ↓              ↓
   Code Agent     Research Agent   Image Agent
        ↓              ↓              ↓
        └──────→ Executor Layer ←────┘
                       ↓
              Execution Engine
                       ↓
              Session Manager
                       ↓
                HF Dataset Store

---

💡 Key Insight

You are building:

❌ NOT just a chatbot ✅ BUT a persistent AI system with memory, execution, and evolution

---

🧪 Debugging Tips

• Check logs:

/workspace/api.log

• Check session:

/workspace/hf/sessions/

• Verify upload:
• HF dataset repo

---

🏁 Summary

Current State:

✔ Functional multi-agent system ✔ Code execution working ✔ Session persistence working ⚠ Needs stabilization

---

Next Milestone:

👉 Multi-model intelligence layer

---

🚀 Final Thought

Once stable, this system becomes:

🧠 A self-building AI platform — not just an interface

---