{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ " # ACE Next — Interactive Demo\n", "\n", "\n", "\n", " This notebook walks through the refactored `ace_next` pipeline.\n", "\n", " It covers:\n", "\n", "\n", "\n", " 1. **Runners** — `ACE` (full pipeline) and `TraceAnalyser` (learning-only)\n", "\n", " 2. **Steps** — individual pipeline steps and `learning_tail()`\n", "\n", " 3. **Manual pipeline construction** — composing steps by hand\n", "\n", " 4. **Custom environments** — writing your own evaluator\n", "\n", " 5. **Checkpointing & deduplication** — production features\n", "\n", " 6. **Observability with Opik** — pipeline traces and LLM cost tracking\n", "\n", " 7. **Skillbook persistence** — save / reload\n", "\n", " 8. **TraceAnalyser** — learning from pre-recorded traces\n", "\n", "\n", "\n", " **Requirements:** `uv sync` from the repo root.\n", "\n", " Set your LLM API key before running:\n", "\n", " ```bash\n", "\n", " export OPENAI_API_KEY=\"sk-...\"\n", "\n", " ```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " ## 1. Setup & Imports" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Project root: /home/david/Desktop/projects/Kayba/agentic-context-engine\n", "Setup OK\n" ] } ], "source": [ "import os\n", "import sys\n", "import logging\n", "import tempfile\n", "from pathlib import Path\n", "\n", "import nest_asyncio\n", "\n", "nest_asyncio.apply()\n", "\n", "# Silence LiteLLM's verbose logging so notebook output stays clean\n", "logging.getLogger(\"LiteLLM\").setLevel(logging.WARNING)\n", "logging.getLogger(\"LiteLLM Router\").setLevel(logging.WARNING)\n", "logging.getLogger(\"LiteLLM Proxy\").setLevel(logging.WARNING)\n", "\n", "# Also suppress litellm's own set_verbose flag\n", "try:\n", " import litellm\n", " litellm.set_verbose = False\n", "except ImportError:\n", " pass\n", "\n", "# Ensure the project root is on sys.path so `ace`, `ace_next`, and `pipeline`\n", "# are importable regardless of where the notebook kernel starts.\n", "_here = Path(__file__).resolve().parent if \"__file__\" in dir() else Path.cwd()\n", "_root = _here\n", "for _p in [_here] + list(_here.parents):\n", " if (_p / \"pipeline\" / \"__init__.py\").exists():\n", " _root = _p\n", " break\n", "sys.path.insert(0, str(_root))\n", "\n", "from dotenv import load_dotenv\n", "\n", "load_dotenv(_root / \".env\")\n", "\n", "print(f\"Project root: {_root}\")\n", "print(\"Setup OK\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " ## 2. Core Imports\n", "\n", "\n", "\n", " Everything lives in `ace_next` — fully self-contained, zero cross-imports." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "All imports OK\n" ] } ], "source": [ "from ace_next import (\n", " # Runners\n", " ACE,\n", " TraceAnalyser,\n", " # Role implementations\n", " Agent,\n", " Reflector,\n", " SkillManager,\n", " # LLM providers\n", " LiteLLMClient,\n", " # Core types\n", " Sample,\n", " Skillbook,\n", " SimpleEnvironment,\n", " TaskEnvironment,\n", " EnvironmentResult,\n", ")\n", "from ace_next.core import AgentOutput, ACEStepContext, SkillbookView\n", "\n", "print(\"All imports OK\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " ## 3. Configure the LLM Client\n", "\n", "\n", "\n", " We use LiteLLM which supports 100+ providers. Swap the model string\n", "\n", " for any provider: `gpt-4o-mini`, `claude-sonnet-4-5-20250929`,\n", "\n", " `bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0`, etc." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "LLM client ready: us.anthropic.claude-haiku-4-5-20251001-v1:0\n" ] } ], "source": [ "MODEL = os.getenv(\"ACE_MODEL\", \"us.anthropic.claude-haiku-4-5-20251001-v1:0\")\n", "client = LiteLLMClient(model=MODEL)\n", "\n", "print(f\"LLM client ready: {MODEL}\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " ## 4. Build Roles\n", "\n", "\n", "\n", " The three ACE roles share the same LLM client. Each is independently\n", "\n", " customisable (prompt templates, retries, etc.)." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Roles created: Agent, Reflector, SkillManager\n" ] } ], "source": [ "agent = Agent(client)\n", "reflector = Reflector(client)\n", "skill_manager = SkillManager(client)\n", "\n", "print(\"Roles created: Agent, Reflector, SkillManager\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " ## 5. Define Training Samples" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Prepared 5 training samples\n" ] } ], "source": [ "samples = [\n", " Sample(question=\"What is the capital of France?\", ground_truth=\"Paris\"),\n", " Sample(question=\"What is the capital of Japan?\", ground_truth=\"Tokyo\"),\n", " Sample(question=\"What is the capital of Brazil?\", ground_truth=\"Brasilia\"),\n", " Sample(question=\"What is the capital of Australia?\", ground_truth=\"Canberra\"),\n", " Sample(question=\"What is the capital of Nigeria?\", ground_truth=\"Abuja\"),\n", "]\n", "\n", "print(f\"Prepared {len(samples)} training samples\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " ---\n", "\n", " ## 6. ACE Runner — Full Adaptive Pipeline\n", "\n", "\n", "\n", " The `ACE` runner is the full closed-loop pipeline:\n", "\n", " ```\n", "\n", " Agent → Evaluate → Reflect → Tag → Update → Apply\n", "\n", " ```\n", "\n", "\n", "\n", " It takes `Sample` objects and an optional `TaskEnvironment`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " ### 6a. With SimpleEnvironment\n", "\n", "\n", "\n", " `SimpleEnvironment` checks if the ground truth appears in the agent's\n", "\n", " answer (case-insensitive substring match)." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Processed 3 samples\n", "\n", " Q: What is the capital of France?\n", " A: The capital of France is Paris.\n", " Q: What is the capital of Japan?\n", " A: Tokyo is the capital of Japan.\n", " Q: What is the capital of Brazil?\n", " A: The capital of Brazil is Brasília.\n" ] } ], "source": [ "skillbook = Skillbook()\n", "\n", "ace = ACE.from_roles(\n", " agent=agent,\n", " reflector=reflector,\n", " skill_manager=skill_manager,\n", " environment=SimpleEnvironment(),\n", " skillbook=skillbook,\n", ")\n", "\n", "results = ace.run(samples[:3], epochs=1)\n", "\n", "print(f\"Processed {len(results)} samples\\n\")\n", "for r in results:\n", " if r.error:\n", " print(f\" ERROR at {r.failed_at}: {r.error}\")\n", " elif r.output:\n", " ctx: ACEStepContext = r.output\n", " answer = ctx.agent_output.final_answer if ctx.agent_output else \"N/A\"\n", " print(f\" Q: {r.sample.question}\")\n", " print(f\" A: {answer}\")\n" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Skillbook after 1 epoch:\n", " Stats: {'sections': 6, 'skills': 8, 'tags': {'helpful': 8, 'harmful': 0, 'neutral': 0}}\n", " - [problem_classification-00001] Classify factual queries as direct recall, not strategic problems\n", " - [knowledge_retrieval-00002] Retrieve geographic facts directly; verify output format matches evaluation ground truth\n", " - [strategy_selection-00003] Avoid complex strategies for straightforward factual queries\n", " - [question_analysis-00004] Recognize factual retrieval questions requiring direct knowledge access\n", " - [verification_patterns-00005] Support factual answers with historical context or authoritative evidence\n" ] } ], "source": [ "print(f\"\\nSkillbook after 1 epoch:\")\n", "print(f\" Stats: {skillbook.stats()}\")\n", "for skill in skillbook.skills()[:5]:\n", " print(f\" - [{skill.id}] {skill.content}\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " ### 6b. Custom Environment\n", "\n", "\n", "\n", " Create your own evaluator by subclassing `TaskEnvironment`." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "ExactMatchEnvironment defined\n" ] } ], "source": [ "class ExactMatchEnvironment(TaskEnvironment):\n", " \"\"\"Strict evaluation: answer must exactly match ground truth.\"\"\"\n", "\n", " def evaluate(self, sample: Sample, agent_output: AgentOutput) -> EnvironmentResult:\n", " expected = (sample.ground_truth or \"\").strip().lower()\n", " predicted = agent_output.final_answer.strip().lower()\n", " correct = expected == predicted\n", "\n", " return EnvironmentResult(\n", " feedback=\"Correct!\" if correct else f\"Wrong. Expected: {sample.ground_truth}\",\n", " ground_truth=sample.ground_truth,\n", " metrics={\"accuracy\": 1.0 if correct else 0.0},\n", " )\n", "\n", "\n", "print(\"ExactMatchEnvironment defined\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "skillbook2 = Skillbook()\n", "\n", "ace2 = ACE.from_roles(\n", " agent=Agent(client),\n", " reflector=Reflector(client),\n", " skill_manager=SkillManager(client),\n", " environment=ExactMatchEnvironment(),\n", " skillbook=skillbook2,\n", ")\n", "\n", "results2 = ace2.run(samples[:2], epochs=1)\n", "\n", "for r in results2:\n", " if r.output:\n", " ctx = r.output\n", " print(f\" Q: {r.sample.question}\")\n", " print(f\" A: {ctx.agent_output.final_answer if ctx.agent_output else 'N/A'}\")\n", " if ctx.reflections:\n", " print(f\" Insight: {ctx.reflections[0].key_insight}\")\n", " print()" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Skillbook after 1 epoch:\n", " Stats: {'sections': 0, 'skills': 0, 'tags': {'helpful': 0, 'harmful': 0, 'neutral': 0}}\n" ] } ], "source": [ "print(f\"\\nSkillbook after 1 epoch:\")\n", "print(f\" Stats: {skillbook2.stats()}\")\n", "for skill in skillbook2.skills()[:5]:\n", " print(f\" - [{skill.id}] {skill.content}\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " ### 6c. Without Environment\n", "\n", "\n", "\n", " When no environment is provided, `EvaluateStep` is a no-op. The Reflector\n", "\n", " still learns from ground-truth comparison in the trace." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:43:07 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:43:13 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:43:13 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:43:13 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:43:19 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:43:19 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:43:23 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:43:23 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:43:29 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:43:32 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:43:32 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:43:42 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n", "Processed 2 samples (no environment)\n", "Skills learned: {'sections': 3, 'skills': 6, 'tags': {'helpful': 6, 'harmful': 0, 'neutral': 0}}\n" ] } ], "source": [ "skillbook3 = Skillbook()\n", "\n", "ace3 = ACE.from_roles(\n", " agent=Agent(client),\n", " reflector=Reflector(client),\n", " skill_manager=SkillManager(client),\n", " skillbook=skillbook3,\n", " # No environment — EvaluateStep passes through\n", ")\n", "\n", "results3 = ace3.run(samples[:2], epochs=1)\n", "print(f\"Processed {len(results3)} samples (no environment)\")\n", "print(f\"Skills learned: {skillbook3.stats()}\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " ### 6d. Multi-Epoch Training\n", "\n", "\n", "\n", " Multiple epochs let the agent revisit samples with an evolving skillbook.\n", "\n", " Skills accumulate and refine across passes." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:43:57 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:04 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:04 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:04 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:09 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:09 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:09 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:14 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:14 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:15 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:15 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:15 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:19 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:20 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:20 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:20 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:22 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:22 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:26 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:26 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:26 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n", "\u001b[92m16:44:26 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n", "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:29 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:29 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:29 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:32 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:32 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:32 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:37 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:37 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:37 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:39 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:39 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:39 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:41 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:44 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:44 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:44 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:46 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:46 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:48 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:50 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:50 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n", "\u001b[92m16:44:50 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n", "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:55 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:56 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:56 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:58 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:44:58 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:45:00 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:45:07 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:45:10 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:45:10 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:45:19 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:45:19 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:45:32 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:45:32 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:45:41 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:45:41 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:45:52 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n", "Total results across 2 epochs: 10\n", "Skills learned: {'sections': 1, 'skills': 8, 'tags': {'helpful': 38, 'harmful': 1, 'neutral': 0}}\n", " Epoch 1: 4/5 correct\n", " Epoch 2: 4/5 correct\n" ] } ], "source": [ "skillbook4 = Skillbook()\n", "\n", "ace4 = ACE.from_roles(\n", " agent=Agent(client),\n", " reflector=Reflector(client),\n", " skill_manager=SkillManager(client),\n", " environment=SimpleEnvironment(),\n", " skillbook=skillbook4,\n", ")\n", "\n", "results4 = ace4.run(samples, epochs=2)\n", "\n", "print(f\"Total results across 2 epochs: {len(results4)}\")\n", "print(f\"Skills learned: {skillbook4.stats()}\")\n", "\n", "# Print per-epoch accuracy\n", "for epoch in range(1, 3):\n", " epoch_results = [r for r in results4 if r.output and r.output.epoch == epoch]\n", " correct = sum(\n", " 1 for r in epoch_results\n", " if r.output and r.output.agent_output\n", " and (r.sample.ground_truth or \"\").lower() in r.output.agent_output.final_answer.lower()\n", " )\n", " print(f\" Epoch {epoch}: {correct}/{len(epoch_results)} correct\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " ---\n", "\n", " ## 7. Manual Step-by-Step Pipeline\n", "\n", "\n", "\n", " Under the hood, runners compose `Pipeline` objects from individual steps.\n", "\n", " Here we build one by hand to see exactly what each step does." ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Pipeline steps: 6\n", " requires: frozenset({'sample', 'skillbook'})\n", " provides: frozenset({'agent_output', 'skill_manager_output', 'reflection', 'trace'})\n" ] } ], "source": [ "from pipeline import Pipeline\n", "from ace_next.steps import (\n", " AgentStep,\n", " EvaluateStep,\n", " ReflectStep,\n", " TagStep,\n", " UpdateStep,\n", " ApplyStep,\n", " learning_tail,\n", ")\n", "\n", "skillbook5 = Skillbook()\n", "env = SimpleEnvironment()\n", "\n", "# Build the full pipeline manually\n", "pipe = Pipeline(\n", " [\n", " AgentStep(Agent(client)),\n", " EvaluateStep(env),\n", " *learning_tail(Reflector(client), SkillManager(client), skillbook5),\n", " ]\n", ")\n", "\n", "print(f\"Pipeline steps: {len(pipe._steps)}\")\n", "print(f\" requires: {pipe.requires}\")\n", "print(f\" provides: {pipe.provides}\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " ### Run a single sample through the manual pipeline" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "sample = samples[0]\n", "\n", "# Build the context the same way ACE._build_context() does\n", "ctx = ACEStepContext(\n", " sample=sample,\n", " skillbook=SkillbookView(skillbook5),\n", " epoch=1,\n", " total_epochs=1,\n", " step_index=0,\n", " total_steps=1,\n", " global_sample_index=0,\n", ")\n", "\n", "print(f\"Before pipeline:\")\n", "print(f\" Skills: {skillbook5.stats()}\")\n", "print(f\" agent_output: {ctx.agent_output}\")\n", "\n", "# Run the full pipeline on a single context\n", "from pipeline.protocol import SampleResult\n", "\n", "results_manual = pipe.run([ctx])\n", "\n", "print(f\"\\nAfter pipeline:\")\n", "for r in results_manual:\n", " if r.error:\n", " print(f\" ERROR: {r.error}\")\n", " elif r.output:\n", " out: ACEStepContext = r.output\n", " print(f\" Agent answer: {out.agent_output.final_answer if out.agent_output else 'N/A'}\")\n", " print(f\" Reflector insight: {out.reflections[0].key_insight if out.reflections else 'N/A'}\")\n", " print(f\" Skills now: {skillbook5.stats()}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " ### Using `learning_tail()` as a building block\n", "\n", "\n", "\n", " `learning_tail()` returns the standard learning steps:\n", "\n", " `[ReflectStep, TagStep, UpdateStep, ApplyStep]` with optional\n", "\n", " deduplication and checkpoint steps appended." ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "learning_tail() returns 4 steps:\n", " - ReflectStep\n", " - TagStep\n", " - UpdateStep\n", " - ApplyStep\n" ] } ], "source": [ "skillbook6 = Skillbook()\n", "\n", "tail = learning_tail(\n", " Reflector(client),\n", " SkillManager(client),\n", " skillbook6,\n", ")\n", "\n", "print(f\"learning_tail() returns {len(tail)} steps:\")\n", "for step in tail:\n", " print(f\" - {type(step).__name__}\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " ---\n", "\n", " ## 8. Checkpointing\n", "\n", "\n", "\n", " Save the skillbook every N successful samples so you can resume after\n", "\n", " interruption or compare skillbook evolution over time." ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:59:25 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:59:31 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:59:31 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:59:31 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:59:36 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:59:36 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:59:36 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:59:41 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:59:41 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:59:42 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:59:42 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:59:42 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:59:46 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:59:48 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:59:48 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:59:48 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:59:50 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:59:50 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:59:52 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:59:54 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:59:54 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:59:57 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:59:58 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m16:59:58 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n", "INFO [ace_next.steps.checkpoint] CheckpointStep: saved checkpoint at sample 2 → /tmp/tmpqnq54gga/checkpoint_2.json\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m17:00:04 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m17:00:05 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m17:00:05 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m17:00:17 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m17:00:17 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n", "INFO [ace_next.steps.checkpoint] CheckpointStep: saved checkpoint at sample 4 → /tmp/tmpqnq54gga/checkpoint_4.json\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m17:00:28 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n", "Checkpoint files:\n", " checkpoint_2.json (2464 bytes)\n", " checkpoint_4.json (4971 bytes)\n", " latest.json (4971 bytes)\n" ] } ], "source": [ "skillbook7 = Skillbook()\n", "\n", "with tempfile.TemporaryDirectory() as tmpdir:\n", " ace7 = ACE.from_roles(\n", " agent=Agent(client),\n", " reflector=Reflector(client),\n", " skill_manager=SkillManager(client),\n", " environment=SimpleEnvironment(),\n", " skillbook=skillbook7,\n", " checkpoint_dir=tmpdir,\n", " checkpoint_interval=2, # save every 2 successful samples\n", " )\n", "\n", " results7 = ace7.run(samples, epochs=1)\n", "\n", " saved = sorted(Path(tmpdir).glob(\"*.json\"))\n", " print(\"Checkpoint files:\")\n", " for f in saved:\n", " print(f\" {f.name} ({f.stat().st_size} bytes)\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " ---\n", "\n", " ## 9. Deduplication\n", "\n", "\n", "\n", " Merge near-duplicate skills to keep the skillbook compact. The\n", "\n", " `DeduplicationManager` runs periodically during training." ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m17:08:53 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m17:08:59 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m17:08:59 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m17:08:59 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m17:09:04 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m17:09:04 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m17:09:04 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m17:09:08 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m17:09:08 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m17:09:09 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m17:09:09 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m17:09:09 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m17:09:14 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m17:09:15 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m17:09:15 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n", "\u001b[92m17:09:15 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n", "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m17:09:17 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m17:09:17 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m17:09:20 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m17:09:21 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m17:09:21 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m17:09:24 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m17:09:24 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m17:09:25 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m17:09:31 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m17:09:33 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m17:09:33 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n", "\n", "\u001b[1;31mGive Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new\u001b[0m\n", "LiteLLM.Info: If you need to debug this error, use `litellm._turn_on_debug()'.\n", "\n", "WARNING [ace_next.deduplication.detector] Failed to compute batch embeddings via LiteLLM: litellm.AuthenticationError: AuthenticationError: OpenAIException - Error code: 401 - {'error': {'message': 'Incorrect API key provided: sk-proj-********************************************************************************************************************************************************T6EA. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'code': 'invalid_api_key', 'param': None}, 'status': 401}\n", "INFO [ace_next.deduplication.detector] Computed 0 embeddings for skills\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m17:09:42 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m17:09:42 - LiteLLM:INFO\u001b[0m: utils.py:3889 - \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] \n", "LiteLLM completion() model= us.anthropic.claude-haiku-4-5-20251001-v1:0; provider = bedrock\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[92m17:09:54 - LiteLLM:INFO\u001b[0m: utils.py:1629 - Wrapper: Completed Call, calling success_handler\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO [LiteLLM] Wrapper: Completed Call, calling success_handler\n", "Skills after training with dedup: {'sections': 3, 'skills': 7, 'tags': {'helpful': 8, 'harmful': 0, 'neutral': 0}}\n" ] } ], "source": [ "from ace_next import DeduplicationManager, SimilarityDetector\n", "from ace_next.protocols import DeduplicationConfig\n", "\n", "skillbook8 = Skillbook()\n", "\n", "dedup = DeduplicationManager(\n", " DeduplicationConfig(similarity_threshold=0.85)\n", ")\n", "\n", "ace8 = ACE.from_roles(\n", " agent=Agent(client),\n", " reflector=Reflector(client),\n", " skill_manager=SkillManager(client),\n", " environment=SimpleEnvironment(),\n", " skillbook=skillbook8,\n", " dedup_manager=dedup,\n", " dedup_interval=3, # run dedup every 3 samples\n", ")\n", "\n", "results8 = ace8.run(samples, epochs=1)\n", "\n", "print(f\"Skills after training with dedup: {skillbook8.stats()}\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " ---\n", "\n", " ## 10. Observability with Opik\n", "\n", "\n", "\n", " `OpikStep` is an explicit, opt-in pipeline step that logs traces to Opik.\n", "\n", " It is **not** wired into `learning_tail()` — you append it yourself.\n", "\n", "\n", "\n", " Three usage patterns:\n", "\n", " 1. **Pipeline traces + LLM cost tracking** — append `OpikStep()` (default)\n", "\n", " 2. **Pipeline traces only** — `OpikStep(register_litellm_callback=False)`\n", "\n", " 3. **LLM cost tracking only** — `register_opik_litellm_callback()` (no step)" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Opik available: True\n" ] } ], "source": [ "from ace_next import OpikStep, OPIK_AVAILABLE, register_opik_litellm_callback\n", "\n", "print(f\"Opik available: {OPIK_AVAILABLE}\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " ### 10a. Append OpikStep to a custom pipeline\n", "\n", "\n", "\n", " Place it at the end — after the learning tail." ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Pipeline steps (with Opik): 7\n", " - AgentStep\n", " - EvaluateStep\n", " - ReflectStep\n", " - TagStep\n", " - UpdateStep\n", " - ApplyStep\n", " - OpikStep\n" ] } ], "source": [ "if OPIK_AVAILABLE:\n", " skillbook_opik = Skillbook()\n", "\n", " pipe_with_opik = Pipeline(\n", " [\n", " AgentStep(Agent(client)),\n", " EvaluateStep(SimpleEnvironment()),\n", " *learning_tail(Reflector(client), SkillManager(client), skillbook_opik),\n", " OpikStep(project_name=\"ace-demo\"),\n", " ]\n", " )\n", " print(f\"Pipeline steps (with Opik): {len(pipe_with_opik._steps)}\")\n", " for step in pipe_with_opik._steps:\n", " print(f\" - {type(step).__name__}\")\n", "else:\n", " print(\"Opik not installed — skipping pipeline example\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " ### 10b. LLM-level cost tracking only\n", "\n", "\n", "\n", " If you only want per-LLM-call token/cost logging without pipeline traces,\n", "\n", " use the standalone helper. This registers an `OpikLogger` callback on\n", "\n", " `litellm.callbacks`." ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "LiteLLM Opik callback registered: True\n" ] } ], "source": [ "if OPIK_AVAILABLE:\n", " registered = register_opik_litellm_callback(project_name=\"ace-demo\")\n", " print(f\"LiteLLM Opik callback registered: {registered}\")\n", "else:\n", " print(\"Opik not installed — skipping callback example\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " ---\n", "\n", " ## 11. Skillbook Persistence — Save & Reload\n", "\n", "\n", "\n", " Save the learned skillbook to disk and reload it in a future session." ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Saved to learned_skillbook.json (5561 bytes)\n", "Reloaded: {'sections': 5, 'skills': 7, 'tags': {'helpful': 7, 'harmful': 0, 'neutral': 0}}\n", "Stats match: True\n" ] } ], "source": [ "with tempfile.TemporaryDirectory() as tmpdir:\n", " path = Path(tmpdir) / \"learned_skillbook.json\"\n", "\n", " # Save\n", " skillbook.save_to_file(str(path))\n", " print(f\"Saved to {path.name} ({path.stat().st_size} bytes)\")\n", "\n", " # Reload\n", " reloaded = Skillbook.load_from_file(str(path))\n", " print(f\"Reloaded: {reloaded.stats()}\")\n", " print(f\"Stats match: {reloaded.stats() == skillbook.stats()}\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " ---\n", "\n", " ## 12. TraceAnalyser — Learning from Pre-Recorded Traces\n", "\n", "\n", "\n", " `TraceAnalyser` runs the learning tail only — no Agent, no Evaluate.\n", "\n", " Feed it raw trace dicts (the same shape ReflectStep expects) and it\n", "\n", " builds a skillbook from historical data." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Analysed 3 traces\n", "Skills learned: {'sections': 2, 'skills': 6, 'tags': {'helpful': 6, 'harmful': 0, 'neutral': 0}}\n", " - [information_retrieval] Use weather.com for direct weather information queries\n", " - [information_retrieval] Navigate tool → search location → extract data for geographic queries\n", " - [information_retrieval] Extract quantified data with specific values and conditions\n", " - [web_ui_patterns] Identify and dismiss cookie consent popups before accessing page content\n", " - [web_ui_patterns] Handle unexpected UI overlays appearing mid-interaction during web tasks\n" ] } ], "source": [ "# Simulate some pre-recorded traces (e.g., from browser-use history logs)\n", "traces = [\n", " {\n", " \"question\": \"Book a flight from NYC to London\",\n", " \"reasoning\": \"Step 1: Opened booking site. Step 2: Searched flights. Step 3: Selected cheapest option.\",\n", " \"answer\": \"Booked flight AA100 for $450\",\n", " \"skill_ids\": [],\n", " \"feedback\": \"Task succeeded in 3 steps\",\n", " \"ground_truth\": None,\n", " },\n", " {\n", " \"question\": \"Find the cheapest hotel in Paris\",\n", " \"reasoning\": \"Step 1: Opened hotel site. Step 2: Set filters. Step 3: Sorted by price. Step 4: Cookie popup blocked view.\",\n", " \"answer\": \"Failed: could not dismiss cookie popup\",\n", " \"skill_ids\": [],\n", " \"feedback\": \"Task failed — cookie popup blocked interaction after step 3\",\n", " \"ground_truth\": None,\n", " },\n", " {\n", " \"question\": \"Check weather in Tokyo\",\n", " \"reasoning\": \"Step 1: Navigated to weather.com. Step 2: Searched Tokyo. Step 3: Read forecast.\",\n", " \"answer\": \"Tokyo: 22C, partly cloudy\",\n", " \"skill_ids\": [],\n", " \"feedback\": \"Task succeeded in 3 steps — fast and accurate\",\n", " \"ground_truth\": None,\n", " },\n", "]\n", "\n", "skillbook9 = Skillbook()\n", "\n", "analyser = TraceAnalyser.from_roles(\n", " reflector=Reflector(client),\n", " skill_manager=SkillManager(client),\n", " skillbook=skillbook9,\n", ")\n", "\n", "results9 = analyser.run(traces, epochs=1)\n", "\n", "print(f\"Analysed {len(results9)} traces\")\n", "print(f\"Skills learned: {skillbook9.stats()}\")\n", "for skill in skillbook9.skills()[:5]:\n", " print(f\" - [{skill.section}] {skill.content}\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " ### Multi-epoch trace analysis\n", "\n", "\n", "\n", " Each epoch re-processes all traces with the evolving skillbook.\n", "\n", " Early epochs extract obvious patterns; later epochs refine." ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Total results across 2 epochs: 6\n", "Skills after 2 epochs: {'sections': 2, 'skills': 6, 'tags': {'helpful': 9, 'harmful': 0, 'neutral': 0}}\n" ] } ], "source": [ "skillbook10 = Skillbook()\n", "\n", "analyser2 = TraceAnalyser.from_roles(\n", " reflector=Reflector(client),\n", " skill_manager=SkillManager(client),\n", " skillbook=skillbook10,\n", ")\n", "\n", "results10 = analyser2.run(traces, epochs=2)\n", "\n", "print(f\"Total results across 2 epochs: {len(results10)}\")\n", "print(f\"Skills after 2 epochs: {skillbook10.stats()}\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " ---\n", "\n", " ## 13. Mixed Workflow — TraceAnalyser then ACE\n", "\n", "\n", "\n", " A common pattern: build an initial skillbook from historical traces,\n", "\n", " then deploy with live learning." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Phase 1 — TraceAnalyser:\n", " Skills from traces: {'sections': 3, 'skills': 9, 'tags': {'helpful': 9, 'harmful': 0, 'neutral': 0}}\n", "\n", "Phase 2 — ACE live learning:\n", " Processed 3 samples\n", " Skills after live learning: {'sections': 3, 'skills': 12, 'tags': {'helpful': 17, 'harmful': 0, 'neutral': 7}}\n" ] } ], "source": [ "# Phase 1: Build skillbook from historical data\n", "shared_skillbook = Skillbook()\n", "\n", "analyser_phase1 = TraceAnalyser.from_roles(\n", " reflector=Reflector(client),\n", " skill_manager=SkillManager(client),\n", " skillbook=shared_skillbook,\n", ")\n", "analyser_phase1.run(traces, epochs=1)\n", "\n", "print(f\"Phase 1 — TraceAnalyser:\")\n", "print(f\" Skills from traces: {shared_skillbook.stats()}\")\n", "\n", "# Phase 2: Deploy with live ACE learning (reuse the evolved skillbook)\n", "ace_phase2 = ACE.from_roles(\n", " agent=Agent(client),\n", " reflector=Reflector(client),\n", " skill_manager=SkillManager(client),\n", " environment=SimpleEnvironment(),\n", " skillbook=shared_skillbook,\n", ")\n", "\n", "results_phase2 = ace_phase2.run(samples[:3], epochs=1)\n", "\n", "print(f\"\\nPhase 2 — ACE live learning:\")\n", "print(f\" Processed {len(results_phase2)} samples\")\n", "print(f\" Skills after live learning: {shared_skillbook.stats()}\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " ---\n", "\n", " ## 14. Error Handling\n", "\n", "\n", "\n", " Failed samples are captured in `SampleResult.error` — the pipeline\n", "\n", " never drops a sample silently. Other samples continue processing." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " [1] OK answer=The capital of France is Paris.\n", " [2] OK answer=No problem to solve. This prompt provides the operational framework and instructions for ACE Agent v2.1, but does not contain a specific question or problem in the 'Question' field. To proceed, please provide: (1) A specific question or problem to solve, (2) Relevant skillbook entries with strategy IDs and content, and (3) Any additional context needed. Once a question is provided, I will apply the skillbook protocol with complete step-by-step reasoning and specific skill citations.\n", " [3] OK answer=Tokyo is the capital of Japan.\n" ] } ], "source": [ "bad_samples = [\n", " samples[0],\n", " Sample(question=\"\", ground_truth=\"\"), # edge case: empty question\n", " samples[1],\n", "]\n", "\n", "skillbook11 = Skillbook()\n", "ace11 = ACE.from_roles(\n", " agent=Agent(client),\n", " reflector=Reflector(client),\n", " skill_manager=SkillManager(client),\n", " environment=SimpleEnvironment(),\n", " skillbook=skillbook11,\n", ")\n", "\n", "results11 = ace11.run(bad_samples, epochs=1)\n", "\n", "for i, r in enumerate(results11, 1):\n", " status = \"OK\" if r.error is None else f\"FAIL ({r.failed_at})\"\n", " if r.output and r.output.agent_output:\n", " answer = r.output.agent_output.final_answer\n", " else:\n", " answer = \"N/A\"\n", " print(f\" [{i}] {status:20s} answer={answer}\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " ---\n", "\n", " ## 15. Inspecting the SkillbookView\n", "\n", "\n", "\n", " Steps receive a read-only `SkillbookView` on the context.\n", "\n", " This prevents accidental mutations from within pipeline steps." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "SkillbookView: SkillbookView(0 skills)\n", " len: 0\n", " stats: {'sections': 0, 'skills': 0, 'tags': {'helpful': 0, 'harmful': 0, 'neutral': 0}}\n", " prompt: skills[0\t]:...\n" ] } ], "source": [ "sb = Skillbook()\n", "view = SkillbookView(sb)\n", "\n", "print(f\"SkillbookView: {view}\")\n", "print(f\" len: {len(view)}\")\n", "print(f\" stats: {view.stats()}\")\n", "print(f\" prompt: {view.as_prompt()[:200]}...\")\n", "\n", "# Iterate over skills in the view\n", "for skill in view:\n", " print(f\" - {skill.id}: {skill.content}\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " ---\n", "\n", " ## Summary\n", "\n", "\n", "\n", " | What | How |\n", "\n", " |------|-----|\n", "\n", " | Full pipeline | `ACE.from_roles(agent=..., reflector=..., skill_manager=...)` |\n", "\n", " | With environment | `ACE.from_roles(..., environment=SimpleEnvironment())` |\n", "\n", " | Without environment | `ACE.from_roles(...)` — EvaluateStep is a no-op |\n", "\n", " | Multi-epoch | `ace.run(samples, epochs=3)` |\n", "\n", " | Checkpointing | `ACE.from_roles(..., checkpoint_dir=\"./ckpts\", checkpoint_interval=10)` |\n", "\n", " | Deduplication | `ACE.from_roles(..., dedup_manager=dedup, dedup_interval=5)` |\n", "\n", " | Opik tracing | `Pipeline([...steps..., OpikStep(project_name=\"my-project\")])` |\n", "\n", " | LLM cost tracking | `register_opik_litellm_callback()` |\n", "\n", " | Trace analysis | `TraceAnalyser.from_roles(reflector=..., skill_manager=...)` |\n", "\n", " | Save skillbook | `ace.save(\"path.json\")` or `skillbook.save_to_file(\"path.json\")` |\n", "\n", " | Load skillbook | `Skillbook.from_file(\"path.json\")` |\n", "\n", " | Manual steps | `Pipeline([AgentStep(a), EvaluateStep(e), *learning_tail(r, sm, sb)])` |\n", "\n", " | Learning tail | `learning_tail(reflector, skill_manager, skillbook)` |\n", "\n", "\n", "\n", " **Pipeline:**\n", "\n", " ```\n", "\n", " ACE: Agent → Evaluate → Reflect → Tag → Update → Apply → [Dedup] → [Checkpoint] → [Opik]\n", "\n", " TraceAnalyser: Reflect → Tag → Update → Apply → [Dedup] → [Checkpoint] → [Opik]\n", "\n", " ```" ] } ], "metadata": { "kernelspec": { "display_name": "ace-framework (3.12.0)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.0" } }, "nbformat": 4, "nbformat_minor": 2 }