docs/guides/full-pipeline.md

# Full Pipeline Guide

This guide walks through building a complete ACE pipeline from scratch — choosing components, defining an environment, running training, and saving results.

## Components

A full pipeline needs four things:

1. **LLM Client** — the language model powering all three roles
2. **Three Roles** — Agent, Reflector, SkillManager
3. **Environment** — evaluates agent outputs
4. **Samples** — training data with questions and ground truth

## Step 1: Create the Roles

Each role takes a model string directly. Supports any [LiteLLM model](https://docs.litellm.ai/) or PydanticAI-native identifier:

```python
from ace import Agent, Reflector, SkillManager

agent = Agent("gpt-4o-mini")
reflector = Reflector("gpt-4o-mini")
skill_manager = SkillManager("gpt-4o-mini")
```

Optionally use a cheaper model for learning:

```python
agent = Agent("gpt-4o")
reflector = Reflector("gpt-4o-mini")
skill_manager = SkillManager("gpt-4o-mini")
```

## Step 3: Define an Environment

The environment evaluates agent outputs. Extend `TaskEnvironment` and implement `evaluate()`:

```python
from ace import TaskEnvironment, EnvironmentResult

class MathEnvironment(TaskEnvironment):
    def evaluate(self, sample, agent_output):
        correct = str(sample.ground_truth).lower() in str(agent_output.final_answer).lower()
        return EnvironmentResult(
            feedback="Correct!" if correct else f"Incorrect. Expected: {sample.ground_truth}",
            ground_truth=sample.ground_truth,
            metrics={"accuracy": 1.0 if correct else 0.0},
        )
```

Or use the built-in `SimpleEnvironment` for basic ground-truth matching:

```python
from ace import SimpleEnvironment

environment = SimpleEnvironment()
```

## Step 4: Prepare Samples

```python
from ace import Sample

samples = [
    Sample(question="What is 2+2?", context="", ground_truth="4"),
    Sample(question="Capital of France?", context="", ground_truth="Paris"),
    Sample(question="Who wrote Hamlet?", context="", ground_truth="Shakespeare"),
]
```

## Step 5: Build and Run the Pipeline

```python
from ace import ACE

runner = ACE.from_roles(
    agent=agent,
    reflector=reflector,
    skill_manager=skill_manager,
    environment=environment,
)

results = runner.run(samples, epochs=3)
```

## Step 6: Save the Skillbook

```python
runner.save("trained.json")
print(f"Learned {len(runner.skillbook.skills())} strategies")
```

## Complete Example

```python
from ace import (
    ACE, Agent, Reflector, SkillManager,
    Sample, SimpleEnvironment,
)

# Roles (each takes a model string directly)
agent = Agent("gpt-4o-mini")
reflector = Reflector("gpt-4o-mini")
skill_manager = SkillManager("gpt-4o-mini")

# Pipeline
runner = ACE.from_roles(
    agent=agent,
    reflector=reflector,
    skill_manager=skill_manager,
    environment=SimpleEnvironment(),
)

# Training data
samples = [
    Sample(question="What is 2+2?", context="", ground_truth="4"),
    Sample(question="Capital of France?", context="", ground_truth="Paris"),
]

# Train and save
results = runner.run(samples, epochs=3)
runner.save("trained.json")
```

## Checkpoints

Save the skillbook automatically during long training runs:

```python
runner = ACE.from_roles(
    agent=agent,
    reflector=reflector,
    skill_manager=skill_manager,
    environment=environment,
    checkpoint_dir="./checkpoints",
    checkpoint_interval=10,  # Save every 10 samples
)
```

This creates:

- `ace_checkpoint_10.json`, `ace_checkpoint_20.json`, etc.
- `ace_latest.json` (always the most recent)

## Deduplication

Prevent duplicate skills from accumulating (requires `uv add ace-framework[deduplication]`):

```python
from ace import DeduplicationConfig, DeduplicationManager

dedup = DeduplicationManager(DeduplicationConfig(
    enabled=True,
    embedding_model="text-embedding-3-small",
    similarity_threshold=0.85,
))

runner = ACE.from_roles(
    ...,
    dedup_manager=dedup,
    dedup_interval=10,
)
```

## Custom Prompts

The default prompts are v2.1 and work well out of the box. You can pass your own templates via the `prompt_template` parameter:

```python
agent = Agent(llm, prompt_template="Your custom agent prompt with {skillbook}, {question}, {context}")
reflector = Reflector(llm, prompt_template="Your custom reflector prompt ...")
skill_manager = SkillManager(llm, prompt_template="Your custom skill manager prompt ...")
```

See [Prompt Engineering](prompts.md) for template variables and more examples.

## Testing Without API Calls

Use `test` as the model to get PydanticAI's built-in test model, or use `unittest.mock` to patch the agent's `run_sync` method:

```python
agent = Agent("test")
reflector = Reflector("test")
skill_manager = SkillManager("test")
```

## Observability

Add Opik tracing to any pipeline via `extra_steps` (requires `uv add ace-framework[observability]`):

```python
from ace import ACE, OpikStep, register_opik_litellm_callback

runner = ACE.from_roles(
    agent=agent,
    reflector=reflector,
    skill_manager=skill_manager,
    environment=environment,
    extra_steps=[OpikStep(project_name="my-experiment")],
)

# Optionally add per-LLM-call cost tracking
register_opik_litellm_callback(project_name="my-experiment")
```

See [Opik Observability](../integrations/opik.md) for full details.

## Going Deeper: Manual Pipeline Composition

The `ACE.from_roles()` runner composes a `Pipeline` internally. You can build the
same pipeline yourself for full control over step ordering, branching, and
custom steps:

```python
from ace import Pipeline, AgentStep, EvaluateStep, learning_tail

pipe = Pipeline([
    AgentStep(agent, skillbook),
    EvaluateStep(environment),
    *learning_tail(reflector, skill_manager, skillbook),
])
```

See [Composing Pipelines](composing-pipelines.md) for the complete guide.

## What to Read Next

- [Composing Pipelines](composing-pipelines.md) — compose custom pipelines from steps
- [Async Learning](async-learning.md) — parallel Reflector execution
- [Prompt Engineering](prompts.md) — customize prompt templates
- [Integration Pattern](integration.md) — wrap existing agents instead
- [Opik Observability](../integrations/opik.md) — monitor costs and traces