| # Situation Analysis: Pydantic-AI + Microsoft Agent Framework Integration | |
| **Date:** November 27, 2025 | |
| **Status:** ACTIVE DECISION REQUIRED | |
| **Risk Level:** HIGH - DO NOT MERGE PR #41 UNTIL RESOLVED | |
| --- | |
| ## 1. The Problem | |
| We almost merged a refactor that would have **deleted** multi-agent orchestration capability from the codebase, mistakenly believing pydantic-ai and Microsoft Agent Framework were mutually exclusive. | |
| **They are not.** They are complementary: | |
| - **pydantic-ai** (Library): Ensures LLM outputs match Pydantic schemas | |
| - **Microsoft Agent Framework** (Framework): Orchestrates multi-agent workflows | |
| --- | |
| ## 2. Current Branch State | |
| | Branch | Location | Has Agent Framework? | Has Pydantic-AI Improvements? | Status | | |
| |--------|----------|---------------------|------------------------------|--------| | |
| | `origin/dev` | GitHub | YES | NO | **SAFE - Source of Truth** | | |
| | `huggingface-upstream/dev` | HF Spaces | YES | NO | **SAFE - Same as GitHub** | | |
| | `origin/main` | GitHub | YES | NO | **SAFE** | | |
| | `feat/pubmed-fulltext` | GitHub | NO (deleted) | YES | **DANGER - Has destructive refactor** | | |
| | `refactor/pydantic-unification` | Local | NO (deleted) | YES | **DANGER - Redundant, delete** | | |
| | Local `dev` | Local only | NO (deleted) | YES | **DANGER - NOT PUSHED (thankfully)** | | |
| ### Key Files at Risk | |
| **On `origin/dev` (PRESERVED):** | |
| ```text | |
| src/agents/ | |
| ├── analysis_agent.py # StatisticalAnalyzer wrapper | |
| ├── hypothesis_agent.py # Hypothesis generation | |
| ├── judge_agent.py # JudgeHandler wrapper | |
| ├── magentic_agents.py # Multi-agent definitions | |
| ├── report_agent.py # Report synthesis | |
| ├── search_agent.py # SearchHandler wrapper | |
| ├── state.py # Thread-safe state management | |
| └── tools.py # @ai_function decorated tools | |
| src/orchestrator_magentic.py # Multi-agent orchestrator | |
| src/utils/llm_factory.py # Centralized LLM client factory | |
| ``` | |
| **Deleted in refactor branch (would be lost if merged):** | |
| - All of the above | |
| --- | |
| ## 3. Target Architecture | |
| ```text | |
| ┌─────────────────────────────────────────────────────────────────┐ | |
| │ Microsoft Agent Framework (Orchestration Layer) │ | |
| │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ | |
| │ │ SearchAgent │→ │ JudgeAgent │→ │ ReportAgent │ │ | |
| │ │ (BaseAgent) │ │ (BaseAgent) │ │ (BaseAgent) │ │ | |
| │ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │ | |
| │ │ │ │ │ | |
| │ ▼ ▼ ▼ │ | |
| │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ | |
| │ │ pydantic-ai │ │ pydantic-ai │ │ pydantic-ai │ │ | |
| │ │ Agent() │ │ Agent() │ │ Agent() │ │ | |
| │ │ output_type= │ │ output_type= │ │ output_type= │ │ | |
| │ │ SearchResult │ │ JudgeAssess │ │ Report │ │ | |
| │ └──────────────┘ └──────────────┘ └──────────────┘ │ | |
| └─────────────────────────────────────────────────────────────────┘ | |
| ``` | |
| **Why this architecture:** | |
| 1. **Agent Framework** handles: workflow coordination, state passing, middleware, observability | |
| 2. **pydantic-ai** handles: type-safe LLM calls within each agent | |
| --- | |
| ## 4. CRITICAL: Naming Confusion Clarification | |
| > **Senior Agent Review Finding:** The codebase uses "magentic" in file names (e.g., `orchestrator_magentic.py`, `magentic_agents.py`) but this is **NOT** the `magentic` PyPI package by Jacky Liang. It's Microsoft Agent Framework (`agent-framework-core`). | |
| **The naming confusion:** | |
| - `magentic` (PyPI package): A different library for structured LLM outputs | |
| - "Magentic" (in our codebase): Our internal name for Microsoft Agent Framework integration | |
| - `agent-framework-core` (PyPI package): Microsoft's actual multi-agent orchestration framework | |
| **Recommended future action:** Rename `orchestrator_magentic.py` → `orchestrator_advanced.py` to eliminate confusion. | |
| --- | |
| ## 5. What the Refactor DID Get Right | |
| The refactor branch (`feat/pubmed-fulltext`) has some valuable improvements: | |
| 1. **`judges.py` unified `get_model()`** - Supports OpenAI, Anthropic, AND HuggingFace via pydantic-ai | |
| 2. **HuggingFace free tier support** - `HuggingFaceModel` integration | |
| 3. **Test fix** - Properly mocks `HuggingFaceModel` class | |
| 4. **Removed broken magentic optional dependency** from pyproject.toml (this was correct - the old `magentic` package is different from Microsoft Agent Framework) | |
| **What it got WRONG:** | |
| 1. Deleted `src/agents/` entirely instead of refactoring them | |
| 2. Deleted `src/orchestrator_magentic.py` instead of fixing it | |
| 3. Conflated "magentic" (old package) with "Microsoft Agent Framework" (current framework) | |
| --- | |
| ## 6. Options for Path Forward | |
| ### Option A: Abandon Refactor, Start Fresh | |
| - Close PR #41 | |
| - Delete `feat/pubmed-fulltext` and `refactor/pydantic-unification` branches | |
| - Reset local `dev` to match `origin/dev` | |
| - Cherry-pick ONLY the good parts (judges.py improvements, HF support) | |
| - **Pros:** Clean, safe | |
| - **Cons:** Lose some work, need to redo carefully | |
| ### Option B: Cherry-Pick Good Parts to origin/dev | |
| - Do NOT merge PR #41 | |
| - Create new branch from `origin/dev` | |
| - Cherry-pick specific commits/changes that improve pydantic-ai usage | |
| - Keep agent framework code intact | |
| - **Pros:** Preserves both, surgical | |
| - **Cons:** Requires careful file-by-file review | |
| ### Option C: Revert Deletions in Refactor Branch | |
| - On `feat/pubmed-fulltext`, restore deleted agent files from `origin/dev` | |
| - Keep the pydantic-ai improvements | |
| - Merge THAT to dev | |
| - **Pros:** Gets both | |
| - **Cons:** Complex git operations, risk of conflicts | |
| --- | |
| ## 7. Recommended Action: Option B (Cherry-Pick) | |
| **Step-by-step:** | |
| 1. **Close PR #41** (do not merge) | |
| 2. **Delete redundant branches:** | |
| - `refactor/pydantic-unification` (local) | |
| - Reset local `dev` to `origin/dev` | |
| 3. **Create new branch from origin/dev:** | |
| ```bash | |
| git checkout -b feat/pydantic-ai-improvements origin/dev | |
| ``` | |
| 4. **Cherry-pick or manually port these improvements:** | |
| - `src/agent_factory/judges.py` - the unified `get_model()` function | |
| - `examples/free_tier_demo.py` - HuggingFace demo | |
| - Test improvements | |
| 5. **Do NOT delete any agent framework files** | |
| 6. **Create PR for review** | |
| --- | |
| ## 8. Files to Cherry-Pick (Safe Improvements) | |
| | File | What Changed | Safe to Port? | | |
| |------|-------------|---------------| | |
| | `src/agent_factory/judges.py` | Added `HuggingFaceModel` support in `get_model()` | YES | | |
| | `examples/free_tier_demo.py` | New demo for HF inference | YES | | |
| | `tests/unit/agent_factory/test_judges.py` | Fixed HF model mocking | YES | | |
| | `pyproject.toml` | Removed old `magentic` optional dep | MAYBE (review carefully) | | |
| --- | |
| ## 9. Questions to Answer Before Proceeding | |
| 1. **For the hackathon**: Do we need full multi-agent orchestration, or is single-agent sufficient? | |
| 2. **For DeepBoner mainline**: Is the plan to use Microsoft Agent Framework for orchestration? | |
| 3. **Timeline**: How much time do we have to get this right? | |
| --- | |
| ## 10. Immediate Actions (DO NOW) | |
| - [ ] **DO NOT merge PR #41** | |
| - [ ] Close PR #41 with comment explaining the situation | |
| - [ ] Do not push local `dev` branch anywhere | |
| - [ ] Confirm HuggingFace Spaces is untouched (it is - verified) | |
| --- | |
| ## 11. Decision Log | |
| | Date | Decision | Rationale | | |
| |------|----------|-----------| | |
| | 2025-11-27 | Pause refactor merge | Discovered agent framework and pydantic-ai are complementary, not exclusive | | |
| | TBD | ? | Awaiting decision on path forward | | |