Abhisingh-18/Jobins

Resource-Constrained Agentic Planning Loop

A ReAct agent that solves tasks with 3 tools under a hard budget of 10 LLM calls and $0.20 per task, running entirely on a free local model (Ollama, llama3.2:3b) with simulated token costs so the monetary enforcer is observably doing real work.

Quick start (single command)

docker compose up --build agent

This starts Ollama, pulls llama3.2:3b (one-time, ~2 GB), then runs the 5-task evaluation suite. Results land in traces/ (per-task JSON traces + test_results.md).

Run a single custom task:

docker compose run agent python main.py "your task here"

Running locally without Docker

pip install -r requirements.txt
ollama serve &
ollama pull llama3.2:3b
python main.py --all-tests          # or: python main.py "your task"

API keys / configuration

Copy .env.example to .env. No API key is required out of the box: the LLM is local (Ollama) and web search uses DuckDuckGo. Every setting (budget limits, model name, mock price, and any future paid SEARCH_API_KEY) is injected into the container via env_file: .env in docker-compose.yml — edit .env and re-run; nothing is baked into the image.

Architecture Overview

A single orchestrating loop (agent/loop.py) drives ReAct steps. Every LLM call goes through one chokepoint — OllamaClient.chat_json() — which is wrapped by the BudgetEnforcer (atomic check-then-record under a lock), so the budget cannot be bypassed from anywhere in the codebase. Tools implement a common Tool interface with a hard wall-clock timeout and typed error categories. All shared data flows through an explicit AgentState dataclass that doubles as the audit trail and the final report.

task ──> planning loop (THINK → ACT → OBSERVE → REFLECT)
              │ every LLM call            │ every tool call
              ▼                           ▼
        BudgetEnforcer ◄──────── Tool.execute() [hard timeout]
              │
              └─ BudgetExceededError ──> graceful partial report

Planning Loop

I chose ReAct because with only 10 LLM calls per task, every call must both reason and commit to an action — ReAct fuses the two into one structured response, whereas plan-then-execute or Tree-of-Thought architectures spend calls on planning artifacts that a 10-call budget cannot afford. Its biggest weakness is myopia: ReAct decides one step at a time with no global plan, so it can wander on multi-part tasks and burn budget on locally-reasonable but globally-suboptimal steps. I mitigate (not eliminate) this by injecting the remaining budget into every prompt and instructing the model to batch lookups and finalize early.

Schema Design

State is a typed AgentState dataclass: the task, a list of StepRecords (thought, action, observation, progress flag, replan flag + reason), a scratchpad holding the last useful observation (the fallback answer if the budget dies mid-task), and a status enum. The LLM never sees raw Python state: each turn it receives the message history plus a fresh system prompt with live budget numbers, and must reply with one JSON object (progress_check / thought / action / final_answer) — Ollama's format: json guarantees parseable output structurally, and a schema check guards the field level. Tools return a typed ToolResult (success, data, error, error_type) that is stringified for the LLM but kept structured in the trace. The final report is just AgentState + the budget ledger serialized — test_results.md is generated from it, not hand-written.

Prompt Strategy

The system prompt enforces three behaviors. (1) Tool use: the response format is a strict JSON contract; the tool catalog with exact argument hints is embedded, and rules forbid repeating an identical call. (2) Progress checking: every response must begin with a progress_check field assessing the previous observation — this piggybacks LLM-side reflection onto the next THINK call so reflection costs zero extra LLM calls; deterministic rules (tool failure, empty results, repeated action) catch no-progress for free before the LLM is even consulted. (3) Budget awareness: remaining calls and dollars are re-injected every single turn with the instruction that a partial answer now beats a perfect answer never; when replanning is forced, the notice repeats the remaining budget so the model can decide to finalize instead of exploring.

Failure Modes

Three concrete modes observed during testing:

1. ^ as exponentiation — a two-stage failure. The model persistently wrote 1000 * (1 + 0.05)^10. Python parses ^ as XOR, so the safe-AST calculator first rejected it outright, and the agent burned three replans (and eventually the whole cost budget) searching the web for "compound interest formula without binop" instead of fixing the syntax. My first fix — mapping the XOR AST node to pow — introduced a subtler bug: ^ inherits XOR's precedence, which binds looser than *, so the expression silently evaluated as (1000 * 1.05) ** 10 = 1.63e+30 and the agent reported "$1.63 billion" while itself noting it "seems implausible." Final fix: textually rewrite ^ to ** before parsing, which restores correct power precedence. Verified: the same expression now returns 1628.89.

2. Evidence-free bail-out. On the budget-drain task (15 countries), the model once returned a final answer full of nulls at step 1 without running a single search — the "partial answer beats no answer" prompt rule backfired. Fix shipped: the orchestrator rejects a final answer produced before any successful tool call (once), forcing the agent to gather real evidence first.

3. Off-by-one reasoning survives correct tooling. On the Fibonacci task the agent recovered well from a crashing recursive attempt (replan → iterative code), but its iterative loop computed F26 (121393) instead of F25 (75025) and no tool can catch a semantically wrong but syntactically fine program. Budget enforcement bounds the cost of such errors; it cannot detect them.

Design note on the cost cap: token counts are only known after a call returns, so the final LLM call can overshoot $0.20 slightly (observed: $0.2365). Execution still halts immediately and the overshoot is visible in the ledger — pre-metering would require a tokenizer-accurate cost oracle before each call.

Future Work

The loop detector only catches literally identical (tool, args) repeats. A naive-but-different paraphrase loop — searching "exact grains of sand count" then "precise number sand grains" forever — is currently broken up only by the prompt rules and budget pressure, not by code. With more time I would add semantic action-similarity detection (embed the last N queries with a tiny local embedding model and flag cosine-similar repeats) so paraphrased loops trigger the same hard replan path as identical ones.

Bonus: live tracking dashboard

A small Flask + Next.js control center for watching the agent work (dashboard/). It shows Ollama/Docker/agent-process status, per-run budget meters (calls + cost), replanning counts, expandable step-by-step traces from traces/*.json, and a live log console; runs can be launched from the UI (single task or the full suite — one at a time).

# terminal 1 — API (port 8788)
python dashboard/api/dashboard_api.py
# terminal 2 — UI (port 3000)
cd dashboard/web && npm install && npm run dev

Repository layout

main.py            # entrypoint (single task or --all-tests)
run_tests.py       # 5-task suite -> test_results.md + traces/*.json
agent/
  budget.py        # BudgetEnforcer: atomic hard limits + ledger
  llm.py           # budget-gated Ollama client (the ONLY path to the model)
  loop.py          # ReAct loop, tiered reflection, loop detector, graceful exit
  prompts.py       # system prompt + replan notice
  state.py         # AgentState / StepRecord schema
  tools/           # web_search, code_exec, calculator (+ base with timeouts)
decisions.md       # engineering trade-offs
test_results.md    # generated evaluation report (5 tasks, 2 adversarial)