docs/report.md

# Project Report

## Objective

<<<<<<< Updated upstream
Built a AI-powered business workflow automation prototype for Shopify customer support.
=======
Build an AI-powered business workflow automation prototype for Shopify customer support.
>>>>>>> Stashed changes

## Architecture

The platform exposes a FastAPI endpoint that receives customer tickets and runs a multi-agent workflow. The current implementation uses local deterministic fallbacks so the project works without paid API keys, while the `models` package contains clear extension points for Groq and Gemini clients.

<<<<<<< Updated upstream

=======
>>>>>>> Stashed changes
## Workflow

1. Customer submits a ticket through `POST /api/tickets`.
2. Supervisor decides which agents should run.
3. Intent agent classifies the support intent and priority.
4. Order agent loads matching Shopify-style order data.
5. Policy agent retrieves relevant policy documents.
6. Reasoning agent creates a grounded customer response.
7. Validation agent checks whether policy context was used.
8. Escalation service routes low-confidence or unverifiable cases to a human.

## Data

Policy documents live in `data/policies`. Sample orders live in `data/orders/orders.json`.

## Production Path

Replace local model fallbacks with real Groq and Gemini SDK calls, replace sample order JSON with Shopify Admin API calls, persist tickets in a database, and deploy the FastAPI service behind Nginx using the Docker files in `deployment`.
<<<<<<< Updated upstream

## Part 1 — AI Research & Evaluation

Goal: research and compare candidate AI platforms and tools to implement the Shopify/eCommerce Automation use case.

### Tools evaluated
- OpenAI (GPT family)
- Google Gemini
- LangChain (framework)
- Pinecone (vector database)

### Comparison table

| Tool / Platform | Capabilities | Pricing | Scalability | Ease of Integration | Limitations | Best Use Cases |
|---|---|---|---|---|---|---|
| OpenAI | High-quality text generation, embeddings, classification, dialogue | Pay-as-you-go token pricing; affordable for prototypes, higher at scale | Managed, auto-scaled | Easy, rich SDKs and REST APIs | Cost at scale, reliance on third-party API, data governance | Chat, support responses, intent classification, RAG |
| Google Gemini | Strong reasoning, multimodal support, conversation quality | Tiered, enterprise pricing for higher throughput | GCP-managed, enterprise SLA | Good if already in Google Cloud; may require IAM setup | GCP lock-in, region restrictions, enterprise onboarding | Complex reasoning, multimodal workflows, enterprise deployments |
| LangChain | Orchestration for LLMs, agents, chains, memory, retrievers | Free OSS; costs from underlying models and infra | Scales with chosen runtime and backend | High for Python, many connectors | Not a model provider, architecture overhead | RAG systems, multi-step agent workflows, orchestration |
| Chroma | Local/single-node vector search, embeddings storage, retrieval | Free/open-source; local compute costs apply | Good for prototypes and small deployments | Easy with Python SDK and LangChain | Less managed than hosted services, scales with local infrastructure | Prototype RAG, semantic search, retrieval, embeddings index |

### Research findings
- OpenAI and Gemini are strongest for general-purpose LLM tasks; choice depends on cost, compliance, and enterprise requirements.
- LangChain is ideal for orchestrating a multi-agent workflow and integrating models with retrieval.
- Chroma is the preferred vector store for this POC and can also support smaller-scale production workflows.

## Part 2 — Build a Prototype / POC

The prototype demonstrates a Shopify customer support workflow using a multi-agent Python architecture.

### Prototype goals
- Automate ticket classification and response generation
- Retrieve relevant policy and order data
- Validate outputs and escalate uncertain cases
- Support a modular path to integrate real LLMs and Shopify APIs

### Prototype implementation
- `app/main.py` provides the API entrypoint.
- `app/agents/` contains modular agent classes:
  - `intent_agent.py` for classifying support type and priority
  - `order_agent.py` for matching orders and order metadata
  - `policy_agent.py` for retrieving policy documents
  - `reasoning_agent.py` for building grounded responses
  - `validation_agent.py` for verifying output quality
  - `supervisor_agent.py` to orchestrate the workflow
- `rag/` contains retrieval utilities, embeddings, and vector store logic.
- `models/` contains the model router and adapters for Gemini and Groq.
- `services/` contains Shopify integration, ticket handling, and escalation logic.

### Prototype workflow
1. A ticket arrives via `POST /api/tickets`.
2. The supervisor agent routes the request to intent, order, policy, reasoning, and validation agents.
3. The intent agent classifies topic and priority.
4. The order agent loads matching sample order data.
5. The policy agent retrieves policy documents from `data/policies`.
6. The reasoning agent generates a response grounded in policy and order context.
7. The validation agent checks that the response uses policy evidence.
8. Low-confidence or unverifiable cases are flagged for human escalation.

### Current POC status
- Prototype is functional with local fallbacks and sample data.
- The repo includes project structure and key components for a production-ready path.
- The POC is designed to be extended with managed models and Shopify API connectivity.

## Part 3 — Recommendation Report

### Recommended architecture (high level)
- Frontend: lightweight UI (Streamlit or React) for agents and support staff.
- API: FastAPI service exposing ticket endpoints and orchestrating agents.
- Orchestrator: Supervisor Agent coordinates Intent, Order, Policy, Reasoning, and Validation agents.
- RAG layer: Embeddings + vector store (Chroma for prototype) and a Retriever.
- Models: Managed LLM provider (OpenAI or Google Gemini) with a router for fallback logic.
- Persistence: Postgres for tickets, Redis for caching, object storage for logs and artifacts.
- Observability: Prometheus/Grafana, structured logs, and Sentry.

### Why these tools/models
- OpenAI/Gemini: high-quality text generation, classification, and embeddings.
- LangChain: provides reusable orchestration patterns and agent-based workflows.
- Chroma: local and open-source vector retrieval that matches the current POC implementation.
- FastAPI & Uvicorn: lightweight, async-ready API hosting.

### Estimated infrastructure cost (monthly)
- Prototype: $0–$50 using local execution or low-cost cloud VMs.
- Development/staging: $200–$1,500 depending on model usage.
- Production (100k queries/month): $2k–$10k+ driven primarily by LLM token costs.

### Cost drivers
- Model token consumption and chosen model tier
- Vector store size, query volume, and index updates
- Logging, storage, and human escalation overhead

### Risks and limitations
- Cost: LLM usage can be expensive; add caching and route low-risk requests to cheaper models.
- Hallucinations: use RAG grounding, validation, and human escalation.
- Data privacy: avoid sending sensitive PII to third-party models without controls.
- Latency and availability: implement retries, fallbacks, and monitoring.

### Scaling considerations
- Use model routing, caching, and prompt optimization.
- Batch embedding workloads and incrementally index new documents.
- Scale API workers horizontally and use autoscaling for LLM gateways.
- Add metrics, tracing, and SLOs for response quality and latency.

### Security and compliance
- Keep API keys in environment variables or a secret manager.
- Protect API access with authentication and role-based permissions.
- Redact or minimize PII in model requests.

### Deliverables and next steps
- GitHub repository with documentation and POC.
- Demo video or walkthrough of the support workflow.
- Screenshots of ticket and escalation flow.
- Harden the prototype with managed LLMs, database persistence, and monitoring.
- Pilot with limited traffic and validate cost, latency, and accuracy.


=======
>>>>>>> Stashed changes