docs/report.md

Project Report

Objective

<<<<<<< Updated upstream Built a AI-powered business workflow automation prototype for Shopify customer support.

Build an AI-powered business workflow automation prototype for Shopify customer support.

Stashed changes

Architecture

The platform exposes a FastAPI endpoint that receives customer tickets and runs a multi-agent workflow. The current implementation uses local deterministic fallbacks so the project works without paid API keys, while the models package contains clear extension points for Groq and Gemini clients.

<<<<<<< Updated upstream

=======

Stashed changes

Workflow

1. Customer submits a ticket through POST /api/tickets.
2. Supervisor decides which agents should run.
3. Intent agent classifies the support intent and priority.
4. Order agent loads matching Shopify-style order data.
5. Policy agent retrieves relevant policy documents.
6. Reasoning agent creates a grounded customer response.
7. Validation agent checks whether policy context was used.
8. Escalation service routes low-confidence or unverifiable cases to a human.

Data

Policy documents live in data/policies. Sample orders live in data/orders/orders.json.

Production Path

Replace local model fallbacks with real Groq and Gemini SDK calls, replace sample order JSON with Shopify Admin API calls, persist tickets in a database, and deploy the FastAPI service behind Nginx using the Docker files in deployment. <<<<<<< Updated upstream

Part 1 — AI Research & Evaluation

Goal: research and compare candidate AI platforms and tools to implement the Shopify/eCommerce Automation use case.

Tools evaluated

• OpenAI (GPT family)
• Google Gemini
• LangChain (framework)
• Pinecone (vector database)

Comparison table

Tool / Platform	Capabilities	Pricing	Scalability	Ease of Integration	Limitations	Best Use Cases
OpenAI	High-quality text generation, embeddings, classification, dialogue	Pay-as-you-go token pricing; affordable for prototypes, higher at scale	Managed, auto-scaled	Easy, rich SDKs and REST APIs	Cost at scale, reliance on third-party API, data governance	Chat, support responses, intent classification, RAG
Google Gemini	Strong reasoning, multimodal support, conversation quality	Tiered, enterprise pricing for higher throughput	GCP-managed, enterprise SLA	Good if already in Google Cloud; may require IAM setup	GCP lock-in, region restrictions, enterprise onboarding	Complex reasoning, multimodal workflows, enterprise deployments
LangChain	Orchestration for LLMs, agents, chains, memory, retrievers	Free OSS; costs from underlying models and infra	Scales with chosen runtime and backend	High for Python, many connectors	Not a model provider, architecture overhead	RAG systems, multi-step agent workflows, orchestration
Chroma	Local/single-node vector search, embeddings storage, retrieval	Free/open-source; local compute costs apply	Good for prototypes and small deployments	Easy with Python SDK and LangChain	Less managed than hosted services, scales with local infrastructure	Prototype RAG, semantic search, retrieval, embeddings index

Research findings

• OpenAI and Gemini are strongest for general-purpose LLM tasks; choice depends on cost, compliance, and enterprise requirements.
• LangChain is ideal for orchestrating a multi-agent workflow and integrating models with retrieval.
• Chroma is the preferred vector store for this POC and can also support smaller-scale production workflows.

Part 2 — Build a Prototype / POC

The prototype demonstrates a Shopify customer support workflow using a multi-agent Python architecture.

Prototype goals

• Automate ticket classification and response generation
• Retrieve relevant policy and order data
• Validate outputs and escalate uncertain cases
• Support a modular path to integrate real LLMs and Shopify APIs

Prototype implementation

• app/main.py provides the API entrypoint.
• app/agents/ contains modular agent classes:
  • intent_agent.py for classifying support type and priority
  • order_agent.py for matching orders and order metadata
  • policy_agent.py for retrieving policy documents
  • reasoning_agent.py for building grounded responses
  • validation_agent.py for verifying output quality
  • supervisor_agent.py to orchestrate the workflow
• rag/ contains retrieval utilities, embeddings, and vector store logic.
• models/ contains the model router and adapters for Gemini and Groq.
• services/ contains Shopify integration, ticket handling, and escalation logic.

Prototype workflow

1. A ticket arrives via POST /api/tickets.
2. The supervisor agent routes the request to intent, order, policy, reasoning, and validation agents.
3. The intent agent classifies topic and priority.
4. The order agent loads matching sample order data.
5. The policy agent retrieves policy documents from data/policies.
6. The reasoning agent generates a response grounded in policy and order context.
7. The validation agent checks that the response uses policy evidence.
8. Low-confidence or unverifiable cases are flagged for human escalation.

Current POC status

• Prototype is functional with local fallbacks and sample data.
• The repo includes project structure and key components for a production-ready path.
• The POC is designed to be extended with managed models and Shopify API connectivity.

Part 3 — Recommendation Report

Recommended architecture (high level)

• Frontend: lightweight UI (Streamlit or React) for agents and support staff.
• API: FastAPI service exposing ticket endpoints and orchestrating agents.
• Orchestrator: Supervisor Agent coordinates Intent, Order, Policy, Reasoning, and Validation agents.
• RAG layer: Embeddings + vector store (Chroma for prototype) and a Retriever.
• Models: Managed LLM provider (OpenAI or Google Gemini) with a router for fallback logic.
• Persistence: Postgres for tickets, Redis for caching, object storage for logs and artifacts.
• Observability: Prometheus/Grafana, structured logs, and Sentry.

Why these tools/models

• OpenAI/Gemini: high-quality text generation, classification, and embeddings.
• LangChain: provides reusable orchestration patterns and agent-based workflows.
• Chroma: local and open-source vector retrieval that matches the current POC implementation.
• FastAPI & Uvicorn: lightweight, async-ready API hosting.

Estimated infrastructure cost (monthly)

• Prototype: $0–$50 using local execution or low-cost cloud VMs.
• Development/staging: $200–$1,500 depending on model usage.
• Production (100k queries/month): $2k–$10k+ driven primarily by LLM token costs.

Cost drivers

• Model token consumption and chosen model tier
• Vector store size, query volume, and index updates
• Logging, storage, and human escalation overhead

Risks and limitations

• Cost: LLM usage can be expensive; add caching and route low-risk requests to cheaper models.
• Hallucinations: use RAG grounding, validation, and human escalation.
• Data privacy: avoid sending sensitive PII to third-party models without controls.
• Latency and availability: implement retries, fallbacks, and monitoring.

Scaling considerations

• Use model routing, caching, and prompt optimization.
• Batch embedding workloads and incrementally index new documents.
• Scale API workers horizontally and use autoscaling for LLM gateways.
• Add metrics, tracing, and SLOs for response quality and latency.

Security and compliance

• Keep API keys in environment variables or a secret manager.
• Protect API access with authentication and role-based permissions.
• Redact or minimize PII in model requests.

Deliverables and next steps

• GitHub repository with documentation and POC.
• Demo video or walkthrough of the support workflow.
• Screenshots of ticket and escalation flow.
• Harden the prototype with managed LLMs, database persistence, and monitoring.
• Pilot with limited traffic and validate cost, latency, and accuracy.

=======

Stashed changes