files/README.md

ICC Interac Manager — Scan Implementation

What This Fixes

The "Scanner maintenant" button now actually connects to Gmail, fetches Interac e-Transfer emails, parses them with AI (Groq/Mistral), and saves parsed transactions to the database — all with real-time progress via WebSocket.

---

File Map

icc-scan-impl/
├── .env.example                          ← Copy to .env and fill in API keys
│
├── shared/                               ← Shared types & constants
│   ├── types/
│   │   ├── scan.ts                       ← ScanPreset, ScanDateRange, resolveScanDates()
│   │   ├── transaction.ts                ← InteracTransaction schema
│   │   └── aiProvider.ts                 ← ProviderSlot, SwitcherEvent
│   └── constants/
│       └── branches.ts                   ← 48-branch email→city mapping
│
├── server/                               ← Backend (Express + SQLite)
│   ├── index.ts                          ← Server entry point
│   ├── db/
│   │   └── database.ts                   ← SQLite schema + operations
│   ├── middleware/
│   │   └── auth.ts                       ← JWT middleware
│   ├── routes/
│   │   ├── auth.ts                       ← Google OAuth routes
│   │   ├── scan.ts                       ← POST /api/scan/start
│   │   └── transactions.ts               ← GET /api/transactions
│   ├── services/
│   │   ├── gmailService.ts               ← Gmail API: query builder + email fetcher
│   │   ├── aiService.ts                  ← AI parsing: Groq + Mistral + auto-switcher
│   │   └── scanService.ts                ← Scan pipeline: Gmail → AI → DB
│   └── websocket/
│       └── setup.ts                      ← WebSocket server for progress events
│
└── web/                                  ← Frontend (React)
    ├── services/
    │   ├── api.ts                        ← HTTP client with auth
    │   └── websocket.ts                  ← WebSocket client
    ├── hooks/
    │   └── useEmailScan.ts               ← React hook for scan state + progress
    ├── components/scan/
    │   └── ScanControls.tsx              ← ★ THE scan button component
    └── pages/
        └── AuthCallback.tsx              ← Handles OAuth redirect

---

Setup Steps

1. Install Dependencies

In your packages/server/ directory:

npm install express cors helmet dotenv googleapis jsonwebtoken better-sqlite3 ws uuid p-limit zod
npm install -D typescript @types/express @types/cors @types/jsonwebtoken @types/better-sqlite3 @types/ws @types/uuid tsx

In your packages/web/ directory (most likely already installed):

npm install react-router-dom

2. Configure Google OAuth

1. Go to Google Cloud Console
2. Create a project (or use existing)
3. Enable the Gmail API
4. Create OAuth 2.0 Client ID (Web application)
5. Add authorized redirect URI: http://localhost:3001/api/auth/google/callback
6. Copy the Client ID and Client Secret

3. Get AI API Keys (Free)

Pick at least one:

• Groq (recommended primary): console.groq.com/keys — 30 RPM free
• Mistral (recommended fallback): console.mistral.ai/api-keys — free tier

4. Create .env File

cp .env.example .env
# Edit .env with your actual keys

5. Copy Files Into Your Project

Copy each file from this package into the corresponding location in your monorepo. The file paths in this package mirror the target paths in your project.

6. Add Vite Proxy (vite.config.ts)

Make sure your Vite dev server proxies API calls to the backend:

// packages/web/vite.config.ts
export default defineConfig({
  server: {
    proxy: {
      '/api': 'http://localhost:3001',
      '/ws': {
        target: 'ws://localhost:3001',
        ws: true,
      },
    },
  },
});

7. Add Route for Auth Callback

In your React Router config:

import AuthCallback from './pages/AuthCallback';

// Add this route:
<Route path="/auth/callback" element={<AuthCallback />} />

8. Wire Up ScanControls in Your Dashboard

Replace your current scan button section with the new ScanControls component:

import ScanControls from '@/components/scan/ScanControls';

function DashboardPage() {
  const { refetch } = useTransactions(); // your existing hook

  return (
    <div>
      <ScanControls
        onScanComplete={(result) => {
          // Refresh transaction table after scan completes
          refetch();
        }}
      />

      {/* ... rest of your dashboard ... */}
    </div>
  );
}

9. Start the Backend

cd packages/server
npx tsx src/index.ts

You should see:

🚀 ICC Interac Manager API running on http://localhost:3001
🔗 WebSocket available at ws://localhost:3001/ws
🔑 Google OAuth: http://localhost:3001/api/auth/google

10. Start the Frontend

cd packages/web
npm run dev

---

How It Works (Flow)

User clicks "Scanner maintenant"
         │
         ▼
┌─ ScanControls.tsx ──────────────────────────┐
│ 1. Reads selected preset (today/7days/custom)│
│ 2. Calls api.startScan({ preset, dates })   │
│ 3. Connects to WebSocket for progress events │
└─────────────────────┬───────────────────────┘
                      │ POST /api/scan/start
                      ▼
┌─ scan.ts (route) ──────────────────────────┐
│ 1. Validates preset + dates                 │
│ 2. Calls scanService.executeScan()          │
│ 3. Returns immediately (async background)   │
└─────────────────────┬──────────────────────┘
                      │
                      ▼
┌─ scanService.ts ───────────────────────────┐
│ PHASE 1: DISCOVER                          │
│  • Build Gmail query: from:notify@...      │
│  • Fetch all matching message IDs          │
│  • Deduplicate against existing DB records │
│                                            │
│ PHASE 2: PIPELINE (parallel)               │
│  • 10x concurrent Gmail fetches            │
│  • 5x concurrent AI parses                 │
│  • Auto-switch Groq↔Mistral on rate limit  │
│  • Batch INSERT 25 rows at a time          │
│  • Emit scan:progress via WebSocket        │
│                                            │
│ PHASE 3: COMPLETE                          │
│  • Flush remaining DB buffer               │
│  • Emit scan:completed                     │
│  • Update scan_logs table                  │
└────────────────────────────────────────────┘

---

WebSocket Events (Frontend receives these)

Event	Data	When
scan:started	{ jobId, totalEmails, newEmails, skipped }	Scan begins
scan:progress	{ processed, total, errored, latest, currentProvider }	Each email processed
scan:completed	{ jobId, summary: { found, parsed, skipped, errors } }	Scan finished
scan:error	{ jobId, error }	Scan failed
ai:switch	{ from, to, reason }	AI provider auto-switched
transaction:new	{ transaction }	New transaction saved

---

Troubleshooting

Issue	Fix
"User has no Gmail tokens"	User needs to log in via Google OAuth first: visit /api/auth/google
"No AI providers configured"	Set GROQ_API_KEY and/or MISTRAL_API_KEY in .env
"All AI providers rate-limited"	Wait 60s for cooldown, or add more API keys
Scan returns 0 emails	Check Gmail query — make sure the connected account actually has emails from notify@payments.interac.ca
WebSocket not connecting	Check Vite proxy config includes /ws → ws://localhost:3001
"Invalid token"	JWT expired — log in again via Google OAuth