files/README.md

# ICC Interac Manager — Scan Implementation

## What This Fixes

The "Scanner maintenant" button now actually **connects to Gmail**, **fetches Interac e-Transfer emails**, **parses them with AI** (Groq/Mistral), and **saves parsed transactions** to the database — all with real-time progress via WebSocket.

---

## File Map

```
icc-scan-impl/
├── .env.example                          ← Copy to .env and fill in API keys
│
├── shared/                               ← Shared types & constants
│   ├── types/
│   │   ├── scan.ts                       ← ScanPreset, ScanDateRange, resolveScanDates()
│   │   ├── transaction.ts                ← InteracTransaction schema
│   │   └── aiProvider.ts                 ← ProviderSlot, SwitcherEvent
│   └── constants/
│       └── branches.ts                   ← 48-branch email→city mapping
│
├── server/                               ← Backend (Express + SQLite)
│   ├── index.ts                          ← Server entry point
│   ├── db/
│   │   └── database.ts                   ← SQLite schema + operations
│   ├── middleware/
│   │   └── auth.ts                       ← JWT middleware
│   ├── routes/
│   │   ├── auth.ts                       ← Google OAuth routes
│   │   ├── scan.ts                       ← POST /api/scan/start
│   │   └── transactions.ts               ← GET /api/transactions
│   ├── services/
│   │   ├── gmailService.ts               ← Gmail API: query builder + email fetcher
│   │   ├── aiService.ts                  ← AI parsing: Groq + Mistral + auto-switcher
│   │   └── scanService.ts                ← Scan pipeline: Gmail → AI → DB
│   └── websocket/
│       └── setup.ts                      ← WebSocket server for progress events
│
└── web/                                  ← Frontend (React)
    ├── services/
    │   ├── api.ts                        ← HTTP client with auth
    │   └── websocket.ts                  ← WebSocket client
    ├── hooks/
    │   └── useEmailScan.ts               ← React hook for scan state + progress
    ├── components/scan/
    │   └── ScanControls.tsx              ← ★ THE scan button component
    └── pages/
        └── AuthCallback.tsx              ← Handles OAuth redirect
```

---

## Setup Steps

### 1. Install Dependencies

In your `packages/server/` directory:

```bash
npm install express cors helmet dotenv googleapis jsonwebtoken better-sqlite3 ws uuid p-limit zod
npm install -D typescript @types/express @types/cors @types/jsonwebtoken @types/better-sqlite3 @types/ws @types/uuid tsx
```

In your `packages/web/` directory (most likely already installed):

```bash
npm install react-router-dom
```

### 2. Configure Google OAuth

1. Go to [Google Cloud Console](https://console.cloud.google.com/apis/credentials)
2. Create a project (or use existing)
3. Enable the **Gmail API**
4. Create **OAuth 2.0 Client ID** (Web application)
5. Add authorized redirect URI: `http://localhost:3001/api/auth/google/callback`
6. Copy the Client ID and Client Secret

### 3. Get AI API Keys (Free)

Pick at least one:

- **Groq** (recommended primary): [console.groq.com/keys](https://console.groq.com/keys) — 30 RPM free
- **Mistral** (recommended fallback): [console.mistral.ai/api-keys](https://console.mistral.ai/api-keys) — free tier

### 4. Create .env File

```bash
cp .env.example .env
# Edit .env with your actual keys
```

### 5. Copy Files Into Your Project

Copy each file from this package into the corresponding location in your monorepo.
The file paths in this package mirror the target paths in your project.

### 6. Add Vite Proxy (vite.config.ts)

Make sure your Vite dev server proxies API calls to the backend:

```typescript
// packages/web/vite.config.ts
export default defineConfig({
  server: {
    proxy: {
      '/api': 'http://localhost:3001',
      '/ws': {
        target: 'ws://localhost:3001',
        ws: true,
      },
    },
  },
});
```

### 7. Add Route for Auth Callback

In your React Router config:

```tsx
import AuthCallback from './pages/AuthCallback';

// Add this route:
<Route path="/auth/callback" element={<AuthCallback />} />
```

### 8. Wire Up ScanControls in Your Dashboard

Replace your current scan button section with the new `ScanControls` component:

```tsx
import ScanControls from '@/components/scan/ScanControls';

function DashboardPage() {
  const { refetch } = useTransactions(); // your existing hook

  return (
    <div>
      <ScanControls
        onScanComplete={(result) => {
          // Refresh transaction table after scan completes
          refetch();
        }}
      />

      {/* ... rest of your dashboard ... */}
    </div>
  );
}
```

### 9. Start the Backend

```bash
cd packages/server
npx tsx src/index.ts
```

You should see:

```
🚀 ICC Interac Manager API running on http://localhost:3001
🔗 WebSocket available at ws://localhost:3001/ws
🔑 Google OAuth: http://localhost:3001/api/auth/google
```

### 10. Start the Frontend

```bash
cd packages/web
npm run dev
```

---

## How It Works (Flow)

```
User clicks "Scanner maintenant"
         │
         ▼
┌─ ScanControls.tsx ──────────────────────────┐
│ 1. Reads selected preset (today/7days/custom)│
│ 2. Calls api.startScan({ preset, dates })   │
│ 3. Connects to WebSocket for progress events │
└─────────────────────┬───────────────────────┘
                      │ POST /api/scan/start
                      ▼
┌─ scan.ts (route) ──────────────────────────┐
│ 1. Validates preset + dates                 │
│ 2. Calls scanService.executeScan()          │
│ 3. Returns immediately (async background)   │
└─────────────────────┬──────────────────────┘
                      │
                      ▼
┌─ scanService.ts ───────────────────────────┐
│ PHASE 1: DISCOVER                          │
│  • Build Gmail query: from:notify@...      │
│  • Fetch all matching message IDs          │
│  • Deduplicate against existing DB records │
│                                            │
│ PHASE 2: PIPELINE (parallel)               │
│  • 10x concurrent Gmail fetches            │
│  • 5x concurrent AI parses                 │
│  • Auto-switch Groq↔Mistral on rate limit  │
│  • Batch INSERT 25 rows at a time          │
│  • Emit scan:progress via WebSocket        │
│                                            │
│ PHASE 3: COMPLETE                          │
│  • Flush remaining DB buffer               │
│  • Emit scan:completed                     │
│  • Update scan_logs table                  │
└────────────────────────────────────────────┘
```

---

## WebSocket Events (Frontend receives these)

| Event | Data | When |
|-------|------|------|
| `scan:started` | `{ jobId, totalEmails, newEmails, skipped }` | Scan begins |
| `scan:progress` | `{ processed, total, errored, latest, currentProvider }` | Each email processed |
| `scan:completed` | `{ jobId, summary: { found, parsed, skipped, errors } }` | Scan finished |
| `scan:error` | `{ jobId, error }` | Scan failed |
| `ai:switch` | `{ from, to, reason }` | AI provider auto-switched |
| `transaction:new` | `{ transaction }` | New transaction saved |

---

## Troubleshooting

| Issue | Fix |
|-------|-----|
| "User has no Gmail tokens" | User needs to log in via Google OAuth first: visit `/api/auth/google` |
| "No AI providers configured" | Set `GROQ_API_KEY` and/or `MISTRAL_API_KEY` in `.env` |
| "All AI providers rate-limited" | Wait 60s for cooldown, or add more API keys |
| Scan returns 0 emails | Check Gmail query — make sure the connected account actually has emails from `notify@payments.interac.ca` |
| WebSocket not connecting | Check Vite proxy config includes `/ws` → `ws://localhost:3001` |
| "Invalid token" | JWT expired — log in again via Google OAuth |