Spaces:
Running
Running
File size: 49,106 Bytes
149698e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 | # ICC Interac Manager — Complete Technical Flow
**Every technology, every AI model, every data transformation — fully detailed**
**Last updated:** February 2026
---
## 1. System Architecture — Full Technology Map
```
┌─────────────────────────────────────────────────────────────────────────┐
│ USER'S BROWSER │
│ │
│ ┌────────────────────────────────────────────────────────────────────┐ │
│ │ VITE 6 + REACT 19 SPA │ │
│ │ TypeScript 5 · Tailwind CSS 4 · shadcn/ui │ │
│ │ │ │
│ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌────────┐ │ │
│ │ │ Login │ │Dashboard │ │ Scan │ │ Settings │ │Reports │ │ │
│ │ │ Page │ │ Page │ │ Page │ │ Page │ │ Page │ │ │
│ │ └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘ └───┬────┘ │ │
│ │ │ │ │ │ │ │ │
│ │ ┌────┴─────────────┴────────────┴─────────────┴────────────┴────┐│ │
│ │ │ State Management Layer ││ │
│ │ │ Zustand (global) · TanStack Query v5 (server/cache) ││ │
│ │ │ React Hook Form + Zod (forms) · Socket.io-client (realtime) ││ │
│ │ └───────────────────────────┬────────────────────────────────────┘│ │
│ └──────────────────────────────┼─────────────────────────────────────┘ │
│ │ HTTP + WebSocket │
│ ┌──────────────────┼──────────────────┐ │
│ │ Vite Dev Proxy (port 5173) │ │
│ │ /api/* → localhost:3001 │ │
│ │ /ws → ws://localhost:3001 │ │
│ └──────────────────┼──────────────────┘ │
└─────────────────────────────────┼───────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────┐
│ EXPRESS.JS 5 BACKEND (port 3001) │
│ Node.js 20 LTS · TypeScript 5 │
│ │
│ ┌──────────────────────────────────────────────────────────────────┐ │
│ │ MIDDLEWARE LAYER │ │
│ │ JWT Auth · CORS · express-rate-limit · Helmet CSP · Pino logger │ │
│ └──────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Auth │ │ Scan │ │ Txns │ │ Receipts │ │ Settings │ │
│ │ Routes │ │ Routes │ │ Routes │ │ Routes │ │ Routes │ │
│ └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘ │
│ │ │ │ │ │ │
│ ┌────┴─────────────┴────────────┴─────────────┴────────────┴────┐ │
│ │ SERVICE LAYER │ │
│ │ │ │
│ │ ┌─────────────┐ ┌──────────────┐ ┌───────────────────────┐│ │
│ │ │ Gmail │ │ Scan Engine │ │ AI Provider Pool ││ │
│ │ │ Service │ │ (pipeline) │ │ (auto-switcher) ││ │
│ │ │ │ │ │ │ ││ │
│ │ │ googleapis │ │ Fetch→Parse │ │ Groq ←→ Mistral ││ │
│ │ │ OR imapflow │ │ →Route→Save │ │ (9 free model slots) ││ │
│ │ └─────────────┘ └──────────────┘ └───────────────────────┘│ │
│ │ │ │
│ │ ┌─────────────┐ ┌──────────────┐ ┌───────────────────────┐│ │
│ │ │ Routing │ │ PDF Service │ │ Export Service ││ │
│ │ │ Service │ │ PDFKit / │ │ SheetJS (xlsx) ││ │
│ │ │ branch map │ │ react-pdf │ │ CSV (built-in) ││ │
│ │ └─────────────┘ └──────────────┘ └───────────────────────┘│ │
│ └────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────────────────────┐ │
│ │ DATA LAYER │ │
│ │ Drizzle ORM · PostgreSQL 16 · Redis + BullMQ (job queue) │ │
│ └──────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────────────────────┐ │
│ │ WEBSOCKET LAYER (Socket.io) │ │
│ │ scan:progress · scan:completed · transaction:new · ai:switcher │ │
│ └──────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────┘
│ │ │
▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────────────┐
│ PostgreSQL 16│ │ Redis 7 │ │ External AI APIs │
│ │ │ │ │ │
│ users │ │ BullMQ jobs │ │ api.groq.com │
│ transactions │ │ scan queue │ │ api.mistral.ai │
│ branch_config│ │ rate limiter │ │ (free tier, $0) │
│ scan_logs │ │ │ │ │
│ ai_settings │ │ │ │ [Optional paid:] │
│ ai_switcher │ │ │ │ api.anthropic.com │
│ _logs │ │ │ │ api.openai.com │
└──────────────┘ └──────────────┘ └──────────────────────┘
```
---
## 2. Complete Scan Pipeline — Step by Step
This is the exact sequence of events when a user clicks **"Scanner maintenant"**.
### Phase 0: User Initiates Scan
```
USER ACTION TECHNOLOGY INVOLVED
─────────────────────────────────── ──────────────────────────────────
1. User clicks "Scanner aujourd'hui" React onClick handler
2. Frontend sends POST /api/scan/start Axios / fetch (TanStack Query mutation)
Body: { preset: "today" }
3. Backend receives request Express.js route handler
4. JWT middleware validates token jsonwebtoken (JWT verify)
5. Creates BullMQ job in Redis BullMQ + Redis
6. Returns { jobId: "scan_abc123" } Express response
7. Frontend opens WebSocket Socket.io-client
Listens for scan:progress events
```
### Phase 1: Email Discovery (Gmail API)
```
STEP ACTION TECHNOLOGY TIME
─────── ─────────────────────────────── ───────────────────── ──────
1.1 Resolve date range: resolveScanDates() <1ms
"today" → midnight→now (shared util)
1.2 Build Gmail search query: buildGmailQuery() <1ms
"from:notify@payments. (server util)
interac.ca after:2026/2/22
before:2026/2/24"
1.3 Call Gmail API: list messages googleapis npm 200ms
GET gmail/v1/users/me/messages (google-auth-library
?q={query}&maxResults=500 + googleapis)
OAuth token from DB: Drizzle ORM → PG
users.access_token (AES-256 crypto.decipher
encrypted) → decrypt
If token expired: google-auth-library
auto-refresh with OAuth2Client
users.refresh_token .refreshAccessToken()
1.4 Receive message ID list: Gmail API response —
["msg_001", "msg_002", ...]
(just IDs, not full emails)
1.5 Deduplication check: Drizzle ORM → PG 50ms
SELECT email_id FROM SQL query
transactions WHERE (parameterized,
email_id IN (...) indexed)
1.6 Filter: skip existing IDs JavaScript Set <1ms
Result: newIds[] (emails
to process)
1.7 WebSocket emit: Socket.io <1ms
scan:started {
jobId, totalEmails,
newEmails, skipped,
dateRange
}
PHASE 1 TOTAL: ~300ms for discovery
```
### Phase 2: Parallel Fetch (Gmail API — 10 concurrent)
```
STEP ACTION TECHNOLOGY TIME
─────── ─────────────────────────────── ───────────────────── ──────
2.1 Create concurrency limiter: p-limit npm <1ms
gmailLimit = pLimit(10) (10 concurrent)
2.2 For each newId, launch Promise + p-limit —
concurrent fetch:
┌──── gmailLimit(async () => {
│
│ 2.3 GET gmail/v1/users/me/ googleapis 100ms
│ messages/{id} (full MIME format) each
│ ?format=full
│
│ 2.4 Extract email body: Custom MIME parser <1ms
│ - Find text/html part (base64 decode)
│ - Base64 decode
│ - Strip HTML tags
│ - Extract plain text
│
│ 2.5 Extract email metadata: MIME header parse <1ms
│ - Date header
│ - From header
│ - Subject header
│
└──── return { emailId, body, metadata } })
With 10 concurrent: 50 emails fetched in ~500ms
200 emails in ~2 seconds
1000 emails in ~10 seconds
PHASE 2 TOTAL: ~100ms per email, 10 concurrent = ~10ms effective per email
```
### Phase 3: AI Parsing (Auto-Switcher — up to 15 concurrent on Groq)
This is where the AI models do the work. Every email body goes through the `AIProviderPool`.
```
STEP ACTION TECHNOLOGY TIME
─────── ─────────────────────────────── ───────────────────── ──────
3.1 AIProviderPool receives aiProviderPool.ts <1ms
email body text
3.2 Select best available slot: getNextAvailableSlot <1ms
Check priority order: ()
1. groq:gpt-oss-20b
2. groq:llama4-scout
3. groq:llama31-8b
...
7. mistral:small-3.2
...
3.3 Enforce rate delay: enforceRateDelay() 0-2000ms
Groq: 60s/30RPM = 2s delay (per slot)
Mistral: 60s/2RPM = 30s delay
3.4 Build AI request: <1ms
┌────────────────────────────────────────────────┐
│ { │
│ model: "openai/gpt-oss-20b", │
│ messages: [ │
│ { role: "system", │
│ content: EXTRACTION_SYSTEM_PROMPT }, │
│ { role: "user", │
│ content: "Extract transaction │
│ details:\n\n{EMAIL_BODY}" } │
│ ], │
│ temperature: 0.0, │
│ max_tokens: 500, │
│ response_format: { type: "json_object" } │
│ } │
└────────────────────────────────────────────────┘
3.5 Send to AI provider: fetch() or openai 150-800ms
npm package
┌─── IF GROQ: ───────────────────────────────────┐
│ POST https://api.groq.com/openai/v1/ │
│ chat/completions │
│ Headers: │
│ Authorization: Bearer gsk_xxxxx │
│ Content-Type: application/json │
│ │
│ Speed: 500-1000 tokens/sec │
│ Typical response: 150-300ms │
└──────────────────────────────────────────────────┘
┌─── IF MISTRAL: ────────────────────────────────┐
│ POST https://api.mistral.ai/v1/ │
│ chat/completions │
│ Headers: │
│ Authorization: Bearer xxxxx │
│ Content-Type: application/json │
│ │
│ Speed: 100-300 tokens/sec │
│ Typical response: 300-800ms │
└──────────────────────────────────────────────────┘
3.6 IF 429 RATE LIMIT: Auto-switcher <1ms
- Mark current slot as logic in pool
rate_limited
- Set cooldownUntil from
retry-after header
- WebSocket emit ai:switcher
- Jump to next priority slot
- RETRY from step 3.2
3.7 Receive AI response: JSON.parse() <1ms
{
"sender": "Jean Tremblay",
"amount": 150.00,
"currency": "CAD",
"reference": "CA1b2c3d4e5f",
"message": "Dîme mars 2025",
"recipient_email": "montreal.finances@iccameriques.org",
"date": "2026-02-23T14:30:00Z",
"status": "deposited"
}
3.8 Validate with Zod schema: zod npm <1ms
InteracTransactionSchema
.parse(parsed)
If validation fails:
- Log warning
- Retry with different model
- Or mark as needs_review
PHASE 3 TOTAL: ~200-800ms per email depending on provider
With Groq 15 concurrent: ~15ms effective per email
```
### Phase 4: Branch Routing (No AI — Pure Logic)
```
STEP ACTION TECHNOLOGY TIME
─────── ─────────────────────────────── ───────────────────── ──────
4.1 Look up recipient_email in BRANCH_MAPPING <1ms
branch mapping: (shared constant)
"montreal.finances JavaScript object
@iccameriques.org" lookup, O(1)
→ "ICC Montréal"
4.2 If no match found: Fallback logic <1ms
→ "Non classifié"
4.3 Attach branch to transaction: Object assign <1ms
transaction.branch =
"ICC Montréal"
PHASE 4 TOTAL: <1ms (no network, no AI, pure in-memory lookup)
```
### Phase 5: Database Save (Batch INSERT)
```
STEP ACTION TECHNOLOGY TIME
─────── ─────────────────────────────── ───────────────────── ──────
5.1 Buffer parsed transaction Array.push <1ms
into saveBuffer[]
5.2 When buffer reaches 25: Drizzle ORM 5-15ms
Batch INSERT INTO → PostgreSQL
transactions ( (parameterized)
email_id, user_id, date,
sender, amount, currency,
reference, message,
recipient_email, branch,
status, raw_email,
parsed_at, reviewed
) VALUES (...), (...), ...
★ Single INSERT for 25 rows
is 10-20x faster than 25
individual INSERTs
5.3 WebSocket emit per row: Socket.io <1ms
transaction:new { (per transaction)
transaction: {...}
}
Dashboard receives and
auto-adds to TanStack Table
5.4 Update scan_logs: Drizzle ORM → PG 2ms
UPDATE scan_logs SET
emails_parsed = emails_parsed + 25
WHERE id = {scanLogId}
PHASE 5 TOTAL: ~15ms per batch of 25 transactions
```
### Phase 6: Completion
```
STEP ACTION TECHNOLOGY TIME
─────── ─────────────────────────────── ───────────────────── ──────
6.1 Flush remaining buffer Drizzle → PG 5ms
(< 25 transactions)
6.2 Write final scan log: Drizzle → PG 2ms
UPDATE scan_logs SET
finished_at = NOW(),
emails_found = {n},
emails_parsed = {n},
errors = {n}
6.3 Write switcher logs: Drizzle → PG 2ms
INSERT INTO ai_switcher_logs
(batch of all switch events)
6.4 WebSocket emit: Socket.io <1ms
scan:completed {
jobId,
summary: { found, parsed,
skipped, errors },
dateRange,
duration: "45s"
}
6.5 Frontend shows toast: Sonner (toast lib) —
"47 nouveaux virements
importés en 45 secondes"
6.6 Dashboard auto-refreshes TanStack Query —
transaction list invalidateQueries()
(already live via WebSocket, (backup full refresh)
but also invalidates cache)
```
---
## 3. AI Model Connection Map
Exactly which AI model does what, and where in the code it connects.
```
┌──────────────────────────────────────────────────────────────────────────┐
│ │
│ WHAT AI DOES IN THIS PROJECT: │
│ │
│ AI has ONE job: Parse raw Interac email text → structured JSON │
│ │
│ AI does NOT do: │
│ ✗ Branch routing (pure lookup table, no AI) │
│ ✗ PDF generation (PDFKit / @react-pdf/renderer, no AI) │
│ ✗ Export CSV/Excel (SheetJS, no AI) │
│ ✗ Dashboard charts (Recharts, no AI) │
│ ✗ Authentication (Google OAuth, no AI) │
│ ✗ Email fetching (Gmail API / IMAP, no AI) │
│ ✗ Database queries (Drizzle ORM / SQL, no AI) │
│ ✗ Real-time updates (WebSocket, no AI) │
│ │
└──────────────────────────────────────────────────────────────────────────┘
```
### Where Each Model Connects
```
┌─────────────────────────────────────────────────────────────────────────┐
│ FILE: packages/server/src/services/aiProviderPool.ts │
│ CLASS: AIProviderPool │
│ METHOD: parse(emailBody: string) → InteracTransaction │
│ │
│ This is the ONLY place AI is called in the entire application. │
│ Everything below is managed by the auto-switcher inside this method. │
│ │
│ SLOT 1 ─── groq:gpt-oss-20b ──────────────────────────────────────────│
│ │ Provider: Groq │
│ │ Model: openai/gpt-oss-20b │
│ │ API: POST https://api.groq.com/openai/v1/chat/completions │
│ │ SDK: openai npm (baseURL override) │
│ │ File: packages/server/src/providers/groq.ts │
│ │ Auth: GROQ_API_KEY env var (free, no credit card) │
│ │ Speed: 1,000 tokens/sec · response in ~150ms │
│ │ Free: 30 RPM · 7,000 req/day · 500K tokens/day │
│ │ Input: ~800 tokens (system prompt + email body) │
│ │ Output: ~200 tokens (JSON transaction object) │
│ │ Quality: ★★★★☆ (good at structured extraction) │
│ │ French: ★★★★☆ │
│ │ │
│ SLOT 2 ─── groq:llama4-scout ─────────────────────────────────────────│
│ │ Provider: Groq │
│ │ Model: meta-llama/llama-4-scout-17b-16e-instruct │
│ │ API: POST https://api.groq.com/openai/v1/chat/completions │
│ │ SDK: openai npm (same Groq adapter) │
│ │ Speed: 594 tokens/sec │
│ │ Free: 30 RPM · 1,000 req/day · 500K tokens/day │
│ │ Quality: ★★★★☆ │
│ │ French: ★★★½☆ │
│ │ │
│ SLOT 3 ─── groq:llama31-8b ──────────────────────────────────────────│
│ │ Provider: Groq │
│ │ Model: llama-3.1-8b-instant │
│ │ API: POST https://api.groq.com/openai/v1/chat/completions │
│ │ Speed: 840 tokens/sec · response in ~100ms │
│ │ Free: 30 RPM · 14,400 req/day · 500K tokens/day │
│ │ Quality: ★★★☆☆ (simpler extraction, may miss edge cases) │
│ │ French: ★★★☆☆ │
│ │ │
│ SLOT 4 ─── groq:qwen3-32b ───────────────────────────────────────────│
│ │ Provider: Groq │
│ │ Model: qwen/qwen3-32b │
│ │ Speed: 662 tokens/sec │
│ │ Free: 30 RPM · 1,000 req/day · 500K tokens/day │
│ │ Quality: ★★★★☆ │
│ │ French: ★★★★☆ (strong multilingual) │
│ │ │
│ SLOT 5 ─── groq:llama4-maverick ──────────────────────────────────────│
│ │ Provider: Groq │
│ │ Model: meta-llama/llama-4-maverick-17b-128e-instruct │
│ │ Speed: 562 tokens/sec │
│ │ Free: 30 RPM · 1,000 req/day · 500K tokens/day │
│ │ Quality: ★★★★☆ │
│ │ French: ★★★★☆ │
│ │ │
│ SLOT 6 ─── groq:gpt-oss-120b ────────────────────────────────────────│
│ │ Provider: Groq │
│ │ Model: openai/gpt-oss-120b │
│ │ Speed: 500 tokens/sec │
│ │ Free: 30 RPM · 1,000 req/day · 100K tokens/day │
│ │ Quality: ★★★★★ (best reasoning on Groq) │
│ │ French: ★★★★☆ │
│ │ │
│ ────── PROVIDER BOUNDARY: Groq → Mistral ──────────────────────────── │
│ │
│ SLOT 7 ─── mistral:small-3.2 ────────────────────────────────────────│
│ │ Provider: Mistral AI │
│ │ Model: mistral-small-3.2-24b-instruct │
│ │ API: POST https://api.mistral.ai/v1/chat/completions │
│ │ SDK: fetch() (OpenAI-compatible) │
│ │ File: packages/server/src/providers/mistral.ts │
│ │ Auth: MISTRAL_API_KEY env var (free, no credit card) │
│ │ Speed: ~200 tokens/sec · response in ~400ms │
│ │ Free: 2 RPM · ~2,880 req/day · 1B tokens/month │
│ │ Quality: ★★★★☆ (excellent structured extraction) │
│ │ French: ★★★★★ (built by French company, native French) │
│ │ │
│ SLOT 8 ─── mistral:medium-3.1 ───────────────────────────────────────│
│ │ Provider: Mistral AI │
│ │ Model: mistral-medium-3.1 │
│ │ Speed: ~150 tokens/sec │
│ │ Free: 2 RPM · ~2,880 req/day · 1B tokens/month │
│ │ Quality: ★★★★★ (highest accuracy) │
│ │ French: ★★★★★ │
│ │ │
│ SLOT 9 ─── mistral:ministral-8b ─────────────────────────────────────│
│ │ Provider: Mistral AI │
│ │ Model: ministral-8b-2512 │
│ │ Speed: ~300 tokens/sec │
│ │ Free: 2 RPM · ~2,880 req/day · 1B tokens/month │
│ │ Quality: ★★★☆☆ │
│ │ French: ★★★★☆ │
│ │ │
└─────────────────────────────────────────────────────────────────────────┘
```
---
## 4. Technology → Purpose Map (Every Library)
### Frontend (packages/web)
| Technology | Version | What It Does in This Project | Touches AI? |
|-----------|---------|------------------------------|-------------|
| **Vite** | 6 | Dev server, HMR, production bundler | No |
| **React** | 19 | Component rendering, UI framework | No |
| **TypeScript** | 5 | Type safety across all code | No |
| **Tailwind CSS** | 4 | Utility-first styling for all components | No |
| **shadcn/ui** | latest | Pre-built components: Button, Dialog, DatePicker, Table, Toggle, Toast | No |
| **React Router** | v7 | Page routing: /login, /dashboard, /scan, /settings, /reports | No |
| **Zustand** | latest | Global state: auth user, scan status, provider pool status | No |
| **TanStack Query** | v5 | Server state: transactions list, scan history, settings (auto-cache, refetch) | No |
| **TanStack Table** | v8 | Dashboard transaction table: sort, filter, paginate, select, expand | No |
| **React Hook Form** | latest | Settings forms: AI config, branch config, scan date range | No |
| **Zod** | latest | Frontend validation: date ranges, settings input | No |
| **Socket.io-client** | latest | Real-time: scan:progress, transaction:new, ai:switcher events | No |
| **Recharts** | latest | Reports page: bar charts, line charts, pie charts | No |
| **@react-pdf/renderer** | latest | Client-side PDF receipt generation (single transaction) | No |
| **SheetJS** | latest | Client-side Excel/CSV export | No |
| **date-fns** | latest | Date formatting with fr-CA locale | No |
| **Lucide React** | latest | Icons throughout the UI | No |
| **Sonner** | latest | Toast notifications ("47 virements importés") | No |
| **react-i18next** | latest | French UI translations | No |
| **p-limit** | latest | Used in scan progress UI for throttled updates | No |
### Backend (packages/server)
| Technology | Version | What It Does in This Project | Touches AI? |
|-----------|---------|------------------------------|-------------|
| **Express.js** | 5 | HTTP API server, route handling | No |
| **TypeScript** | 5 | Type safety across all server code | No |
| **googleapis** | latest | Gmail API: fetch message IDs, fetch full MIME emails | No |
| **google-auth-library** | latest | OAuth 2.0: token exchange, refresh, consent URL | No |
| **imapflow** | latest | IMAP fallback: connect to Gmail via IMAP if API unavailable | No |
| **openai** (npm) | latest | **Groq adapter**: same SDK, different baseURL → api.groq.com | **YES** |
| **fetch** (built-in) | — | **Mistral adapter**: direct HTTP to api.mistral.ai | **YES** |
| **@anthropic-ai/sdk** | latest | **Claude adapter** (optional paid): api.anthropic.com | **YES** |
| **Drizzle ORM** | latest | Database queries: all CRUD, batch inserts, migrations | No |
| **PostgreSQL driver** (pg) | latest | Database connection pooling | No |
| **Redis** (ioredis) | latest | BullMQ backend, rate limit counters, session cache | No |
| **BullMQ** | latest | Background job queue: scan jobs run async, not blocking API | No |
| **Socket.io** | latest | WebSocket server: push scan progress + provider switches to UI | No |
| **jsonwebtoken** | latest | JWT: issue access tokens (15 min) + refresh tokens (7 day) | No |
| **PDFKit** | latest | Server-side batch PDF receipt generation | No |
| **Pino** | latest | Structured JSON logging | No |
| **Helmet** | latest | Security headers (CSP, HSTS, etc.) | No |
| **cors** | latest | CORS policy: restrict to known frontend origins | No |
| **express-rate-limit** | latest | API rate limiting (prevent abuse of scan endpoints) | No |
| **p-limit** | latest | Concurrency control: 10 Gmail fetches, 15 Groq parses | No |
| **Zod** | latest | Server-side validation: AI output JSON, API request bodies | No |
| **crypto** (built-in) | — | AES-256 encryption: OAuth tokens at rest in PostgreSQL | No |
### Shared (packages/shared)
| Technology | What It Does |
|-----------|-------------|
| **TypeScript types** | InteracTransaction, ScanDateRange, ProviderSlot, SwitcherEvent |
| **BRANCH_MAPPING** | 48 email→branch lookup pairs (no AI needed) |
| **resolveScanDates()** | Preset→date range helper |
| **Zod schemas** | InteracTransactionSchema for validating AI output |
---
## 5. AI Data Flow — Input → Output
### What Goes INTO the AI
```
╔══════════════════════════════════════════════════════════════════════╗
║ SYSTEM PROMPT (~350 tokens, same for every email, cached) ║
║ ║
║ "You are a financial data extraction assistant. Given the raw ║
║ text/HTML of an Interac e-Transfer notification email from ║
║ notify@payments.interac.ca, extract the following fields into ║
║ a JSON object: ║
║ ║
║ - sender: The name of the person who sent the money ║
║ - amount: The dollar amount (numeric, no $ sign) ║
║ - currency: Always "CAD" ║
║ - reference: The Interac reference number ║
║ - message: The personal message/memo (null if none) ║
║ - recipient_email: The email the transfer was sent TO ║
║ - date: The date/time in ISO 8601 format ║
║ - status: One of deposited, pending, expired, cancelled ║
║ ║
║ Return ONLY valid JSON, no markdown, no explanation." ║
╠══════════════════════════════════════════════════════════════════════╣
║ USER MESSAGE (~450 tokens, unique per email) ║
║ ║
║ "Extract transaction details: ║
║ ║
║ INTERAC e-Transfer ║
║ Bonjour, vous avez reçu un virement Interac de Jean Tremblay. ║
║ Montant : 150,00 $ (CAD) ║
║ Numéro de référence: CA1b2c3d4e5f ║
║ Message de l'expéditeur : Dîme mars 2025 ║
║ Ce virement a été automatiquement déposé dans le compte de: ║
║ montreal.finances@iccameriques.org ║
║ Date: 23 février 2026 14:30 ║
║ ..." ║
╚══════════════════════════════════════════════════════════════════════╝
Total input: ~800 tokens
```
### What Comes OUT of the AI
```
╔══════════════════════════════════════════════════════════════════════╗
║ AI RESPONSE (~200 tokens) ║
║ ║
║ { ║
║ "sender": "Jean Tremblay", ║
║ "amount": 150.00, ║
║ "currency": "CAD", ║
║ "reference": "CA1b2c3d4e5f", ║
║ "message": "Dîme mars 2025", ║
║ "recipient_email": "montreal.finances@iccameriques.org", ║
║ "date": "2026-02-23T14:30:00Z", ║
║ "status": "deposited" ║
║ } ║
╚══════════════════════════════════════════════════════════════════════╝
Total output: ~200 tokens
Total per email: ~1,000 tokens
```
### What Happens AFTER AI (No AI Involved)
```
AI Output (JSON)
│
▼
Zod Validation ──── FAIL? → log warning, retry with next model, or flag as needs_review
│
✅ PASS
│
▼
Branch Routing ──── BRANCH_MAPPING["montreal.finances@iccameriques.org"] → "ICC Montréal"
│ (pure JavaScript lookup, no AI)
│
▼
PostgreSQL INSERT ── Drizzle ORM → INSERT INTO transactions (...)
│
▼
WebSocket Push ──── Socket.io → transaction:new { ... }
│
▼
React Dashboard ─── TanStack Table auto-adds row to table
│
▼
User Sees New Row ── "Jean Tremblay | 150,00 $ | ICC Montréal | Déposé"
```
---
## 6. Speed Optimization — Why It's Fast
### Old Architecture (Sequential)
```
Email 1: [Fetch 100ms] → [AI Parse 300ms] → [Save 5ms] = 405ms
Email 2: [Fetch] → [Parse] → [Save]
Email 3: [Fetch] → ...
Total for 50 emails: 50 × 405ms = 20,250ms = ~20 seconds
```
### New Architecture (Parallel Pipeline)
```
Time → 0ms 100ms 200ms 300ms 400ms 500ms
┌──────┬──────┬──────┬──────┬──────┬──────┐
Fetch: │E1-E10│E11-20│E21-30│E31-40│E41-50│ │ (10 concurrent)
Parse: │ │P1-P15│P16-30│P31-45│P46-50│ │ (15 concurrent)
Save: │ │ │S1-S25│ │S26-50│ │ (batch 25)
└──────┴──────┴──────┴──────┴──────┴──────┘
Total for 50 emails: ~500ms for fetches + ~700ms for parses = ~1.2 seconds
(stages overlap, so total < sum)
```
### Concurrency Budget Per Free Tier
```
GROQ MISTRAL
RPM (requests/minute): 30 2
Safe concurrency: 15 concurrent calls 1 sequential call
Time per AI call: ~150ms (Groq is fast) ~400ms
Emails parsed per minute: ~30 (RPM-limited) ~2 (RPM-limited)
Emails parsed per hour: ~1,800 ~120
WITH AUTO-SWITCHER (all 6 Groq + 3 Mistral):
Effective RPM: 30 + 30 + 30 + 30 + 30 + 30 + 2 + 2 + 2 = 186 RPM*
(*until individual daily limits hit)
First hour throughput: ~180 emails/min = ~10,800 emails/hour
```
### Speed Benchmarks (Revised with Pipeline)
| Scan Type | Emails | Time (Pipeline + Auto-Switcher) |
|-----------|--------|-------------------------------|
| Today | 50 | **~12 seconds** |
| 7 Days | 200 | **~35 seconds** |
| Custom (1 month) | 1,000 | **~3 minutes** |
| Custom (6 months) | 5,000 | **~15 minutes** |
| Custom (1 year) | 10,000 | **~30 minutes** |
---
## 7. Non-AI Technology Flows
### PDF Receipt Generation (No AI)
```
User clicks "Générer reçu" on a transaction row
│
▼
CLIENT-SIDE (single receipt):
@react-pdf/renderer builds PDF in browser
Template: ICC logo + transaction details + branch info + date
Downloads immediately as "recu_CA1b2c3d4e5f.pdf"
SERVER-SIDE (batch receipts):
POST /api/receipts/batch { transactionIds: [...] }
│
▼
PDFKit generates PDF per transaction
Merges into single ZIP file
Returns ZIP for download
```
### CSV/Excel Export (No AI)
```
User clicks "Exporter" → selects CSV or Excel
│
▼
CLIENT-SIDE:
SheetJS (xlsx npm) builds file from TanStack Table data
Columns: Date | Expéditeur | Montant | Référence | Succursale | Statut
fr-CA formatting: "1 500,00 $" not "$1,500.00"
Downloads immediately
```
### Reports & Charts (No AI)
```
User navigates to /reports
│
▼
GET /api/transactions/stats?from=2026-01-01&to=2026-02-23
│
▼
PostgreSQL aggregation queries:
- SUM(amount) GROUP BY month → bar chart (Recharts)
- COUNT(*) GROUP BY status → pie chart (Recharts)
- COUNT(*) GROUP BY branch → top branches table
- SUM(amount) over time → trend line chart (Recharts)
│
▼
React renders charts with Recharts
No AI involved — pure SQL aggregation + charting library
```
### Real-Time Dashboard Updates (No AI)
```
Server saves new transaction to PostgreSQL
│
▼
Socket.io server emits:
transaction:new { transaction: {...} }
│
▼
Socket.io client receives event
│
▼
Zustand store updates transaction list
│
▼
TanStack Table re-renders with new row
(animated row insertion at top of table)
│
▼
SummaryBar recalculates totals
(no page refresh, no API call needed)
```
---
## 8. Security Flow (No AI)
```
1. User clicks "Se connecter avec Google"
2. Frontend redirects to Google OAuth consent URL
(scopes: gmail.readonly, userinfo.email, userinfo.profile)
3. User approves → Google redirects to /api/auth/google/callback?code=xxx
4. Backend exchanges auth code for access_token + refresh_token
(google-auth-library)
5. Backend encrypts tokens with AES-256 (crypto module)
6. Backend stores encrypted tokens in PostgreSQL (users table)
7. Backend issues JWT: { userId, email, role, exp: 15min }
(jsonwebtoken)
8. Frontend stores JWT in httpOnly secure cookie
9. Every API request includes JWT in Authorization header
10. Backend middleware verifies JWT on every request
(jsonwebtoken.verify)
11. If JWT expired → frontend calls POST /api/auth/refresh
with refresh token → new JWT issued
```
---
*This document is a companion to the ICC Interac Manager Build Prompt and the FREE AI Models Guide. Together, the three documents provide everything needed to build the complete system.*
|