File size: 6,890 Bytes
fbb917f
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
# BeFree Developer Contribution Guide

This guide is designed to onboard new developers and interns onto the BeFree Bank Statement Processor project, detailing environment setups, codebase architecture tours, feature development patterns, and type checking.

---

## 1. Local Developer Setup

Follow these steps to configure your environment and run BeFree locally.

### 1.1 Prerequisites
- **Python 3.11+** installed.
- **Node.js 18+** and **npm** installed.
- **Git** installed.
- **Sibling Repositories**: The internal editable packages (`bank-extractor` and `bank-hitl`) must be cloned in directories alongside this repository.

### 1.2 Copy Environment Configuration
```bash
cp .env.example .env
```
Open `.env` and fill in:
- `GOOGLE_API_KEY`: Get a Gemini developer key from Google AI Studio.
- `DATABASE_URL`: Setup a free tier PostgreSQL database on NeonDB and input the URL (ensure `?sslmode=require` is appended).

### 1.3 Setup Python Virtual Environment
```bash
# Create virtual environment
python3 -m venv venv

# Activate virtual environment
source venv/bin/activate

# Install requirements (which installs local packages in editable mode)
pip install -r requirements.txt
```

### 1.4 Database Migrations & Seeding
```bash
# Run schema migrations
python auth/migrations.py

# Seed default employee users
python auth/seed_users.py
```

### 1.5 Run Backend
```bash
python api_server.py
# The backend will start on http://localhost:8000
# OpenAPI interactive Swagger docs are available at http://localhost:8000/docs
```

### 1.6 Run Frontend
```bash
cd befree-frontend
npm install
npm run dev
# The frontend will start on http://localhost:3000
```

### 1.7 Run Both Simultaneously
Alternatively, launch both backend and frontend servers with one command:
```bash
./run.sh
```

---

## 2. Project Layout Tour

- [api_server.py](file:///home/luv/Desktop/new/api_server.py) β€” The API server entry point that instantiates the FastAPI app, configures CORS/middlewares, and mounts the feature-domain sub-routers.
- [models/](file:///home/luv/Desktop/new/models) β€” Pydantic request/response payload schemas grouped by domain (e.g., `statements.py`, `transactions.py`).
- [core/](file:///home/luv/Desktop/new/core) β€” Core business logic, pipeline initialization, statement IO, and global HTTP middlewares.
- [routers/](file:///home/luv/Desktop/new/routers) β€” Modular FastAPI APIRouter endpoints split by feature domain (e.g., `upload.py`, `dashboard.py`).
- [auth/](file:///home/luv/Desktop/new/auth) β€” Database schemas and services for user session validation, JWT logic, and classification rules storage.
- [overwatch/](file:///home/luv/Desktop/new/overwatch) β€” Observable stage log database store, event definitions, and context managers.
- [shared/](file:///home/luv/Desktop/new/shared) β€” Shared Pydantic data schemas and constants.
- [bank-classifier/](file:///home/luv/Desktop/new/bank-classifier) β€” Sequential 3-stage classification engine logic, Gemini connections, and rule promotion learners.
- [befree-frontend/](file:///home/luv/Desktop/new/befree-frontend) β€” Next.js 15 frontend application.
  - `befree-frontend/app/` β€” Routing structure (pages).
  - `befree-frontend/features/` β€” Feature UI components (e.g. `overwatch`, `dashboard`, `coordinate-editor`).
  - `befree-frontend/services/api-client.ts` β€” Central Axios API caller class.
- [scripts/](file:///home/luv/Desktop/new/scripts) β€” Diagnostic, data migration, and comparative verification utility scripts.
- [data/](file:///home/luv/Desktop/new/data) β€” Local runtime file folders (uploads, output, logs) ignored by git.

---

## 3. Adding a New API Endpoint

Backend routes are modularized inside the `routers/` directory using FastAPI `APIRouter` objects. To declare a new route:

1. Locate the appropriate router file in `routers/` (e.g., `routers/review.py` or `routers/transactions.py`).
2. Add a route using the `@router` decorator, specifying request/response validation models from `models/` and authentication dependencies:
   ```python
   @router.post("/new-feature", response_model=MyResponseModel)
   async def handle_new_feature(
       payload: MyRequestPayload,
       current_user: User = Depends(get_current_user) # Enforces JWT validation
   ):
       # Ensure roles are verified if admin-only:
       # if current_user.role != "admin":
       #     raise HTTPException(status_code=403, detail="Admin only")
       
       # Business logic here
       return {"status": "success"}
   ```
3. If creating a new router file, define the `router = APIRouter(...)` and mount it in `api_server.py` using `app.include_router(new_router)`.

---

## 4. Adding a New Classification Rule

New classification rules can be added in two ways:
- **Admin Interface**: Log in as an admin role, navigate to the Rules settings dashboard, and add keywords. This calls `POST /api/v1/admin/rules`.
- **Database Rules Seeding**: To add standard rules into the system database:
  1. Open `bank-classifier/bank_classifier/rules/default_rules.yaml`.
  2. Add your category keyword configuration.
  3. Execute `python scripts/migrate_rules_to_db.py` to idempotently upsert the definitions.

---

## 5. Adding a New Frontend Feature

The frontend organizes code by features inside `befree-frontend/features/`.
1. **Component Design**: Place your components inside `befree-frontend/features/my-feature/`.
2. **API Client Integration**: Declare backend fetch calls in `befree-frontend/services/api-client.ts` as client methods.
3. **Zustand Store / Custom Hooks**: Create local stores or query hooks inside `befree-frontend/hooks/` to retrieve data.
4. **App View**: Include the feature component in a route inside `befree-frontend/app/`.

---

## 6. Running the Classification Pipeline Standalone

You can invoke the classification pipeline directly from a Python script without booting the FastAPI web server:
```python
from dotenv import load_dotenv
load_dotenv()

from bank_classifier.pipeline import ClassificationPipeline
from shared.models import NormTransaction

# Initialize pipeline
pipeline = ClassificationPipeline()

# Prepare mock transaction
tx = NormTransaction(
    page_num=1,
    row_num=1,
    date_raw="24/05/2026",
    desc_raw="Netflix subscription",
    debit_raw="18.99",
    amount=-18.99
)

# Run classification batch
results = pipeline.classify_batch([tx])
print(results[0].category)  # Output: Recreation & Entertainment
```

---

## 7. Linting and Type Checking

- **Frontend (TypeScript)**: Run static TypeScript compiler checks before submitting code:
  ```bash
  cd befree-frontend
  npm run type-check
  ```
- **Backend (Python)**: The project currently does not package standard linters. It is highly recommended to install `ruff` for code linting and formatting:
  ```bash
  pip install ruff
  
  # Lint code
  ruff check .
  
  # Format code
  ruff format .
  ```