Update README.md

README.md

@@ -1,3 +1,161 @@
+# Text-to-SQL Evaluation Pipeline
+
+## Overview
+
+This repository implements a **Text-to-SQL evaluation pipeline** using the OpenAI API. The system is designed for the **banking domain**, where strict business rules and deterministic SQL generation are required.
+
+Key goals:
+
+- Translate **natural language questions** into **valid SQLite SQL queries**
+- Enforce **domain-specific constraints** via prompt engineering
+- Benchmark model outputs using **multiple evaluation metrics**
+
+---
+
+## Key Features
+
+> **Important:** The existing `MyModel` class and its formatting are **kept exactly as-is**.\
+> This project does **not** modify, refactor, or reformat the demo code. The README only documents how the current implementation works.
+
+- The `MyModel` class structure, method names, and prompt formatting remain unchanged
+- No code auto-formatting or refactoring is applied
+- All behavior described below reflects the **original demo code**
+
+## Key Features
+
+- Deterministic SQL generation (`temperature = 0`)
+- Strong prompt constraints (no markdown, no explanations, SQL only)
+- Banking-specific metric grouping and unit conversion logic
+- Multi-metric evaluation for both syntactic and semantic quality
+
+---
+
+## High-level Architecture
+
+```text
+User Question
+     │
+     ▼
+Prompt Builder (System + User)
+     │
+     ▼
+OpenAI ChatCompletion API
+     │
+     ▼
+Generated SQL
+     │
+     ▼
+Evaluation Metrics
+```
+
+---
+
+## Suggested Project Structure
+
+```text
+.
+├── main.py              # Entry point, runs inference and prints metrics
+├── model.py             # OpenAI client wrapper (MyModel)
+├── evaluator.py         # Evaluation metrics implementation
+├── prompts/
+│   └── text2sql.txt     # System prompt with banking rules
+├── README.md
+└── requirements.txt
+```
+
+---
+
+## Prompt Design
+
+### System Prompt Responsibilities
+
+The system prompt enforces the following rules:
+
+- **Always perform** an `INNER JOIN` between `system_data` and `branch`
+- **Always SELECT** the following columns:
+  - `system_data.data_code`
+  - `system_data.year`
+  - `system_data.branch_id`
+  - `branch.name`
+  - `system_data.value`
+- SQL keywords must be **UPPERCASE**
+- Text filters must use `LIKE '%keyword%'`
+- Vietnamese location names must use **exact accents**
+- Output **SQL only** (no markdown, no explanations)
+
+### Metric Grouping Logic
+
+Metrics are classified by `metric_code` prefix:
+
+| Group | Description                          |
+| ----- | ------------------------------------ |
+| A     | Inbound metrics (`MET_A_%`)          |
+| B     | Outbound metrics (`MET_B_%`)         |
+| C     | Stock / snapshot metrics (`MET_C_%`) |
+| D     | Exposure / obligation metrics        |
+| E     | Resource mobilization metrics        |
+| F     | Ratio & efficiency metrics           |
+
+### Unit Conversion Rule
+
+- Stored unit: **Million VND**
+- If the question mentions **"Billion VND"**, multiply value by `1000`
+
+---
+
+## Example Input (Schema)
+
+```sql
+CREATE TABLE entity_a (
+    id INTEGER,
+    group_id INTEGER,
+    org_id INTEGER,
+    code VARCHAR(100),
+    name VARCHAR(255),
+    attr_1 VARCHAR(255),
+    attr_2 VARCHAR(255),
+    attr_3 TEXT
+);
+
+CREATE TABLE entity_b (
+    id INTEGER,
+    group_id INTEGER,
+    entity_a_id INTEGER,
+    time_key INTEGER,
+    metric_name VARCHAR(255),
+    metric_code VARCHAR(100),
+    metric_value REAL,
+    metric_unit VARCHAR(100)
+);
+```
+
+---
+
+## Evaluation Metrics
+
+Evaluation results are printed **at the very top of the output**:
+
+| Label          | Value        |
+| -------------- | ------------ |
+| rouge          | 0.9290708304 |
+| meteor         | 0.9191570862 |
+| binary         | 0.55         |
+| llm-as-a-judge | 0.65         |
+
+### Metric Definitions
+
+- **ROUGE**: Token-level overlap between generated and reference SQL
+- **METEOR**: Semantic similarity with synonym awareness
+- **Binary Match**: Exact string match (0 or 1)
+- **LLM-as-a-Judge**: LLM-based holistic judgment of correctness
+
+---
+
+## Full Demo Code (Kept Exactly As-Is)
+
+The following is the **original demo code**, included verbatim for clarity and ease of understanding. No refactoring, no reformatting, no behavioral changes have been applied.
+
+```python
 import argparse
 
 from openai import OpenAI
@@ -12,7 +170,6 @@ DEFAULT_QUESTION = """CREATE TABLE entity_a (
     attr_2 VARCHAR(255),
     attr_3 TEXT
 );
-
 CREATE TABLE entity_b (
     id INTEGER,
     group_id INTEGER,
@@ -36,7 +193,6 @@ ENTITIES = {
     },
     "time_key": [year],
 Query:
-
 }
 
 
@@ -46,8 +202,7 @@ Query:
 class MyModel(object):
     def __init__(self, model_name: str, api_key: str):
         self.model_name = model_name
-        self.client = OpenAI(base_url=""", api_key=api_key)
-
+        self.client = OpenAI(base_url="", api_key=api_key)
     def get_prompt(
         self,
         question: str,
@@ -58,7 +213,6 @@ class MyModel(object):
                 "content": """
 You are a problem solving model working on task_description XML block:
 <task_description>You are a specialized Text-to-SQL assistant in the banking domain. Your objective is to translate natural language questions into valid SQLite queries using the provided schema and banking business logic.
-
 ### Input:
 - Schema: Table definitions in SQL DDL format.
 - Relationships: Key linking logic between tables (system_data.branch_id = branch.id).
@@ -96,7 +250,6 @@ You are a problem solving model working on task_description XML block:
         rule:
   - Unit Logic: {Which dmain} data is stored in 'Triệu VND'. If the Question mentions 'Tỷ', multiply the value by 1000.
 - Entities: Extracted key information including data_code, year, and branch filtering criteria.
-
 ### Rules:
 1. ALWAYS perform an INNER JOIN between system_data and branch on system_data.branch_id = branch.id.
 2. ALWAYS SELECT system_data.data_code, system_data.year, system_data.branch_id, branch.name, system_data.value.
@@ -112,14 +265,12 @@ Generate only the answer, do not generate anything else
             {
                 "role": "user",
                 "content": f"""
-
 Now for the real task, solve the task in question block.
 Generate only the solution, do not generate anything else
 <question>{question}</question>
 """,
             },
         ]
-
     def invoke(self, question: str) -> str:
         chat_response = self.client.chat.completions.create(
             model=self.model_name,
@@ -129,15 +280,41 @@ Generate only the solution, do not generate anything else
         )
         return chat_response.choices[0].message.content
 
-
 if __name__ == "__main__":
     parser = argparse.ArgumentParser()
     parser.add_argument("--question", type=str, default=DEFAULT_QUESTION, required=False)
     parser.add_argument("--api-key", type=str, default="", required=False)
     parser.add_argument("--model", type=str, default="model", required=False)
-
     args = parser.parse_args()
-
     client = MyModel(model_name=args.model, api_key=args.api_key)
+    print(client.invoke(args.question))
+```
+
+---
+
+## How to Run
+
+```bash
+python main.py \
+  --question "<QUESTION_TEXT>" \
+  --api-key "YOUR_OPENAI_API_KEY" \
+  --model "gpt-4.1-mini"
+```
+
+---
+
+## Important Notes
+
+- `temperature = 0` ensures reproducible results
+- Function calling is intentionally avoided to prevent JSON-wrapped SQL
+- The prompt is optimized for **SQLite dialect**
+
+---
+
+## Possible Extensions
+
+- Multi-year queries using `IN` or ranges
+- Queries combining multiple metric groups
+- Execution-based evaluation (SQL result comparison)
+- Support for additional SQL dialects (PostgreSQL, MySQL)
 
-    print(client.invoke(args.question))