File size: 4,804 Bytes
12eff8e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 | # MorphSQL — Project Context Document
> **Purpose:** Give this file to any AI assistant or new developer to continue work without prior conversation context.
## Naming
| Layer | Name |
|-------|------|
| **Product (UI, docs, Hugging Face Space title)** | **MorphSQL** |
| Python import | `morphsql` |
| PyPI package | `morphsql` |
| GitHub repo | `morphsql` |
| HF Space / model slug | `dgvj-work/morphsql` |
Always use **MorphSQL** / `morphsql` consistently (product, package, CLI, Hub).
## What this project is
**MorphSQL** is an open-source **SQL migration toolkit** (package import: `morphsql`).
It is **not** only a single-query converter. It analyzes legacy SQL repositories and produces:
1. **Discovery** — scan zip files/directories for SQL, procedures, views, dbt models
2. **Lineage** — dependency graphs (table/object level)
3. **Risk assessment** — complexity scoring, unsupported syntax, downstream impact
4. **Workload rationalization** — migrate / review / rewrite / retire per object
5. **Hybrid translation** — rule-based + sqlglot dialect conversion (pandas / PySpark / warehouse / dbt)
6. **dbt architecture** — decompose procedures into staging/intermediate/mart
7. **Validation** — reconciliation tests (row count, null rate, checksums)
8. **Migration runbook** — phased cutover plan with object action table
9. **Migration copilot** — LLM advisor grounded in scan context (HF Inference API)
## Repository layout
```
morphsql/
├── app.py # Hugging Face Gradio Space (MorphSQL UI)
├── demo/
│ ├── handlers.py # Gradio event handlers
│ └── theme.py # Dark theme CSS/constants
├── morphsql/ # Core Python package
│ ├── scanner/ # Repository discovery
│ ├── parser/ # SQL parsing (sqlglot)
│ ├── lineage/ # Dependency graphs (networkx)
│ ├── translator/ # Hybrid SQL conversion (+ pandas/pyspark codegen)
│ ├── risk/ # Complexity & risk scoring
│ ├── dbt_generator/ # dbt project decomposition
│ ├── validation/ # Reconciliation tests
│ ├── knowledge/ # Platform behavior differences KB
│ ├── intelligence/ # Runbook, rationalization, lineage viz
│ ├── assistant/ # LLM copilot (huggingface_hub)
│ ├── report/ # HTML report generation
│ ├── pipeline.py # Agent orchestration
│ ├── models.py # Pydantic data models
│ └── cli.py # `morphsql` CLI
├── examples/vertica_legacy/ # Sample legacy Vertica repository
├── tests/
├── PROJECT.md # This file
├── README.md
├── README_HF_SPACE.md # HF Space card (title: MorphSQL)
├── MODEL_CARD.md
└── pyproject.toml
```
## Architecture
```
Repository (zip / directory)
↓
MigrationPipeline.analyze() → Discovery + Lineage + Risk
↓
MigrationPipeline.convert() → Hybrid SQL translation
↓
MigrationPipeline.validate() → Reconciliation tests
↓
generate_runbook() → Cutover plan
generate_rationalization() → Migrate/retire recommendations
decompose_to_dbt() → dbt scaffold
MigrationCopilot.respond() → LLM Q&A grounded in report
```
## Key modules
| Module | File | Purpose |
|--------|------|---------|
| Pipeline | `morphsql/pipeline.py` | Orchestrates full workflow |
| Translator | `morphsql/translator/engine.py` | ZEROIFNULL, DATEADD, procedure wrappers, sqlglot |
| Pandas / PySpark | `morphsql/translator/*_codegen.py` | Notebook / Spark DataFrame codegen |
| Copilot | `morphsql/assistant/copilot.py` | HF Inference API + fallback KB |
| Runbook | `morphsql/intelligence/runbook.py` | Migration runbook markdown |
| Lineage viz | `morphsql/intelligence/lineage_viz.py` | Plotly network graph |
| Rationalization | `morphsql/intelligence/rationalization.py` | Workload action plan |
## CLI commands
```bash
morphsql analyze PATH -s vertica -t snowflake
morphsql convert PATH -s snowflake -t pandas
morphsql migrate PATH -s vertica -t snowflake -o migration-output
morphsql version
```
## Hugging Face
- Space title: **MorphSQL**
- Space URL: https://huggingface.co/spaces/dgvj-work/morphsql
- Deploy: `./scripts/deploy_hf.sh`
- Preflight: `python scripts/check_space.py`
- Copilot model env: `MORPHSQL_MODEL`
## Gradio UI tabs
1. **Convert** — primary: paste/upload SQL → pandas/PySpark/SQL/dbt + preview + download
2. **Guide** — how-to for DS / AI users
3. **More** — object assess, repo workbench, feature SQL, copilot, RAG, eval
|