|
|
--- |
|
|
license: mit |
|
|
--- |
|
|
|
|
|
# CSV Data Processing & Merging Pipeline |
|
|
|
|
|
## Overview |
|
|
This project processes and merges multiple CSV datasets by intelligently mapping column headers, cleaning data, and ensuring consistency. It leverages AI-based semantic similarity to standardize headers and applies filtering rules for valid data entries. |
|
|
|
|
|
## Features |
|
|
- **Recursive CSV File Search:** Collects all CSV files from a specified directory. |
|
|
- **AI-Powered Header Mapping:** Uses Sentence-BERT (`paraphrase-MiniLM-L6-v2`) and DeepSeek-R1:14B to map headers to predefined target columns. |
|
|
- **Data Cleaning & Validation:** |
|
|
- Filters valid phone numbers (7 to 15 digits). |
|
|
- Normalizes column names for consistency. |
|
|
- **Merging & Structuring:** Combines multiple datasets into a structured format. |
|
|
|
|
|
## Installation |
|
|
|
|
|
### Step 1: Create a Virtual Environment |
|
|
```sh |
|
|
python -m venv venv |
|
|
source venv/bin/activate # On macOS/Linux |
|
|
venv\Scripts\activate # On Windows |
|
|
``` |
|
|
|
|
|
### Step 2: Install Dependencies |
|
|
```sh |
|
|
pip install -r requirements.txt |
|
|
``` |
|
|
|
|
|
### Step 3: Ensure Required Models are Installed |
|
|
```sh |
|
|
python -m sentence_transformers.sentence_transformer |
|
|
``` |
|
|
|
|
|
## Usage |
|
|
|
|
|
### Running the Script |
|
|
```sh |
|
|
python main.py --root_directory path/to/csv/files |
|
|
``` |
|
|
|
|
|
### Arguments |
|
|
- `--root_directory`: Path to the folder containing CSV files. |
|
|
|
|
|
## Project Structure |
|
|
``` |
|
|
project/ |
|
|
│── 1_CleanLargeMultipleExcels.py # Main script |
|
|
``` |
|
|
|
|
|
## Functions |
|
|
|
|
|
### `read_csv_files(root_directory)` |
|
|
Recursively scans a directory to find all CSV files. |
|
|
|
|
|
### `calculate_similarity(source_list, target_list)` |
|
|
Computes cosine similarity between dataset headers and target headers. |
|
|
|
|
|
### `get_header_similarity(headers)` |
|
|
Uses DeepSeek AI to match dataset headers to predefined target columns. |
|
|
|
|
|
### `is_valid_mobile(value)` |
|
|
Checks if a given value is a valid phone number. |
|
|
|
|
|
### `merge_dataframes(file_paths)` |
|
|
Reads, maps, cleans, and merges CSV data. |
|
|
|
|
|
## Future Enhancements |
|
|
- **Parallel Processing:** Implement multiprocessing for handling large datasets. |
|
|
- **Improved Header Mapping:** Train a custom model for better accuracy. |
|
|
- **Batch Processing:** Process CSV files in chunks to optimize memory usage. |
|
|
|
|
|
## License |
|
|
This project is licensed under the MIT License. |
|
|
|
|
|
--- |
|
|
**NEEL N SONI:** |
|
|
|
