update readme

README.md

@@ -1,3 +1,152 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+---
+
+# Polish Text Quality Classifier
+
+## 1. Overview
+
+This project provides a high-performance script for classifying the quality of Polish texts using a pre-trained XGBoost model. The classifier assigns one of three quality categories (`LOW`, `MEDIUM`, `HIGH`) to each text and provides a confidence score (probability).
+
+The classification is based on over 200 linguistic features extracted from each text, such as the count of nouns and verbs, NER entities, sentence length statistics, and the number of out-of-vocabulary words. These features are calculated by a companion module (`predictor_parquet_spacy.py`).
+
+The script is designed for efficient, large-scale data processing. It leverages Python's `multiprocessing` to parallelize computations across all available CPU cores, significantly speeding up the analysis of large datasets. It supports processing files in both **Parquet** and **JSONL** formats.
+
+## 2. Features
+
+- **High-Performance Processing**: Utilizes `multiprocessing` to process texts in parallel, maximizing CPU usage.
+- **Dual Format Support**: Ingests data from either `.parquet` or `.jsonl` files.
+- **Robust Feature Extraction**: Relies on a sophisticated feature engineering module (`predictor_parquet_spacy.py`) to generate over 200 linguistic metrics for accurate classification.
+- **Scalable**: Capable of handling millions of documents by processing files sequentially and texts in parallel.
+- **Seamless Integration**: Appends classification results (`quality_ai` and `confidence`) directly to the original data, preserving all existing columns/keys.
+- **User-Friendly Progress**: Displays a `tqdm` progress bar to monitor the analysis in real-time.
+
+## 3. How It Works
+
+The script follows a simple yet powerful workflow:
+
+1.  **Model Loading**: At startup, the script loads the XGBoost model (`model.joblib`) and the feature scaler (`scaler.pkl`) into memory. This is done only once to avoid I/O overhead during processing.
+2.  **File Discovery**: It scans a specified input directory (`input_parquet/` or `input_jsonl/`) for files to process.
+3.  **Data Ingestion**:
+    -   For **Parquet files**, it reads the data into a pandas DataFrame using the efficient `pyarrow` engine.
+    -   For **JSONL files**, it streams the file line by line, parsing each line as a separate JSON object.
+4.  **Parallel Processing**: The core task of text analysis is distributed across a pool of worker processes.
+    -   A list of texts is extracted from the input file.
+    -   Each text is sent to a worker process.
+    -   The worker process uses the `process_text_file` function to compute the 200+ linguistic features.
+    -   These features are then scaled and fed into the XGBoost model to predict the quality category and confidence score.
+5.  **Output Generation**:
+    -   The results (category and confidence) are collected from all worker processes.
+    -   The script appends two new fields to each original record:
+        -   `quality_ai`: The predicted category (`LOW`, `MEDIUM`, or `HIGH`).
+        -   `confidence`: The model's confidence score for the prediction (e.g., `95.5`).
+    -   The enriched data is saved to a new file in the output directory (`output/`), preserving the original data structure.
+
+## 4. Setup and Installation
+
+### Prerequisites
+
+- Python 3.8+
+- Required Python packages
+
+### Installation
+
+1.  **Clone the repository (if applicable):**
+    ```bash
+    git clone <your-repo-url>
+    cd <your-repo-name>
+    ```
+
+2.  **Install the dependencies:**
+    A `requirements.txt` file is recommended. Create one with the following content:
+    ```
+    joblib
+    pandas
+    pyarrow
+    scikit-learn
+    xgboost
+    tqdm
+    # Add spacy and any other dependencies from predictor_parquet_spacy.py
+    spacy
+    ```
+    Then install them:
+    ```bash
+    pip install -r requirements.txt
+    ```
+    If you don't have a `requirements.txt` file, install the packages manually:
+    ```bash
+    pip install joblib pandas pyarrow scikit-learn xgboost tqdm spacy
+    python -m spacy download pl_core_news_md 
+    ```
+
+3.  **Download SpaCy Model:**
+    The feature extraction module likely requires a SpaCy model for Polish. Download it via the command line:
+    ```bash
+    python -m spacy download pl_core_news_lg
+    ```
+
+### Directory Structure
+
+Ensure your project follows this structure:
+
+```
+.
+├── input_parquet/
+│   └── docs1.parquet
+│   └── docs2.parquet
+├── input_jsonl/
+│   └── data1.jsonl
+│   └── data2.jsonl
+├── models/
+│   ├── model.joblib           # The trained XGBoost model
+│   └── scaler.pkl             # The scikit-learn scaler
+├── output/                    # Output directory for processed files
+├── main_script.py             # The main processing script
+└── predictor_parquet_spacy.py # The feature extraction module
+```
+
+## 5. Usage
+
+The script is configured to run out-of-the-box. Simply place your data files in the appropriate input directory and run the script.
+
+### Step 1: Place Your Data Files
+
+-   For **Parquet** files, place them in the `input_parquet/` directory. The script expects a column named `text` in each file.
+-   For **JSONL** files, place them in the `input_jsonl/` directory. The script expects a key named `text` in each JSON object.
+
+### Step 2: Run the Script
+
+Open your terminal and execute the Python script:
+
+```bash
+python -W ignore main_parquet_spacy_jsonl.py
+or
+python -W ignore main_parquet_spacy_parquet.py
+```
+
+### Step 3: Check the Output
+
+The script will create an `output/` directory (if it doesn't exist) and save the processed files there. Each output file will be named `calculated_<original_filename>`.
+
+For example, `input_parquet/docs.parquet` will be processed and saved as `output/calculated_docs.parquet`.
+
+The script automatically skips files that have already been processed and exist in the output directory, making it safe to re-run.
+
+### Example Console Output
+
+```
+
+python -W ignore main_parquet_spacy_parquet.py
+Znaleziono 1 plików w folderze wejściowym
+input_parquet\docs.parquet
+
+Podsumowanie:
+- Plików do przetworzenia: 1
+- Plików już przetworzonych: 0
+
+Rozpoczynanie przetwarzania 1 plików...
+Dodawanie do kolejki: input_parquet\docs.parquet
+<dataframe head>
+Processing time: 47.61986541748047 seconds
+Wszystkie pliki zostały przetworzone!
+```