| --- |
| title: IfGPT DataQualityComponents |
| emoji: ⚡ |
| colorFrom: gray |
| colorTo: green |
| sdk: static |
| pinned: false |
| --- |
| |
| # IfGPT DATASET Qiality Components |
|
|
| ## Objectives of the project IfGPT |
| The **IfGPT Qiality Pipeline** is developed within the project **IfGPT: Infrastructure for Fine-tuning Pre-trained Large Language Models** which aims to establish a freely accessible infrastructure for the selection and pre-processing of large datasets for Bulgarian as well as tailored data for particular industries and fine-tuning suitable freely available large language models for specific purposes. |
|
|
| ## IfGPT Dataset Quality Pipeline |
|
|
| Modular Java pipeline to process and add new text documents to the IfGPT Dataset, |
| which includes cleaning, deduplication and quality evaluation of Bulgarian texts. |
| The pipeline includes: |
|
|
| - **Source-specific extraction** — metadata and plain text extracted from heterogeneous |
| corpora (MARCELL, CURLICAT, BulNC, Wikipedia), each handled by a dedicated class |
| that implements the shared `SourceProcessor` interface and extends `BaseSourceProcessor` |
| - **Sentence splitting** — `BulgarianSentenceSplitter` wraps the Apache OpenNLP Bulgarian |
| UD sentence-detection model, splitting every document into a sentence-per-line sidecar |
| file used by all downstream stages |
| - **Boilerplate cleaning** — `FileCleanProcessor` learns site-specific boilerplate from a |
| sample directory (lines appearing in ≥ 50 % of files) and removes them alongside |
| hardcoded patterns for HTML tags, navigation menus, URLs, cookie banners, and more |
| - **MinHash / LSH deduplication** — `DeduplicationProcessor` builds a MinHash signature |
| index over the full existing corpus and detects near-duplicate sentences in the new |
| batch (Jaccard ≥ 0.90), writing a ranked TSV report and optionally removing duplicates |
| - **Per-sentence PII scoring** — `PIIDetector` runs every sentence through the Phileas |
| engine (names, emails, phone numbers, IBANs, IP addresses, etc.) and stores the |
| proportion of flagged tokens as the `PersonallyIdentifiableInformation` vector in metadata |
| - **Per-sentence bias scoring** — `BiasAnalyser` matches tokens against `BiasLexicon` |
| (3 787-entry Bulgarian Bias Dictionary v4) to detect signal–evaluator pairs across five |
| categories (gender, race/ethnicity, religion, disability, appearance), storing the |
| per-sentence coverage ratio as the `BiasedInformation` vector in metadata |
|
|
| The full schema is enforced by `DocumentMetadata` (15 mandatory + 8 optional fields) and |
| the complete flow is managed by `IfGPTPipeline`, with `IfGPTDatasetProcessor` as the |
| main entry point. |
|
|
| ``` |
| source processors → sentence split → clean → deduplication → PII → bias → counts → final structuring |
| ``` |
|
|
| ## License |
|
|
| Creative Commons Attribution 4.0 International (CC-BY-4.0) |
| __________________________________________ |
|
|
| This work is part of the project **Infrastructure for Fine-tuning Pre-trained Large Language Models**, Grant Agreement No. ПВУ – 55 from 12.12.2024 /BG-RRP-2.017-0030-C01/. |
|
|
| https://ifgpt.dcl.bas.bg/en/ |
|
|
