| --- |
| library_name: transformers |
| pipeline_tag: text-generation |
| license: apache-2.0 |
| language: |
| - en |
| tags: |
| - etl |
| - data-engineering |
| - databricks |
| - spark |
| - sql |
| - lora |
| - qlora |
| - qwen2.5 |
| - instruction-tuning |
| base_model: |
| - Qwen/Qwen2.5-1.5B-Instruct |
| --- |
| |
| # ETL Error Explainer v2 |
|
|
| ## Model Description |
|
|
| ETL Error Explainer v2 is a domain-specific instruction-tuned language model designed to analyze ETL and data pipeline failures. |
|
|
| Given an ETL error message and execution context (environment, severity, cloud provider), the model produces a structured JSON response containing: |
|
|
| - category |
| - root_cause |
| - immediate_fix |
| - long_term_fix |
|
|
| The model is intended to assist Data Engineers, Analytics Engineers, Platform Engineers, and DevOps teams during troubleshooting and incident response. |
|
|
| --- |
|
|
| # Model Details |
|
|
| ## Developed by |
|
|
| Fırat Çelik |
|
|
| ## Model type |
|
|
| Instruction-tuned causal language model |
|
|
| ## Base model |
|
|
| Qwen/Qwen2.5-1.5B-Instruct |
|
|
| ## Fine-tuning method |
|
|
| QLoRA (4-bit NF4 Quantization + LoRA) |
|
|
| ## Language |
|
|
| English |
|
|
| ## License |
|
|
| Apache-2.0 |
|
|
| --- |
|
|
| # Model Sources |
|
|
| ## Hugging Face Repository |
|
|
| https://huggingface.co/firfircelik/etl-error-explainer-v2 |
|
|
| ## Dataset |
|
|
| Synthetic ETL troubleshooting instruction dataset created by the author. |
|
|
| --- |
|
|
| # Intended Uses |
|
|
| ## Direct Use |
|
|
| The model is designed for structured ETL troubleshooting. |
|
|
| Example applications include: |
|
|
| - Spark error explanation |
| - SQL exception analysis |
| - Databricks job failures |
| - Azure Data Factory pipeline failures |
| - Kafka ingestion errors |
| - Snowflake query failures |
| - Schema evolution issues |
| - Authentication and permission errors |
| - Data validation failures |
|
|
| Input: |
|
|
| ``` |
| Context: |
| environment=prod |
| severity=High |
| cloud=Azure |
| |
| Analyze the following Databricks failure: |
| |
| org.apache.spark.sql.AnalysisException: |
| cannot resolve 'customer_email' |
| ``` |
|
|
| Output: |
|
|
| ```json |
| { |
| "category": "...", |
| "root_cause": "...", |
| "immediate_fix": "...", |
| "long_term_fix": "..." |
| } |
| ``` |
|
|
| --- |
|
|
| ## Downstream Uses |
|
|
| The model can be integrated into: |
|
|
| - AI Data Engineering copilots |
| - Internal troubleshooting assistants |
| - Incident response systems |
| - RAG applications |
| - Chatbots |
| - IDE assistants |
| - Knowledge base generation |
| - Support automation |
|
|
| --- |
|
|
| ## Out-of-Scope Uses |
|
|
| The model is **not** intended for: |
|
|
| - General-purpose chat |
| - Code generation |
| - SQL execution |
| - Production system diagnosis |
| - Security analysis |
| - Legal advice |
| - Medical advice |
|
|
| Outputs should always be reviewed before applying fixes to production environments. |
|
|
| --- |
|
|
| # Bias, Risks and Limitations |
|
|
| The model has been trained specifically for ETL troubleshooting. |
|
|
| Limitations include: |
|
|
| - It does not inspect live systems. |
| - It may suggest common fixes that are not applicable in every environment. |
| - It cannot validate infrastructure configurations. |
| - It may hallucinate missing context if insufficient information is provided. |
| - It should complement—not replace—human expertise. |
|
|
| --- |
|
|
| # Recommendations |
|
|
| Best performance is achieved when prompts include: |
|
|
| - Full error messages |
| - Stack traces |
| - Cloud provider |
| - Environment (dev/test/prod) |
| - Severity level |
| - Relevant SQL or Spark snippets |
|
|
| --- |
|
|
| # Getting Started |
|
|
| ```python |
| from transformers import AutoTokenizer, AutoModelForCausalLM |
| |
| model_id = "firfircelik/etl-error-explainer-v2" |
| |
| tokenizer = AutoTokenizer.from_pretrained(model_id) |
| model = AutoModelForCausalLM.from_pretrained(model_id) |
| |
| prompt = """ |
| Context: |
| environment=prod |
| severity=High |
| cloud=Azure |
| |
| Analyze the following Databricks failure: |
| |
| org.apache.spark.sql.AnalysisException: |
| cannot resolve 'customer_email' |
| """ |
| |
| inputs = tokenizer(prompt, return_tensors="pt") |
| |
| outputs = model.generate( |
| **inputs, |
| max_new_tokens=256 |
| ) |
| |
| print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
| ``` |
|
|
| --- |
|
|
| # Training Details |
|
|
| ## Base Model |
|
|
| Qwen/Qwen2.5-1.5B-Instruct |
|
|
| ## Fine-Tuning Method |
|
|
| QLoRA |
|
|
| - 4-bit NF4 quantization |
| - LoRA adapters |
| - PEFT |
|
|
| ## LoRA Configuration |
|
|
| | Parameter | Value | |
| |-----------|------:| |
| | r | 16 | |
| | alpha | 32 | |
| | dropout | 0.05 | |
| | target modules | q_proj, k_proj, v_proj, o_proj | |
|
|
| ## Optimizer |
|
|
| paged_adamw_8bit |
|
|
| ## Training Hyperparameters |
|
|
| | Parameter | Value | |
| |-----------|------:| |
| | Epochs | 3 | |
| | Batch Size | 4 | |
| | Gradient Accumulation | 4 | |
| | Effective Batch Size | 16 | |
| | Learning Rate | 2e-4 | |
| | Sequence Length | 512 | |
|
|
| --- |
|
|
| # Training Data |
|
|
| The model was trained on a synthetic instruction dataset covering realistic ETL failures. |
|
|
| Covered domains include: |
|
|
| - Apache Spark |
| - Databricks |
| - SQL |
| - Azure Data Factory |
| - Snowflake |
| - Kafka |
| - Delta Lake |
| - Data validation |
| - Schema evolution |
| - Authentication |
| - File ingestion |
| - Cloud storage |
|
|
| Each sample contains: |
|
|
| - execution context |
| - instruction |
| - structured target JSON |
|
|
| --- |
|
|
| # Evaluation |
|
|
| A held-out validation split (10%) was used during training. |
|
|
| Additionally, the model was manually tested using unseen ETL failure scenarios. |
|
|
| Evaluation focused on: |
|
|
| - JSON validity |
| - Root cause correctness |
| - Practical remediation quality |
| - Instruction following |
|
|
| --- |
|
|
| # Environmental Impact |
|
|
| ## Hardware |
|
|
| NVIDIA Tesla T4 |
|
|
| ## Platform |
|
|
| Kaggle Notebooks |
|
|
| ## Training Method |
|
|
| QLoRA (4-bit) |
|
|
| Using QLoRA significantly reduces GPU memory usage and energy consumption compared to full fine-tuning. |
|
|
| --- |
|
|
| # Technical Specifications |
|
|
| ## Architecture |
|
|
| - Transformer Decoder |
| - Causal Language Modeling |
|
|
| ## Objective |
|
|
| Instruction-following generation of structured ETL troubleshooting responses. |
|
|
| Output format: |
|
|
| ```json |
| { |
| "category": "...", |
| "root_cause": "...", |
| "immediate_fix": "...", |
| "long_term_fix": "..." |
| } |
| ``` |
|
|
| --- |
|
|
| # Supported Domains |
|
|
| | Domain | Supported | |
| |----------|-----------| |
| | Apache Spark | ✅ | |
| | Databricks | ✅ | |
| | SQL | ✅ | |
| | Azure Data Factory | ✅ | |
| | Kafka | ✅ | |
| | Snowflake | ✅ | |
| | Delta Lake | ✅ | |
| | Schema Evolution | ✅ | |
| | Authentication | ✅ | |
| | Data Validation | ✅ | |
| | Cloud Storage | ✅ | |
|
|
| --- |
|
|
| # Citation |
|
|
| ```bibtex |
| @misc{celik2026etlexplainer, |
| author = {Fırat Çelik}, |
| title = {ETL Error Explainer v2}, |
| year = {2026}, |
| publisher = {Hugging Face}, |
| howpublished = {\url{https://huggingface.co/firfircelik/etl-error-explainer-v2}} |
| } |
| ``` |
|
|
| --- |
|
|
| # Contact |
|
|
| GitHub |
|
|
| https://github.com/firfircelik |
|
|
| Hugging Face |
|
|
| https://huggingface.co/firfircelik |
