README.md

---
library_name: transformers
pipeline_tag: text-generation
license: apache-2.0
language:
- en
tags:
- etl
- data-engineering
- databricks
- spark
- sql
- lora
- qlora
- qwen2.5
- instruction-tuning
base_model:
- Qwen/Qwen2.5-1.5B-Instruct
---

# ETL Error Explainer v2

## Model Description

ETL Error Explainer v2 is a domain-specific instruction-tuned language model designed to analyze ETL and data pipeline failures.

Given an ETL error message and execution context (environment, severity, cloud provider), the model produces a structured JSON response containing:

- category
- root_cause
- immediate_fix
- long_term_fix

The model is intended to assist Data Engineers, Analytics Engineers, Platform Engineers, and DevOps teams during troubleshooting and incident response.

---

# Model Details

## Developed by

Fırat Çelik

## Model type

Instruction-tuned causal language model

## Base model

Qwen/Qwen2.5-1.5B-Instruct

## Fine-tuning method

QLoRA (4-bit NF4 Quantization + LoRA)

## Language

English

## License

Apache-2.0

---

# Model Sources

## Hugging Face Repository

https://huggingface.co/firfircelik/etl-error-explainer-v2

## Dataset

Synthetic ETL troubleshooting instruction dataset created by the author.

---

# Intended Uses

## Direct Use

The model is designed for structured ETL troubleshooting.

Example applications include:

- Spark error explanation
- SQL exception analysis
- Databricks job failures
- Azure Data Factory pipeline failures
- Kafka ingestion errors
- Snowflake query failures
- Schema evolution issues
- Authentication and permission errors
- Data validation failures

Input:

```
Context:
environment=prod
severity=High
cloud=Azure

Analyze the following Databricks failure:

org.apache.spark.sql.AnalysisException:
cannot resolve 'customer_email'
```

Output:

```json
{
  "category": "...",
  "root_cause": "...",
  "immediate_fix": "...",
  "long_term_fix": "..."
}
```

---

## Downstream Uses

The model can be integrated into:

- AI Data Engineering copilots
- Internal troubleshooting assistants
- Incident response systems
- RAG applications
- Chatbots
- IDE assistants
- Knowledge base generation
- Support automation

---

## Out-of-Scope Uses

The model is **not** intended for:

- General-purpose chat
- Code generation
- SQL execution
- Production system diagnosis
- Security analysis
- Legal advice
- Medical advice

Outputs should always be reviewed before applying fixes to production environments.

---

# Bias, Risks and Limitations

The model has been trained specifically for ETL troubleshooting.

Limitations include:

- It does not inspect live systems.
- It may suggest common fixes that are not applicable in every environment.
- It cannot validate infrastructure configurations.
- It may hallucinate missing context if insufficient information is provided.
- It should complement—not replace—human expertise.

---

# Recommendations

Best performance is achieved when prompts include:

- Full error messages
- Stack traces
- Cloud provider
- Environment (dev/test/prod)
- Severity level
- Relevant SQL or Spark snippets

---

# Getting Started

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "firfircelik/etl-error-explainer-v2"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

prompt = """
Context:
environment=prod
severity=High
cloud=Azure

Analyze the following Databricks failure:

org.apache.spark.sql.AnalysisException:
cannot resolve 'customer_email'
"""

inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=256
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

---

# Training Details

## Base Model

Qwen/Qwen2.5-1.5B-Instruct

## Fine-Tuning Method

QLoRA

- 4-bit NF4 quantization
- LoRA adapters
- PEFT

## LoRA Configuration

| Parameter | Value |
|-----------|------:|
| r | 16 |
| alpha | 32 |
| dropout | 0.05 |
| target modules | q_proj, k_proj, v_proj, o_proj |

## Optimizer

paged_adamw_8bit

## Training Hyperparameters

| Parameter | Value |
|-----------|------:|
| Epochs | 3 |
| Batch Size | 4 |
| Gradient Accumulation | 4 |
| Effective Batch Size | 16 |
| Learning Rate | 2e-4 |
| Sequence Length | 512 |

---

# Training Data

The model was trained on a synthetic instruction dataset covering realistic ETL failures.

Covered domains include:

- Apache Spark
- Databricks
- SQL
- Azure Data Factory
- Snowflake
- Kafka
- Delta Lake
- Data validation
- Schema evolution
- Authentication
- File ingestion
- Cloud storage

Each sample contains:

- execution context
- instruction
- structured target JSON

---

# Evaluation

A held-out validation split (10%) was used during training.

Additionally, the model was manually tested using unseen ETL failure scenarios.

Evaluation focused on:

- JSON validity
- Root cause correctness
- Practical remediation quality
- Instruction following

---

# Environmental Impact

## Hardware

NVIDIA Tesla T4

## Platform

Kaggle Notebooks

## Training Method

QLoRA (4-bit)

Using QLoRA significantly reduces GPU memory usage and energy consumption compared to full fine-tuning.

---

# Technical Specifications

## Architecture

- Transformer Decoder
- Causal Language Modeling

## Objective

Instruction-following generation of structured ETL troubleshooting responses.

Output format:

```json
{
  "category": "...",
  "root_cause": "...",
  "immediate_fix": "...",
  "long_term_fix": "..."
}
```

---

# Supported Domains

| Domain | Supported |
|----------|-----------|
| Apache Spark | ✅ |
| Databricks | ✅ |
| SQL | ✅ |
| Azure Data Factory | ✅ |
| Kafka | ✅ |
| Snowflake | ✅ |
| Delta Lake | ✅ |
| Schema Evolution | ✅ |
| Authentication | ✅ |
| Data Validation | ✅ |
| Cloud Storage | ✅ |

---

# Citation

```bibtex
@misc{celik2026etlexplainer,
  author = {Fırat Çelik},
  title = {ETL Error Explainer v2},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/firfircelik/etl-error-explainer-v2}}
}
```

---

# Contact

GitHub

https://github.com/firfircelik

Hugging Face

https://huggingface.co/firfircelik