vanishingMonk/Contrastive_Explainer

ReCon: Contrastive Explanation Model for OWL Reasoning (LoRA)

Model Details

Model Description

ReCon is a LoRA fine-tuned language model based on meta-llama/Llama-3.1-8B-Instruct, designed to generate contrastive natural language explanations for OWL reasoning tasks.

The model translates structured outputs from a Description Logic (DL) reasoner into human-readable explanations that answer:

“Why does fact A satisfy a class expression while foil B does not?”

Unlike standard explanations, ReCon focuses on contrastive reasoning, highlighting:

• Common evidence (shared properties)
• Differences (missing or distinguishing properties)
• Conflicts (inconsistencies)

This work is part of a system combining:

• Symbolic reasoning (OWL reasoner)
• Natural language explanation (LLM)
• Graph-based visualization

---

Key Contributions

• Converts formal OWL reasoning outputs (JSON) into natural language

• Reduces hallucination via strict evidence-grounded prompting

• Supports contrastive explanations (fact vs foil)

• Integrated into:

  • Flask-based UI

---

Model Information

• Developed by: Akash, Ashik and Sam (DICE Group, Paderborn University)
• Model type: Causal Language Model (LoRA adapted)
• Base model: meta-llama/Llama-3.1-8B-Instruct
• Language(s): English (primary), German (supported)
• License: Same as base model (Llama 3.1 license)

---

Intended Use

Direct Use

• OWL reasoning explanation generation
• Contrastive reasoning (fact vs foil)
• Explainable AI (XAI) workflows
• Knowledge graph interpretation

Downstream Use

• Integration with reasoning pipelines
• Explainability layer for symbolic + neural systems
• Educational tools for ontology reasoning

Out-of-Scope Use

• General chatbot usage
• Open-ended reasoning without structured input
• Tasks requiring external world knowledge

---

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model = "meta-llama/Llama-3.1-8B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(base_model)
base = AutoModelForCausalLM.from_pretrained(base_model)

model = PeftModel.from_pretrained(base, "vanishingMonk/Contrastive_Explainer")

---

Training Details

Training Objective

The model is fine-tuned to generate faithful, structured explanations grounded strictly in reasoning evidence, rather than free-form generation.

---

Training Data

Training data was constructed from OWL reasoning tasks using datasets such as:

• Family ontology
• Carcinogenesis
• Lymphography

Each sample includes:

• Class expression

• Fact–foil pair

• Structured reasoning output:

  • common
  • different
  • conflicts

Both formats were used:

• Instruction-based samples
• Multi-turn chat-based samples

---

Prompt Engineering

A strict three-layer prompt structure was used:

1. System Prompt

   • Enforces evidence-only reasoning
   • Controls output format
   • Language constraints

2. Role Prompt

   • Defines model as:

     "Formal logician + OWL reasoner debugger"

3. Context Prompt

   • Defines semantics of:

     • common / different / conflicts

   • Enforces exact reasoning patterns

   • Fixes verdict:

     • FACT = satisfies
     • FOIL = fails

This significantly reduces hallucinations and improves logical consistency.

---

Training Setup

• Method: LoRA (PEFT)

• Rank (r): 8

• Alpha: 16

• Target modules:

  • q_proj, v_proj, o_proj
  • gate_proj, up_proj, down_proj

• Precision: bf16

• Framework: Torchtune

---

Evaluation

General Reasoning Benchmarks

Benchmark	Metric	Fine-tuned	Base
ARC-Challenge	Accuracy	0.544	0.553
HellaSwag (EN)	Accuracy	0.790	0.796
HellaSwag (DE)	Accuracy	0.615	0.615
TruthfulQA	Accuracy	0.533	0.545

→ Minimal degradation in general reasoning performance

---

Explanation Quality

Metric	Fine-tuned	Base
ROUGE-1	0.4066	0.5845
ROUGE-2	0.1424	0.3443
ROUGE-L	0.2981	0.4013
BERTScore	0.8475	0.8705

Important: Lower ROUGE/BERTScore reflects stylistic differences, not worse reasoning. Human evaluation shows higher logical fidelity and stability.

---

System Integration

ReCon is part of a full pipeline:

1. OWL reasoner generates contrastive explanation (JSON)

2. ReCon converts it into natural language

3. Graphviz generates a visual explanation

4. UI presents:

   • Explanation
   • Graph
   • Chat interface

Also integrated into:

• Flask web interface
• Protégé plugin

---

Limitations

• Requires structured reasoning input
• Cannot perform standalone logical inference
• Sensitive to prompt format
• May hallucinate if constraints are relaxed

---

Future Work

• Scaling to larger ontologies
• Support for more DL constructs

---

Citation

This model is part of an academic project at Paderborn University. If you use this model, please reference this repository.

---

Contact

For questions, collaborations, or research discussions, feel free to reach out to Akash. Email: akbaum@mail.uni-paderborn.de