madImad/questcrafter-distilgpt2

🧙 QuestCrafter — Instruction-Tuned distilgpt2 (TinyStories) 📌 Overview

QuestCrafter is a fine-tuned version of distilgpt2 trained on a structured fantasy quest dataset derived from TinyStories.

The model generates short RPG-style quests conditioned on structured control tokens:

<LEVEL=X>

<SETTING=...>

<TONE=...>

<LENGTH=...>

This project was developed as part of the MSc AI Machine Learning course.

🗂 Dataset

Base dataset: TinyStories (Hugging Face)

We reformatted the dataset into an instruction-style supervised format:

<LEVEL=3> <SETTING=forest> <TONE=dark> <LENGTH=short>

Instruction:

Create a quest involving betrayal.

Response:

...

Train/Validation/Test split: 80/10/10.

🏗 Training Setup

Base model: distilgpt2

Framework: PyTorch + Hugging Face Transformers

Epochs: 3

Learning rate: 5e-5

Batch size: 8

Max length: 256

Mixed precision (fp16) enabled on GPU

🎮 Controllability

The model supports structured control tokens:

Level (1–10)

Setting (forest, desert city, medieval town, etc.)

Tone (epic, dark, humorous)

Length (short / medium)

This enables conditional generation and improved prompt-faithfulness compared to baseline distilgpt2.

📊 Evaluation

We compared baseline distilgpt2 vs fine-tuned model on a fixed test set of 50 prompts.

Metrics used:

Validation loss

Perplexity

Distinct-2 (diversity metric)

Human evaluation rubric (1–5 scale):

Coherence

Prompt-faithfulness

Creativity

The fine-tuned model showed:

Lower perplexity

Higher prompt-faithfulness

Improved structural consistency

🛡 Safety

A lightweight keyword-based safety filter is applied during generation to prevent harmful or inappropriate content. Unsafe generations are automatically regenerated.

⚠️ Limitations

May occasionally ignore control tokens

May produce repetitive text for long outputs

Not trained for real-world advisory use

Fantasy-only domain

🎓 Academic Context

This model was developed for the QuestCrafter AI project (MSc AI). It demonstrates:

Instruction-style fine-tuning

Controlled generation

Baseline vs tuned comparison

Structured evaluation pipeline