license: mit
language:
- en
library_name: transformers
tags:
- gpt2
- question-answering
- reinforcement-learning
- ppo
- squad
- fine-tuned
datasets:
- rajpurkar/squad
base_model: openai-community/gpt2
pipeline_tag: text-generation
GPT2 Fine-tuned with Reinforcement Learning for Question Answering
This model is a GPT2 (openai-community/gpt2) fine-tuned using Reinforcement Learning (PPO) on the SQuAD dataset for question-answering tasks.
Model Description
- Base Model: openai-community/gpt2
- Training Method: Proximal Policy Optimization (PPO)
- Dataset: SQuAD (Stanford Question Answering Dataset)
- Task: Question Answering with formatted responses
- Language: English
Training Details
Reinforcement Learning Approach
This model was trained using PPO (Proximal Policy Optimization) with shaped rewards to encourage a specific response format:
Response Format:
- Starts with:
"That is a great question! " - Ends with:
" Let me know if you have any other questions."
Reward Shaping
| Reward | Condition |
|---|---|
| +5 | Response starts with correct prefix |
| +5 | Response ends with correct suffix |
| +3 | Contains meaningful content |
| +5 | Contains reference answer |
| -3 | Missing prefix or suffix |
Training Configuration
- Epochs: 3
- Batch Size: 1
- Learning Rate: 1e-5
- Max Sequence Length: 128
- Training Samples: 300 (from SQuAD)
Usage
Using Transformers
from transformers import AutoTokenizer, AutoModelForCausalLM
Load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained("StevenHuo/StevenHuo-gpt2-squad-rl") model = AutoModelForCausalLM.from_pretrained("StevenHuo/StevenHuo-gpt2-squad-rl")
Prepare input
question = "What is the capital of France?" context = "France is a country in Western Europe. Its capital is Paris, which is known for the Eiffel Tower." prompt = f"Question: {question}\nContext: {context}\nAnswer: That is a great question! "
Generate
inputs = tokenizer(prompt, return_tensors="pt") outputs = model.generate( **inputs, max_new_tokens=100, temperature=0.8, do_sample=True, top_k=50, top_p=0.95, pad_token_id=tokenizer.eos_token_id )
Decode
response = tokenizer.decode(outputs[0], skip_special_tokens=True) print(response)### Example Output
Input:
Question: What is the capital of France? Context: France is a country in Western Europe. Its capital is Paris, which is known for the Eiffel Tower.
Output: That is a great question! The capital of France is Paris. Let me know if you have any othertuning of language models
- Question-answering tasks with formatted responses
- Learning about PPO (Proximal Policy Optimization) for NLP
Limitations
- Based on GPT2 (124M parameters), which has limited reasoning capabilities
- Response format may not always be perfectly adhered to
- Training was done on a subset of SQuAD (300 samples)
- Best suited for simple factual questions