README.md

---
language: en
tags:
- question-answering
- squad
- gpt2
- fine-tuned
license: mit
---

# ChatMachine_v1: GPT-2 Fine-tuned on SQuAD

This model is a GPT-2 variant fine-tuned on the Stanford Question Answering Dataset (SQuAD) for question-answering tasks. It has been trained to understand context and generate relevant answers to questions based on provided information.

## Model Description

- **Base Model**: GPT-2 (124M parameters)
- **Training Data**: Stanford Question Answering Dataset (SQuAD)
- **Task**: Question Answering
- **Framework**: PyTorch with Hugging Face Transformers

## Training Details

The model was fine-tuned using:
- Mixed precision training (bfloat16)
- Learning rate: 2e-5
- Batch size: 16
- Gradient accumulation steps: 8
- Warmup steps: 1000
- Weight decay: 0.1

## Usage

```python
from transformers import GPT2LMHeadModel, GPT2Tokenizer

# Load model and tokenizer
model = GPT2LMHeadModel.from_pretrained("houcine-bdk/chatMachine_v1")
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
tokenizer.pad_token = tokenizer.eos_token

# Format your input
context = "Paris is the capital and largest city of France."
question = "What is the capital of France?"
input_text = f"Context: {context} Question: {question} Answer:"

# Generate answer
inputs = tokenizer(input_text, return_tensors="pt", padding=True)
outputs = model.generate(
    **inputs,
    max_new_tokens=50,
    temperature=0.3,
    do_sample=True,
    top_p=0.9,
    num_beams=4,
    early_stopping=True,
    pad_token_id=tokenizer.pad_token_id,
    eos_token_id=tokenizer.eos_token_id,
)

# Extract answer
answer = tokenizer.decode(outputs[0], skip_special_tokens=True).split("Answer:")[-1].strip()
print(f"Answer: {answer}")
```

## Performance and Limitations

The model performs best with:
- Simple, focused questions
- Clear, concise context
- Factual questions (who, what, when, where)

Limitations:
- May struggle with complex, multi-part questions
- Performance depends on the clarity and relevance of the provided context
- Best suited for short, focused answers rather than lengthy explanations

## Example Questions

```python
test_cases = [
    {
        "context": "George Washington was the first president of the United States, serving from 1789 to 1797.",
        "question": "Who was the first president of the United States?"
    },
    {
        "context": "The brain uses approximately 20 percent of the body's total energy consumption.",
        "question": "How much of the body's energy does the brain use?"
    }
]
```

Expected outputs:
- "George Washington"
- "20 percent"

## Training Infrastructure

The model was trained on an RTX 4090 GPU using:
- PyTorch with CUDA optimizations
- Mixed precision training (bfloat16)
- Gradient accumulation for effective batch size scaling

## Citation

If you use this model, please cite:

```bibtex
@misc{chatmachine_v1,
  author = {Houcine BDK},
  title = {ChatMachine_v1: GPT-2 Fine-tuned on SQuAD},
  year = {2024},
  publisher = {Hugging Face},
  journal = {Hugging Face Model Hub},
  howpublished = {\url{https://huggingface.co/houcine-bdk/chatMachine_v1}}
}
```

## License

This model is released under the MIT License.