---
license: mit
language:
- en
pipeline_tag: text-classification
library_name: transformers
tags:
- custom_dataset
- High_quality
- Gpt
---
# GPT-Classification: Custom Transformer for Text Classification

This model is a custom **Transformer-based classifier** built from scratch using PyTorch. Unlike standard pre-trained models, this was designed with a specific focus on understanding character-level patterns for short to medium-length text classification.

## Model Architecture
- **Type:** GPT-style (Decoder-only architecture adapted for classification)
- **Layers:** 4 Transformer Blocks
- **Heads:** 4 Multi-Head Self-Attention
- **Embedding Dimension:** 128
- **Context Window:** 128 characters
- **Classification Head:** Linear layer applied to the mean of sequence embeddings.

## Tokenization
- **Level:** Character-level
- **Vocabulary Size:** 62 unique characters
- **Robustness:** The `encode` function is designed to ignore unknown characters to prevent runtime crashes during inference.

## Dataset Information
- **Source:** Custom JSONL dataset
- **Samples:** 9,999 after cleaning
- **Preprocessing:** Removed malformed template labels and handled various special characters.

## Files in this Folder
- `model.pt`: The PyTorch state dictionary containing the trained weights.
- `config.json`: Contains the exact hyperparameters, the character-to-index mapping (stoi), and the label mapping for inference.
- `README.md`: This documentation file.

## How to Use
1. Load the `config.json` to reconstruct the `stoi` mapping and model hyperparameters.
2. Initialize the `GPTClassification` class with the saved hyperparameters.
3. Load the weights using `torch.load('model.pt')`.
4. Ensure input strings are encoded using the character map and padded/truncated to 128 characters.