File size: 1,795 Bytes
58374f3 753c84a 58374f3 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 | ---
license: mit
language:
- en
pipeline_tag: text-classification
library_name: transformers
tags:
- custom_dataset
- High_quality
- Gpt
---
# GPT-Classification: Custom Transformer for Text Classification
This model is a custom **Transformer-based classifier** built from scratch using PyTorch. Unlike standard pre-trained models, this was designed with a specific focus on understanding character-level patterns for short to medium-length text classification.
## Model Architecture
- **Type:** GPT-style (Decoder-only architecture adapted for classification)
- **Layers:** 4 Transformer Blocks
- **Heads:** 4 Multi-Head Self-Attention
- **Embedding Dimension:** 128
- **Context Window:** 128 characters
- **Classification Head:** Linear layer applied to the mean of sequence embeddings.
## Tokenization
- **Level:** Character-level
- **Vocabulary Size:** 62 unique characters
- **Robustness:** The `encode` function is designed to ignore unknown characters to prevent runtime crashes during inference.
## Dataset Information
- **Source:** Custom JSONL dataset
- **Samples:** 9,999 after cleaning
- **Preprocessing:** Removed malformed template labels and handled various special characters.
## Files in this Folder
- `model.pt`: The PyTorch state dictionary containing the trained weights.
- `config.json`: Contains the exact hyperparameters, the character-to-index mapping (stoi), and the label mapping for inference.
- `README.md`: This documentation file.
## How to Use
1. Load the `config.json` to reconstruct the `stoi` mapping and model hyperparameters.
2. Initialize the `GPTClassification` class with the saved hyperparameters.
3. Load the weights using `torch.load('model.pt')`.
4. Ensure input strings are encoded using the character map and padded/truncated to 128 characters. |