---
license: mit
tags:
  - gpt2
  - causal-lm
  - pytorch
  - transformer
  - from-scratch
  - instruction-tuned
  - educational
  - small-llm
  - pocketGPT
language:
  - en
dataset:
  - custom-ml-corpus
  - custom-instruction-data
model_creator: your-name
paper: ""
---



# pocketGPT-27M — A Custom 27M Parameter GPT Model Trained From Scratch

**pocketGPT-27M** is a fully custom GPT-style language model, trained entirely from scratch using:

- A **24k Byte-Level BPE tokenizer**
- A **Transformer architecture** (10 layers, 6 heads, 384-d hidden size)
- A **384-token context window**
- **~165M-token pretraining corpus**
- **~4.5M-token instruction tuning dataset**

This project demonstrates how a compact GPT model can be designed, trained, and deployed end-to-end without relying on any pretrained weights.

---

## Model Highlights

### Architecture

| Component      | Value      |
|----------------|------------|
| Layers         | 10         |
| Hidden size    | 384        |
| Attention heads| 6          |
| FFN size       | 1536       |
| Vocab size     | 24,000     |
| Context length | 384        |
| Parameters     | ~27–35M    | (depending on tokenizer)

---

## Training Overview

### Pretraining
- **Objective:** Causal Language Modeling  
- **Dataset:** ~165M tokens of ML/AI literature  
- **Hardware:** NVIDIA T4 (Kaggle)  
- **Precision:** FP16  
- **Epochs:** 3  
- **Optimizer:** AdamW  

### Instruction Finetuning
- **Dataset:** ~4.5M tokens  
- **Format:**  <|bos|>Instruction: ... Response: ... <|eos|>
- **Purpose:** Improve conversational and Q&A ability  
- **Result:** Evaluation loss decreased consistently (no overfitting observed)

---

## Intended Use

- Educational LLMs  
- Lightweight research models  
- Offline/local small-scale chatbots  
- ML students exploring LLM training  

**Not intended for production or safety-critical use.**

---

## Usage Example

```python
from transformers import GPT2LMHeadModel, GPT2TokenizerFast
import torch
import os

os.environ["HUGGINGFACE_HUB_TOKEN"] = "Your Tokens"

model = GPT2LMHeadModel.from_pretrained("Amogh1221/PocketGPT_27M")
tokenizer = GPT2TokenizerFast.from_pretrained("Amogh1221/PocketGPT_27M")

def ask(prompt):
    formatted = f"<|bos|>Instruction: {prompt}\nResponse:"
    
    inputs = tokenizer.encode(formatted, return_tensors="pt")
    inputs = inputs.to(model.device) 

    with torch.no_grad():
        outputs = model.generate(
            inputs,
            max_length=384,
            do_sample=True,
            top_p=0.9,
            temperature=0.8,
            eos_token_id=tokenizer.eos_token_id,
            pad_token_id=tokenizer.pad_token_id
        )

    print(tokenizer.decode(outputs[0], skip_special_tokens=True))

ask("what is an Artificial Neural Network?")
```