Nefflymicn/amazon-sentiment-transformer

Custom Transformer for Amazon Sentiment Analysis

This repository contains a custom-built Transformer Encoder model for binary sentiment classification, trained on the Amazon Polarity dataset.

🚀 Model Overview

Unlike standard pre-trained models (like BERT), this architecture was built from scratch to demonstrate the implementation of Self-Attention and Positional Encodings in PyTorch.

• Architecture: 4-Layer Transformer Encoder
• Task: Binary Sentiment Analysis (Positive/Negative)
• Accuracy: 89.73% on Test Set
• Parameters: Optimized for efficient inference on edge devices

🛠️ Technical Specifications

• Embedding Dimension: 128
• Attention Heads: 8
• Feed-Forward Dimension: 512
• Sequence Length: 300 tokens
• Optimizer: AdamW with Linear Learning Rate Warmup

💻 Training Environment

This model was trained locally on an Apple Mac mini M4 with 24GB of Unified Memory.

• Accelerator: Metal Performance Shaders (MPS)
• Training Time: ~1.5 hours
• Dataset: Subset of 500,000 samples from Amazon Polarity

📈 Performance & Insights

During development, the model was benchmarked against a Bidirectional LSTM. The Transformer architecture achieved a ~5% improvement in accuracy, demonstrating its superior ability to capture long-range dependencies in product reviews.

📖 How to Use

To use this model, ensure you have torch and transformers installed.

from transformers import DistilBertTokenizer
import torch

# 1. Initialize Tokenizer
tokenizer = DistilBertTokenizer.from_pretrained('Nefflymicn/amazon-sentiment-transformer')

# 2. Load Model (Architecture must match)
model = TransformerSentimentModel(
            vocab_size=tokenizer.vocab_size,
            embed_dim=128,
            num_heads=8,
            ff_dim=512,
            num_layers=4,
            output_dim=2
        ) 
model.load_state_dict(torch.load("pytorch_model.bin", map_location='cpu'))
model.eval()