CLSE-v1 by Lloid

Custom Encoder Transformer trained on SST-2 sentiment classification from scratch. Accuracy: 77% on SST-2 validation set.

Usage

import torch
from transformers import AutoTokenizer

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("Masharyy/CLSE-v1")

# Load model (copy the Model class from the repo)
model = Model(vocab_size=30522, d_model=256, n_heads=8, N_layers=4, T=128, out_features=2)
model.load_state_dict(torch.load("model.pth", map_location="cpu"))
model.eval()

# Inference
sentence = "This movie was great!"
inputs = tokenizer(sentence, return_tensors="pt", padding="max_length", 
                   truncation=True, max_length=128)
with torch.no_grad():
    logits = model(inputs["input_ids"])
    pred = torch.argmax(logits, dim=-1).item()

print("Positive" if pred == 1 else "Negative")

Architecture

• 4 Encoder layers
• 8 attention heads
• d_model: 256
• Trained from scratch on SST-2