modern-BERT-MTG-Commander

A fine-tune of modern-BERT trained in a Siamese Network (Bi-Encoder) architecture on card data to predict how likely it is to be included in a Commander deck. This models an embedding space that has a much richer semantic understanding of Magic cards than the base model

Before

Base model neighbors for 'Yuriko, the Tiger's Shadow':

1. Nashi, Moon Sage's Scion (cos=0.9907)
2. Fallen Shinobi (cos=0.9889)
3. A-Nashi, Moon Sage's Scion (cos=0.9889)
4. Walker of Secret Ways (cos=0.9884)
5. Orochi Soul-Reaver (cos=0.9883)
6. A-Dokuchi Silencer (cos=0.9875)
7. Higure, the Still Wind (cos=0.9874)
8. Prosperous Thief (cos=0.9870)
9. Ninja of the Deep Hours (cos=0.9869)
10. Zareth San, the Trickster (cos=0.9868)

(high density, low relevance)

After

Fine-tuned neighbors for 'Yuriko, the Tiger's Shadow':

1. Ingenious Infiltrator (cos=0.7685)
2. Higure, the Still Wind (cos=0.7446)
3. Moon-Circuit Hacker (cos=0.7341)
4. Dokuchi Silencer (cos=0.7043)
5. A-Dokuchi Silencer (cos=0.7041)
6. Covert Technician (cos=0.7028)
7. Sakashima's Student (cos=0.6982)
8. Ninja of the Deep Hours (cos=0.6972)
9. A-Prosperous Thief (cos=0.6883)
10. Azra Smokeshaper (cos=0.6842)

(low density, high relevance)

Usage

import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer

def mean_pool(model_output, attention_mask):
    token_embeddings = model_output.last_hidden_state
    mask = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    summed = torch.sum(token_embeddings * mask, dim=1)
    counts = torch.clamp(mask.sum(dim=1), min=1e-9)
    return summed / counts

tokenizer = AutoTokenizer.from_pretrained("nishtahir/modern-BERT-MTG-Commander")
model = AutoModel.from_pretrained(model_name).eval()

# Cards are serialized in the Scryfall Text Format
card_data = """Commit {3}{U}
Instant
Put target spell or nonland permanent into its owner's library second from the top.
----
Memory {4}{U}{U}
Sorcery
Aftermath (Cast this spell only from your graveyard. Then exile it.)
Each player shuffles their hand and graveyard into their library, then draws seven cards.
"""

encoded = tokenizer([card_data],padding=True, truncation=True, return_tensors="pt")

ids = encoded["input_ids"].to(device)
mask = encoded["attention_mask"].to(device)
with torch.no_grad():
    outputs = model(ids, mask)

    # Mean pool to get a single embedding vector
    pooled = mean_pool(outputs, mask)

    # Normalize
    embeddings = F.normalize(torch.cat(all_vecs, dim=0), p=2, dim=1)