arXiv-WVY-43M

arXiv-WVY-43M is a 43.5M parameter causal language model developed by Starpower Technology.

Kaggle [https://www.kaggle.com/code/starpowertechnology/arxiv-wvy-43m-demo]

The model uses a compact DeepSeek-V3-style architecture and was trained from scratch on arXiv titles and abstracts.

This is a base language model, not an instruction-tuned or chat-tuned model.

Model Details

Property	Value
Parameters	43,489,608
Vocabulary	24,000
Hidden size	384
Transformer layers	6
MTP layers	1
Attention heads	6
MLP intermediate size	1,024
Routed experts	8
Experts selected per token	2
Shared experts	1
Q LoRA rank	128
KV LoRA rank	96
QK RoPE head dim	16
QK non-RoPE head dim	48
Value head dim	64
RoPE theta	10,000
YaRN factor	4.0
Original max position	1,024

Architecture

WVY-43M uses a compact DeepSeek-V3-style causal language model architecture containing:

• Multi-head latent attention
• Mixture-of-Experts layers
• 8 routed experts with Top-2 routing
• 1 shared expert
• Q/KV low-rank projections
• Rotary positional embeddings
• YaRN RoPE scaling
• Multi-Token Prediction layer

The architecture is intentionally kept small for research into compact language models, training from scratch, experimentation, and low-compute deployment.

Training

The model was pretrained from random initialization.

Training corpus: arXiv paper titles and abstracts

Observed training tokens: approximately 726 million

Checkpoint: step 5,600

The training corpus gives the model significant exposure to scientific and technical language, particularly terminology appearing in academic research.

Using the Model on Kaggle

Enable Internet access for the Kaggle notebook so the model can be downloaded from Hugging Face.

1. Install dependencies

!pip install -q -U transformers huggingface_hub tokenizers accelerate

2. Download and load WVY-43M

import os
import torch

from huggingface_hub import hf_hub_download
from transformers import (
    AutoConfig,
    AutoModelForCausalLM,
    PreTrainedTokenizerFast,
)

REPO_ID = "StarpowerTechnology/arXiv-WVY-43M"

# ---------------------------------------------------------
# Download tokenizer and weights
# ---------------------------------------------------------

tokenizer_path = hf_hub_download(
    repo_id=REPO_ID,
    filename="tokenizer.json"
)

weights_path = hf_hub_download(
    repo_id=REPO_ID,
    filename="model.pt"
)

# ---------------------------------------------------------
# Tokenizer
# ---------------------------------------------------------

tokenizer = PreTrainedTokenizerFast(
    tokenizer_file=tokenizer_path
)

# ---------------------------------------------------------
# Load the custom DeepSeek-V3 configuration from Hugging Face
# ---------------------------------------------------------

config = AutoConfig.from_pretrained(
    REPO_ID,
    trust_remote_code=True
)

# Build the model architecture without looking for
# pytorch_model.bin / safetensors because WVY uses model.pt.
model = AutoModelForCausalLM.from_config(
    config,
    trust_remote_code=True
)

# ---------------------------------------------------------
# Load WVY-43M weights
# ---------------------------------------------------------

checkpoint = torch.load(
    weights_path,
    map_location="cpu",
    weights_only=False
)

# Support common checkpoint formats.
if isinstance(checkpoint, dict):
    for key in ["state_dict", "model_state_dict", "model"]:
        if key in checkpoint and isinstance(checkpoint[key], dict):
            checkpoint = checkpoint[key]
            break

# Remove DataParallel prefix if present.
if (
    isinstance(checkpoint, dict)
    and len(checkpoint) > 0
    and all(k.startswith("module.") for k in checkpoint)
):
    checkpoint = {
        k[len("module."):]: v
        for k, v in checkpoint.items()
    }

model.load_state_dict(checkpoint, strict=True)

# ---------------------------------------------------------
# Device
# ---------------------------------------------------------

device = torch.device(
    "cuda" if torch.cuda.is_available() else "cpu"
)

model = model.to(device)
model.eval()

print("Model:", REPO_ID)
print("Device:", device)

params = sum(p.numel() for p in model.parameters())

print(f"Parameters: {params:,}")

Generate Text

Because arXiv-WVY-43M is a base causal language model, prompts can be passed directly as text.

prompt = "Quantum entanglement is"

inputs = tokenizer(
    prompt,
    return_tensors="pt",
    add_special_tokens=False
)

input_ids = inputs["input_ids"]

# WVY-43M configuration:
# BOS = 0
# EOS = 1

bos_id = config.bos_token_id

bos = torch.full(
    (input_ids.shape[0], 1),
    bos_id,
    dtype=torch.long
)

input_ids = torch.cat(
    [bos, input_ids],
    dim=1
).to(device)

attention_mask = torch.ones_like(input_ids)

with torch.no_grad():
    output = model.generate(
        input_ids=input_ids,
        attention_mask=attention_mask,
        max_new_tokens=150,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        eos_token_id=config.eos_token_id,
        pad_token_id=config.eos_token_id,
    )

generated_tokens = output[
    0,
    input_ids.shape[1]:
]

text = tokenizer.decode(
    generated_tokens,
    skip_special_tokens=True
)

print(text)

Simple Generation Function

def generate(
    prompt,
    max_new_tokens=150,
    temperature=0.7,
    top_p=0.9
):
    encoded = tokenizer(
        prompt,
        return_tensors="pt",
        add_special_tokens=False
    )

    input_ids = encoded["input_ids"]

    bos = torch.full(
        (input_ids.shape[0], 1),
        config.bos_token_id,
        dtype=torch.long
    )

    input_ids = torch.cat(
        [bos, input_ids],
        dim=1
    ).to(device)

    attention_mask = torch.ones_like(input_ids)

    with torch.no_grad():
        output = model.generate(
            input_ids=input_ids,
            attention_mask=attention_mask,
            max_new_tokens=max_new_tokens,
            do_sample=True,
            temperature=temperature,
            top_p=top_p,
            eos_token_id=config.eos_token_id,
            pad_token_id=config.eos_token_id,
        )

    generated = output[
        0,
        input_ids.shape[1]:
    ]

    return tokenizer.decode(
        generated,
        skip_special_tokens=True
    )


print(
    generate(
        "The relationship between gravity and spacetime is"
    )
)

Intended Use

arXiv-WVY-43M is intended for research involving:

• Small language models
• Language-model pretraining
• Scientific text generation
• Physics and technology language modeling
• Mixture-of-Experts architectures
• Low-parameter language-model experimentation
• Fine-tuning and continued pretraining
• Educational experiments with models trained from scratch

Limitations

WVY-43M contains approximately 43.5 million parameters and should be evaluated as a small experimental language model rather than as a replacement for modern large language models.

The released checkpoint is a base pretrained model and has not been instruction-tuned for assistant-style conversations.

No formal benchmark results are currently included in this model card.

License

MIT

Developer

Starpower Technology

Hugging Face organization: StarpowerTechnology