---
library_name: transformers
pipeline_tag: fill-mask
tags:
  - bert
  - spiking-neural-network
  - masked-language-modeling
  - pytorch
license: apache-2.0
---

# SpikingLM

SpikingLM is a BERT-base style masked-language model with spiking attention blocks.

This checkpoint uses:

- temporal Spiking BERT with `T=4`
- learnable Q/K/V scaling parameters initialized from `7`
- LIF nodes for projection, Q, K, V, attention output, and MLP blocks
- `FP16OptimizedExp2Softmax` through `self.learnmax(attention_scores)` for attention normalization

## Files

```text
config.json
model.safetensors
tokenizer.json
tokenizer_config.json
special_tokens_map.json
vocab.txt
spiking_bert/modeling_spiking_bert.py
scripts/finetune_glue.py
requirements.txt
```

## Usage

```python
from safetensors.torch import load_file
from transformers import AutoConfig, AutoTokenizer
from spiking_bert import BertForMaskedLM

repo_or_path = "YOUR_USERNAME/SpikingLM"

config = AutoConfig.from_pretrained(repo_or_path)
config.T = 4
config._attn_implementation = "eager"

tokenizer = AutoTokenizer.from_pretrained(repo_or_path)
model = BertForMaskedLM(config)

state = load_file("model.safetensors")
model.load_state_dict(state)
```

If you clone the repository locally, replace `repo_or_path` with the local clone path and load `model.safetensors` from that directory.

## Results

Masked-language-model evaluation stored with the checkpoint:

```json
{"perplexity": 57.81198691730261}
```

## Notes

Large binary weights are stored as `model.safetensors`. The model code requires `torch`, `transformers`, `safetensors`, and `spikingjelly`.