---
language:
- id
- en
pipeline_tag: feature-extraction
tags:
- spiking-neural-network
- snn
- sentence-similarity
- neuromorphic
- hebbian-learning
- indonesian
license: apache-2.0
---

# Spiking Sentence Embedder (PulseNet-Labs)

This is the official PyTorch/HuggingFace implementation of the **Spiking Sentence Embedder** featuring **Sparse Coincidence-Based Semantic Attention**. The model was originally implemented in Rust to simulate true neuromorphic hardware constraints and has been carefully ported to PyTorch to guarantee 100% mathematical bit-exact parity for seamless deployment.

## Model Details
- **Architecture**: Spiking Neural Network (SNN) with Leaky-Integrate-and-Fire (LIF) neurons.
- **Layers**: 
  - Token-level Temporal Embedding
  - Coincidence-Based Semantic Attention
  - Dense BPTT Pooler with residual dynamics
- **Task**: Semantic Textual Similarity / Sentence Embeddings
- **Languages**: Indonesian (ID), English (EN)
- **Training Paradigm**: Knowledge Distillation via Hebbian Plasticity (Contrastive Hebbian Learning) from a teacher model.
- **Publication / DOI**: [10.5281/zenodo.20743764](https://doi.org/10.5281/zenodo.20743764)

## Usage

This model requires custom architecture code (`modeling_spiking.py`) to run. You must set `trust_remote_code=True` when loading the model.

```python
import torch
import torch.nn.functional as F
from transformers import AutoTokenizer, AutoModel

# 1. Load Tokenizer and Spiking Model
tokenizer = AutoTokenizer.from_pretrained("PulseNet-Labs/spiking-sentence-embedder", trust_remote_code=True)
model = AutoModel.from_pretrained("PulseNet-Labs/spiking-sentence-embedder", trust_remote_code=True)
model.eval()

# 2. Input sentences
sentences = [
    "Sistem neuromorfik ini sangat hemat energi.",
    "Jaringan saraf spiking mengonsumsi daya yang rendah."
]

# 3. Tokenize
inputs = tokenizer(sentences, padding="max_length", max_length=128, truncation=True, return_tensors="pt")

# Convert PAD tokens to 0 to align with SNN initialization behavior
inputs.input_ids[inputs.input_ids == tokenizer.pad_token_id] = 0

# 4. Forward Pass (Temporal SNN Simulation)
with torch.no_grad():
    embeddings = model(**inputs)

# 5. Compute Pearson/Cosine Similarity
# Note: For strict SNN metric space validation, mean-centering is recommended
emb_centered = embeddings - embeddings.mean(dim=-1, keepdim=True)
similarity = F.cosine_similarity(emb_centered[0].unsqueeze(0), emb_centered[1].unsqueeze(0))

print(f"Semantic Similarity: {similarity.item():.4f}")
```

## Performance & Benchmarks
Tested on the bilingual STS-B dataset:
- **Pearson Correlation (vs Teacher)**: 0.7514
- Achieves zero-shot generalization with highly sparse binary activations, drastically reducing theoretical energy consumption compared to dense attention counterparts.

## Citing & Authors
If you use this model in your research, please refer to our DOI manuscript: [https://doi.org/10.5281/zenodo.20739462](https://doi.org/10.5281/zenodo.20739462).
**Organization:** [PulseNet-Labs](https://huggingface.co/PulseNet-Labs)