BEAT: Behavioral Encoder for Action Trajectories — initial release

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,94 @@
+---
+license: apache-2.0
+tags:
+  - behavioral-ai
+  - user-embeddings
+  - sequence-modeling
+  - e-commerce
+  - churn-prediction
+  - personalization
+pipeline_tag: feature-extraction
+---
+
+# BEAT: Behavioral Encoder for Action Trajectories
+
+A foundation transformer model that encodes sequences of human behavioral events into dense, reusable embeddings.
+
+## What is BEAT?
+
+Every company predicting churn, recommending products, or segmenting users starts by manually engineering features from behavioral data (RFM scores, click counts, session metrics). This feature engineering is where most prediction quality is lost.
+
+**BEAT eliminates that step.** Feed it raw event sequences — page views, purchases, searches, support tickets — and get a rich 768-dimensional embedding that captures the user's behavioral state.
+
+## Key Innovation
+
+Unlike text transformers (BERT, GPT) that encode language, BEAT is designed specifically for **action sequences with temporal dynamics**:
+
+- **Temporal encoding**: Learns from time gaps between events (a purchase 1 day after browsing means something different than 30 days after)
+- **Action vocabulary**: Encodes event types, not words
+- **Behavioral context**: Understands that the same action means different things in different sequences
+
+## Usage
+
+```python
+from transformers import AutoModel
+import torch
+
+# Load model
+model = AutoModel.from_pretrained("your-org/beat-encoder")
+
+# Encode a behavioral sequence
+action_ids = torch.tensor([[1, 2, 3, 5, 1, 6, 2, 5]])  # page_view, product_view, cart, purchase...
+property_ids = torch.tensor([[12, 45, 45, 45, 8, 3, 22, 22]])  # category/property context
+time_gaps = torch.tensor([[0.0, 0.1, 0.5, 1.2, 3.0, 3.1, 7.0, 7.5]])  # days between events
+
+outputs = model(action_ids, property_ids, time_gaps)
+embedding = outputs["embedding"]  # [1, 768] — user behavioral state
+```
+
+## Pre-training Objectives
+
+1. **Masked Event Prediction**: Randomly mask 15% of events, predict the action type (like MLM in BERT)
+2. **Next Event Prediction**: Given a sequence, predict what action comes next
+3. **Contrastive Learning**: Different time windows of the same user should produce similar embeddings
+
+## Downstream Tasks
+
+BEAT embeddings can be used for:
+
+| Task | Method | Expected Improvement |
+|------|--------|---------------------|
+| Churn prediction | Linear probe on embedding | +8-15% AUC vs. manual features |
+| User segmentation | Cluster embeddings | More stable, interpretable clusters |
+| Next-best-action | Fine-tune prediction head | Captures temporal patterns manual features miss |
+| Personalization | Nearest-neighbor in embedding space | Real behavioral similarity, not just demographics |
+
+## Training Data
+
+Pre-trained on the REES46 e-commerce behavioral dataset (20M+ events from a multi-category online store). The model generalizes to other behavioral domains through fine-tuning.
+
+## Architecture
+
+| Parameter | Value |
+|-----------|-------|
+| Hidden size | 768 |
+| Layers | 12 |
+| Attention heads | 12 |
+| Parameters | ~85M |
+| Embedding output | 768-dim |
+| Max sequence length | 256 events |
+| Temporal encoding | Learned + sinusoidal (90-day window) |
+
+## Citation
+
+```bibtex
+@article{beat2026,
+  title={BEAT: A Foundation Model for Human Behavioral Sequences},
+  author={Brijesh Dhanani},
+  year={2026},
+}
+```
+
+## License
+
+Apache 2.0

config.json

@@ -0,0 +1,27 @@
+{
+  "vocab_size": 512,
+  "max_properties": 128,
+  "max_seq_length": 256,
+  "hidden_size": 768,
+  "num_hidden_layers": 12,
+  "num_attention_heads": 12,
+  "intermediate_size": 3072,
+  "hidden_dropout_prob": 0.1,
+  "attention_probs_dropout_prob": 0.1,
+  "time_embedding_size": 64,
+  "max_time_gap": 90,
+  "pooling_strategy": "cls",
+  "masked_event_ratio": 0.15,
+  "next_event_prediction": true,
+  "contrastive_learning": true,
+  "learning_rate": 0.0001,
+  "weight_decay": 0.01,
+  "warmup_steps": 1000,
+  "max_steps": 100000,
+  "batch_size": 16,
+  "embedding_size": 768,
+  "model_type": "beat",
+  "architectures": [
+    "BEATModel"
+  ]
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:070d0792c4f14a816df7407bf9af75af94c4fd6a7976bcdf61444d0a55c53d0f
+size 345573360