software-ses
/

RavenBERT

@@ -1,144 +1,93 @@
 ---
 tags:
 - sentence-transformers
-- sentence-similarity
-- feature-extraction
-- dense
-pipeline_tag: sentence-similarity
 library_name: sentence-transformers
 ---
-# SentenceTransformer
-This is a [sentence-transformers](https://www.SBERT.net) model trained. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
-## Model Details
-### Model Description
-- **Model Type:** Sentence Transformer
-<!-- - **Base model:** [Unknown](https://huggingface.co/unknown) -->
-- **Maximum Sequence Length:** 512 tokens
-- **Output Dimensionality:** 768 dimensions
-- **Similarity Function:** Cosine Similarity
-<!-- - **Training Dataset:** Unknown -->
-<!-- - **Language:** Unknown -->
-<!-- - **License:** Unknown -->
-### Model Sources
-- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
-- **Repository:** [Sentence Transformers on GitHub](https://github.com/huggingface/sentence-transformers)
-- **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers)
-### Full Model Architecture
-```
-SentenceTransformer(
-  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'RobertaModel'})
-  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
-  (2): Normalize()
-)
-```
-## Usage
-### Direct Usage (Sentence Transformers)
-First install the Sentence Transformers library:
-```bash
-pip install -U sentence-transformers
-```
-Then you can load this model and run inference.
 ```python
 from sentence_transformers import SentenceTransformer
-# Download from the 🤗 Hub
 model = SentenceTransformer("MojtabaEshghie/RavenBERT")
-# Run inference
 sentences = [
-    'The weather is lovely today.',
-    "It's so sunny outside!",
-    'He drove to the stadium.',
 ]
-embeddings = model.encode(sentences)
-print(embeddings.shape)
-# [3, 768]
-# Get the similarity scores for the embeddings
-similarities = model.similarity(embeddings, embeddings)
-print(similarities)
-# tensor([[1.0000, 0.5997, 0.5681],
-#         [0.5997, 1.0000, 0.8354],
-#         [0.5681, 0.8354, 1.0000]])
 ```
-<!--
-### Direct Usage (Transformers)
-<details><summary>Click to see the direct usage in Transformers</summary>
-</details>
--->
-<!--
-### Downstream Usage (Sentence Transformers)
-You can finetune this model on your own dataset.
-<details><summary>Click to expand</summary>
-</details>
--->
-<!--
-### Out-of-Scope Use
-*List how the model may foreseeably be misused and address what users ought not to do with the model.*
--->
-<!--
-## Bias, Risks and Limitations
-*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
--->
-<!--
-### Recommendations
-*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
--->
-## Training Details
-### Framework Versions
-- Python: 3.12.9
-- Sentence Transformers: 5.1.2
-- Transformers: 4.57.1
-- PyTorch: 2.9.0
-- Accelerate: 1.11.0
-- Datasets: 4.4.0
-- Tokenizers: 0.22.1
 ## Citation
-### BibTeX
-<!--
-## Glossary
-*Clearly define terms in order to be accessible across audiences.*
--->
-<!--
-## Model Card Authors
-*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
--->
-<!--
-## Model Card Contact
-*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
--->

+Here’s a concise **README.md** you can drop into the model repo.
+````markdown
 ---
 tags:
 - sentence-transformers
+- embeddings
+- roberta
+- code
+- solidity
+- ethereum
+- smart-contracts
+- security
 library_name: sentence-transformers
+pipeline_tag: sentence-similarity
+base_model: web3se/SmartBERT-v2
+model-index:
+- name: RavenBERT
+  results: []
 ---
+# RavenBERT
+RavenBERT is a **SentenceTransformers** embedding model specialized for **smart-contract invariants** (e.g., `require(...)`, `assert(...)`, `if (...) revert`) from Ethereum/Vyper sources.
+It starts from **`web3se/SmartBERT-v2`** and is **contrastively fine-tuned** so that cosine similarity reflects *semantic intent* of guards used in transaction-reverting checks.
+- **Architecture:** BERT-family encoder (SmartBERT-v2) → MeanPooling → L2 Normalize
+- **Embedding dimension:** 768
+- **Normalization:** Enabled (unit-norm vectors; cosine ≡ dot product)
+- **Intended use:** clustering / semantic search / dedup / taxonomy building for short guard predicates (and optional messages)
+## Quick start
 ```python
 from sentence_transformers import SentenceTransformer
 model = SentenceTransformer("MojtabaEshghie/RavenBERT")
 sentences = [
+    "amountOut >= amountOutMin",
+    "deadline >= block.timestamp",
+    "balances[msg.sender] >= amount"
 ]
+emb = model.encode(sentences, convert_to_numpy=True, show_progress_bar=False)
+# emb are L2-normalized; use cosine similarity for comparisons
 ```
+## Training summary (contrastive)
+* **Base model:** `web3se/SmartBERT-v2`
+* **Objective:** `CosineSimilarityLoss` (positives near 1.0, negatives near 0.0)
+* **Pair construction:** L2-normalized seed embeddings →
+  positives if **cosine ≥ 0.80**, negatives if **cosine ≤ 0.20** (nearest-neighbor candidates, `top_k=10`, max 5 positives/item)
+* **This release stats:** 1,647 unique texts → **16,470 pairs** (8,235 pos / 8,235 neg)
+* **Hyperparams:** epochs=1, batch_size=16, max_seq_len=512
+* **Saved as:** canonical SentenceTransformers layout (`0_Transformer/`, `1_Pooling/`, `2_Normalize/`)
+A more detailed methodology and evaluation appear in the RAVEN paper (semantic clustering of revert-inducing invariants).
+## Intended uses & limitations
+**Good for**
+* Measuring semantic relatedness of short invariant predicates
+* Clustering guards by intent (e.g., access control, slippage, timeouts)
+* Deduplicating near-equivalent checks across contracts
+**Not ideal for**
+* Long code blocks or whole-function embeddings
+* General code understanding outside invariant-style snippets
+* Non-EVM ecosystems without adaptation
+## Evaluation (paper)
+When paired with DBSCAN on predicate-only text, RavenBERT produced **compact, well-separated clusters** (e.g., Silhouette ≈ 0.93, S_Dbw ≈ 0.043 at ~52% coverage), surfacing meaningful categories of defenses from reverted transactions. See paper for full protocol, ablations, and metrics.
+## Reproducibility
+* Pair thresholds: **τ₊ = 0.80**, **τ₋ = 0.20**
+* Normalization: L2 via `sentence_transformers.models.Normalize()`
+* Training log: `ravenbert_training_stats.json` (included in repo)
 ## Citation
+If you use RavenBERT, please cite the RAVEN paper and this model:
+```
+TBD
+```
+## License
+MIT