Farseen0/scar-weights

SCAR: Sparse Code Audit Retriever (v1 research checkpoints)

SCAR retrieves vulnerable Solidity code from natural-language audit findings. It encodes text with Qwen2.5-Coder-1.5B plus a backbone LoRA, projects the layer-19 residual stream through a trained low-rank sparse projection with 16,384 dimensions (initialized from a sparse autoencoder trained on scar-corpus), and pools the result into an inverted-index-compatible sparse vector.

What this repository contains

The April 2026 research checkpoints scar-25ep and scar-15ep, and the sparse-autoencoder checkpoint they were initialized from. They were trained on the v1 pool scar-pairs, which shares 800 pairs with the development split scar-eval; under the v2 protocol below they are provenance artifacts rather than the reference models. Checkpoints retrained on scar-pairs-clean will be released with the paper.

Benchmark state (v2 protocol, September 2026)

• Splits. scar-eval (838 queries) is the development split. scar-eval-2026-holdout (764 queries; Solodit findings ingested after March 2026, audit dates May 2025 to May 2026, disjoint from every training pool by exact-hash and near-duplicate gates) is the held-out split. The held-out split is private at the time of writing; release is planned with the paper.
• Training pool. Models scored on either split are trained on scar-pairs-clean (6,202 pairs, revision ec2750c), which is deduplicated against scar-eval at the document level. scar-pairs (7,552 pairs) and scar-pairs-extended (11,961 pairs) are the v1 pools: they share 800 (query, positive) pairs and all 838 positives with scar-eval, so a model trained on them is not scored on scar-eval.
• Scoring. R@k over the full corpus (scar-corpus, 231,269 contracts) with the split's gold documents appended (development pool 232,025 documents, held-out pool 232,033). Rank = 1 + the number of non-gold documents scoring at or above the gold document, so ties count against the system; float32 scores; one rule for every row.
• Superseded numbers. Tables shown on these pages before September 2026 were computed under the v1 protocol, before the training pool was deduplicated against the development split. They are superseded and no longer shown.

Reference numbers under the v2 protocol (R@10, development / held-out): SCAR retrained on scar-pairs-clean (seed 42, checkpoint at 25 % of the schedule) 0.652 / 0.344; the same backbone with a dense head 0.662 / 0.486; stock Qwen3-Embedding-0.6B at 512 tokens 0.655 / 0.651; stock SPLADE-code-0.6B with top-400 terms 0.735 / 0.827. The full table, with three-seed means, is on the scar-eval card. The held-out split separates systems that the development split does not; the paper in preparation analyses that gap.

Architecture note. In the v2 analysis, initializing the sparse projection from random directions at matched column norms gives the same retrieval quality as initializing it from the sparse autoencoder (single-seed: 0.557 vs 0.549 R@10 on the development split under the earlier evaluator). This card therefore describes the model as a backbone LoRA plus a trained low-rank sparse projection and makes no claim about the semantics of individual sparse dimensions.

Efficiency (v2 checkpoint, 607,530-document index: the corpus, 375,507 DISL contracts and the gold documents). Sparse index 220 MB against 1.9 GB for the dense reference (storage estimates); CPU serving on one host with 12 threads and the same queries: scoring 40.7 ms (sparse) vs 26.5 ms (dense), end to end 315.5 ms vs 364.1 ms; process memory after load 9.5 GB vs 12.8 GB.

Architecture

Input text
    │
    ▼
Qwen2.5-Coder-1.5B + LoRA (rank 64 on Q/K/V/O)
    │
    ▼ Layer 19 residual stream (1536-dim, bidirectional attention)
    │
Sparse projection W_e + A·B  (W_e: SAE encoder, frozen; A·B: rank 256, trained)
    │
    ▼ 16,384 sparse dimensions (ReLU with learned thresholds)
    │
Per-token TopK (k=64)  →  Sum-pool  →  log1p saturation
    │
    ▼
IDF weighting  →  Document TopK (q=100, d=400)  →  L2 norm
    │
    ▼
Sparse retrieval vector (inverted-index compatible)

Component	Spec
Backbone	Qwen2.5-Coder-1.5B (28 layers, hidden 1536)
Sparse projection	16,384 dimensions, initialized from a sparse autoencoder trained on scar-corpus at layer 19; base weights frozen, low-rank update (rank 256, 4.6M parameters) trained with the backbone LoRA
Backbone LoRA	rank 64 on Q/K/V/O, 17.4M parameters
Pooling	Sum-pool + log1p saturation
Sparsity	Per-token TopK = 64; document TopK = 400; query TopK = 100
Total trainable	about 22M parameters (1.5 % of the backbone)

Repository Layout

sae/
├── checkpoint_final.pt   # Sparse-autoencoder checkpoint (base weights of the projection; shared by both variants)
└── config.json
scar-25ep/
├── checkpoint_final.pt   # Projection low-rank update + IDF weights + training config
├── config.json
└── lora_adapter/         # PEFT-compatible backbone LoRA
    ├── adapter_model.safetensors
    └── adapter_config.json
scar-15ep/
├── checkpoint_final.pt
├── config.json
└── lora_adapter/
    ├── adapter_model.safetensors
    └── adapter_config.json

Loading

import torch
from huggingface_hub import snapshot_download
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

local_dir = snapshot_download("Farseen0/scar-weights")
variant = "scar-25ep"   # or "scar-15ep"

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-Coder-1.5B")
backbone = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-Coder-1.5B",
    torch_dtype=torch.bfloat16,
)

# Backbone LoRA via PEFT
model = PeftModel.from_pretrained(
    backbone,
    f"{local_dir}/{variant}/lora_adapter",
)

# Projection low-rank update + IDF + config
ckpt = torch.load(f"{local_dir}/{variant}/checkpoint_final.pt", map_location="cpu")
sae_lora_state = ckpt["sae_lora_state"]   # A, B matrices of the projection update
idf_weights    = ckpt["idf_weights"]      # (16384,) corpus-derived IDF
config         = ckpt["config"]           # Full training config dict

# Sparse-autoencoder checkpoint (base weights of the projection)
sae_ckpt = torch.load(f"{local_dir}/sae/checkpoint_final.pt", map_location="cpu")

End-to-end inference (encode, sparse vector, retrieve) is in the GitHub repository.

Training data of the v1 checkpoints

Dataset	Purpose	Size
Farseen0/scar-corpus	Sparse-autoencoder pretraining + retrieval corpus	231,269 contracts
Farseen0/scar-pairs	Contrastive training pairs (v1 pool)	7,552 pairs
Farseen0/scar-eval	Development split	838 queries

Pairs are drawn from professional audit findings (Solodit, MSC, FORGE-Curated, DeFiHackLabs, EVuLLM, SmartBugs-Curated). Each pair: (query = severity-prefixed finding, positive = vulnerable code, hard_negative = different vulnerability from the same protocol).

Training setup of the v1 checkpoints

• Hardware: NVIDIA H100 (Modal Labs)
• Sparse-autoencoder pretraining: 84,594 steps, lr = 2e-4; final variance explained 0.97; final L0 = 0.59 under the checkpoint's own thresholds
• Retrieval fine-tuning: 25 epochs (5,900 steps), batch size 32, lr = 5e-5, τ = 0.1
• Loss: InfoNCE + margin-MSE distillation (λ = 0.5) + DF-FLOPS (λ = 1e-6)
• Total compute: about 70 H100-hours

The v2 recipe behind the benchmark numbers above keeps the architecture and trains on scar-pairs-clean (6,202 pairs): a 25-epoch schedule of 4,825 optimizer steps, no distillation term, checkpoint at 25 % of the schedule.

Limitations

• Single backbone: only Qwen2.5-Coder-1.5B has been trained; transfer to other code models is untested.
• Held-out behaviour: the v2 held-out split separates systems that the development split does not (see the scar-eval card); the paper in preparation analyses the gap between domain-trained and stock retrievers.
• No feature-level interpretation of the sparse dimensions is claimed (see the architecture note above).
• Solidity / EVM only: other smart-contract languages (Move, Sway, Vyper, Cairo) are out of distribution.
• Single-contract granularity: the indexer treats each contract as one document; cross-contract vulnerabilities may rank below their per-file evidence.

Citation

@misc{shaikh2026scar,
  title  = {SCAR: a benchmark and analysis for retrieving smart-contract audit findings},
  author = {Shaikh, Farseen},
  year   = {2026},
  note   = {Paper in preparation},
  url    = {https://github.com/FarseenSh/scar-retrieval}
}

Links

• 🗂️ All artifacts — HuggingFace Collection
• 📄 Paper — in preparation (2026)
• 🤗 Datasets — scar-corpus · scar-pairs · scar-pairs-extended · scar-eval
• 🐙 Code — github.com/FarseenSh/scar-retrieval

License

Apache 2.0 — free for research and commercial use with attribution.

---

SCAR is independent research by Farseen Shaikh. Built on Qwen2.5-Coder by Alibaba; the sparse projection is initialized from a sparse autoencoder in the style of Rajamanoharan et al. (2024).