Instructions to use sourceslicer/scalar_lm_v3 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use sourceslicer/scalar_lm_v3 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("token-classification", model="sourceslicer/scalar_lm_v3")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForMaskedLM tokenizer = AutoTokenizer.from_pretrained("sourceslicer/scalar_lm_v3") model = AutoModelForMaskedLM.from_pretrained("sourceslicer/scalar_lm_v3", device_map="auto") - Notebooks
- Google Colab
- Kaggle
SCALAR 3.0 identifier tagger
The release model for SCALAR 3.0. It assigns a part-of-speech tag to each word of a source-code identifier (getUserToken โ V NM N). It is DistilBERT (uncased) followed by a CRF that decodes over identifier words, with feature tokens for the identifier's context (FUNCTION, PARAMETER, ATTRIBUTE, ...) and its declared type.
The CRF layer is custom, so load the model through SCALAR rather than AutoModel:
pip install "scalar-tagger @ git+https://github.com/SCANL/scanl_tagger.git@v3.0.0"
scalar-tagger serve --stdio
The installed command loads this repo at a pinned commit. See the SCALAR README for the batch JSON contract, per-token confidence, and what to send in context and type.
Results
| Evaluation | Identifier accuracy |
|---|---|
| Real code (4,261 identifiers, 5-fold out-of-fold + holdout), mean of 3 seeds | 90.45% (ยฑ0.29) |
| Real identifiers in this checkpoint's holdout (853) | 92.4% |
| 375 random identifiers from 15 unseen open-source projects, reviewed by hand | 97.3% (C/C++/Java), 100% (C#) |
Tags: N, NPL, NM, V, VM, P, DT, CJ, D, PRE. confidence_temperature (1.5) in config.json scales the confidence marginals and never changes the tags.
Citation
@inproceedings{newman2025scalar,
author = {Christian Newman and Brandon Scholten and Sophia Testa and others},
title = {SCALAR: A Part-of-speech Tagger for Identifiers},
booktitle = {ICPC Tool Demonstrations Track},
year = {2025}
}
- Downloads last month
- -
Model tree for sourceslicer/scalar_lm_v3
Base model
distilbert/distilbert-base-uncased