paper-classifier: any list of options or a topic, one forward pass

2ae7d29 7 days ago

10.8 kB

---
license: apache-2.0
base_model: Qwen/Qwen3.5-2B
language:
- en
library_name: transformers
pipeline_tag: zero-shot-classification
tags:
- scientific-papers
- classification
- zero-shot-classification
- field-of-science
- taxonomy
- research-classification
- science-of-science
---
# paper-classifier-2B

**Give it a scientific paper and any list of options: it returns a probability for every option.** The options can be
any classification of research, at any level (OpenAlex fields, arXiv categories, MeSH, your own labels, the groups of
a map you made), or a single topic: "is this topic a central subject of the paper?" (yes / no).

- **No text is generated.** One forward pass; the answer is read from the next-token logits of the option codes.
- **Fast and cheap:** about 28,000 papers an hour on one RTX 3090 for a 176-option question ($9 per million papers at
  $0.25 an hour), 0.14 s for one paper on its own; yes/no questions 0.035 s (155,000 an hour).
- **Works on classifications it never saw:** tested on a held-out expert benchmark and on held-out schemes (below).
- 2B parameters, fine-tuned from [Qwen/Qwen3.5-2B](https://huggingface.co/Qwen/Qwen3.5-2B).
  Sibling: [paper-classifier-0.8B](https://huggingface.co/maximerivest/paper-classifier-0.8B)
  (about twice as fast, a little less accurate).
- Made for [Science Made Readable](https://sciencemadereadable.com). Apache 2.0.

## Quick start

```bash
pip install "transformers>=5.3" torch huggingface_hub
```

```python
from huggingface_hub import hf_hub_download
import importlib.util, sys
spec = importlib.util.spec_from_file_location("paper_classifier",
          hf_hub_download("maximerivest/paper-classifier-2B", "paper_classifier.py"))
pc = importlib.util.module_from_spec(spec); spec.loader.exec_module(pc)
clf = pc.PaperClassifier("maximerivest/paper-classifier-2B")
paper = pc.paper_text(
    title="Habitat fragmentation reduces bird diversity in tropical forests",
    abstract="We surveyed bird communities in 40 forest fragments ...",
    venue="Biological Conservation", year=2019)          # venue and year are optional
clf.classify(paper, ["Ecology", "Ornithology", "Particle physics", "Economics"], scheme="Fields of science")
# {'Ornithology': 0.65, 'Ecology': 0.35, 'Economics': 0.0004, 'Particle physics': 0.0001}
clf.is_on_topic(paper, "habitat fragmentation")   # 1.0
clf.is_on_topic(paper, "dark matter")             # 0.0
```

`paper_classifier.py` (in this repo, standard transformers + torch) builds exactly the prompt the model was trained
on. Use it, or copy its ~40 lines of prompt code, rather than writing your own prompt.

## The two tasks, as functions

```
classify(paper, options, scheme, question) -> {option: probability}     # probabilities sum to 1
    paper     str   "Title: ...\nVenue: ...\nYear: ...\nAbstract: ..."   (paper_text(); "(no abstract)" if none)
    options   list  names, or {"name", "about" (optional), "parent" (optional)}; 2 to 562 of them
    scheme    str   name of the classification, e.g. "OpenAlex topic classification" or "My lab's reading groups"
    question  str   e.g. "Which subfield best describes the research content of this paper?"
is_on_topic(paper, topic, definition=None) -> P(yes)
    topic       str  e.g. "habitat fragmentation"
    definition  str  optional: what the topic covers, to settle ambiguous words
```

### Exactly what the model sees

Every question uses the same system message and the Qwen3.5 chat template **with thinking off**
(`enable_thinking=False`, so the assistant turn starts with an empty `<think>` block). The paper comes first, the
question after.

System message (verbatim, also `SYSTEM` in `paper_classifier.py`):

```
You classify scientific papers. You get a paper (its title, and its abstract when available), then a question. For a list of options, answer with the code of the option that best describes the research content of the paper, following the classification's own meaning of each option. For a single topic, answer yes if the topic is a central subject of the paper, else no. On topic: the paper studies the topic itself as a main subject: the phenomenon, its causes, consequences, mechanisms, measurement, methods, or how it is managed, in any system or organism. Not on topic: the topic is absent or only mentioned in passing; or the words mean something else in this paper; or the paper studies a neighbouring subject without studying the topic itself.
```

User message, list of options (here 3 of the 176 Science-Metrix subfields; each option gets a two-letter code):

```
Title: Combined androgen excess and Western-style diet accelerates adipose tissue dysfunction ...
Venue: Human Reproduction
Year: 2017
Abstract: STUDY QUESTION: What are the separate and combined effects of ...
## Classification: Science-Metrix classification of science (176 subfields within 22 fields and 6 domains)
Which subfield best describes the research content of this paper?
Options:
AA. Agronomy & Agriculture — in Agriculture, Fisheries & Forestry
AB. Dairy & Animal Science — in Agriculture, Fisheries & Forestry
AC. Fisheries — in Agriculture, Fisheries & Forestry
Answer with the option code only.
```

An option line is `CODE. name`, then optionally ` — ` followed by its description and/or `in <parent>`, joined by
`; `. Codes are `AA, AB, AC, ...` in the order listed: the 562 two-letter codes that are single tokens
(`classifier.json` → `codes`, with their token ids in `code_ids`).

User message, one topic:

```
Title: ...
Abstract: ...
## Topic: habitat fragmentation
Is this topic a central subject of the paper? Answer yes or no.
```

(with a definition: `## Topic: habitat fragmentation (what it covers: ...)`)

### Reading the answer (no generation)

Run the prompt once and take the logits at the next position. Keep only the logits of the codes you offered
(`code_ids[:len(options)]`; for yes/no the `yes` and `no` ids in `classifier.json`) and apply a softmax over those.
That is the distribution the model was trained to match. Do not sample or parse generated text.

```python
logits = model(**inputs, logits_to_keep=1).logits[0, -1]        # transformers
probs = logits[code_ids[:len(options)]].float().softmax(-1)
```

## Tips for the best results (closest to training)

- **Use `paper_text()`**: title, venue and year when known, then the abstract (the first 1,500 characters; the
  helper cuts the paper part to 700 tokens). Title-only works too (`(no abstract)`), a little less well.
- **Name the classification** in `scheme`, and ask about the level by its name: "Which *field* ...", "Which
  *topic* ...". The model follows the scheme's own meaning of each option.
- **Short descriptions help** when option names are ambiguous (`about`). For a step down a hierarchy, put the
  parent in the scheme: `scheme="OpenAlex topic classification — within: Ecology"` (what `classify_tree` does), or
  give each option its parent (`parent`).
- **Up to 562 options in one question.** Bigger schemes: ask level by level (`classify_tree`). For lists that fit,
  one flat question was better than going level by level on the expert benchmark.
- **The order of the options matters somewhat** (shuffling them keeps the same top pick 76% of the time).
  For important decisions use `classify(..., orders=3)`: it averages 3 orders, at 3x the cost.
- **"None of these"**: `none_option=True` adds an option for papers outside the list.
- Probabilities are fairly well calibrated (expected calibration error 0.053 on the expert benchmark): use them,
  not just the top pick.

## Fast bulk scoring with vLLM

vLLM's text generation path reads at most 128 token probabilities per request and re-reads the prompt for more.
`vllm_fast.py` in this repo uses vLLM's pooling runner instead: it returns the hidden state at the answer position and
computes the option scores with the model's output layer, one request per question, any number of options. Same
results as transformers (same top pick, probabilities within about 0.02). Tested with vLLM 0.29 on an RTX 3090:
throughput was the same from 16 to 256 questions at a time (the GPU is busy computing the prompt), CUDA graphs on
gave +13%.

## Results

### Expert benchmark: 176 subfields of science, a classification it never trained on

200 articles, each classified by 5 or 6 experts into the 176 Science-Metrix subfields (Rivest, Vignola-Gagné &
Archambault 2021, PLOS ONE, [doi:10.1371/journal.pone.0251493](https://doi.org/10.1371/journal.pone.0251493),
"Benchmark 2"). Score: agreement with the experts' most common first choice. The Science-Metrix scheme and these
articles were excluded from training (checked: no training question used the scheme; none of the 200 titles is in
any training data). One question per article, with title, venue, year and abstract.

| | specialist journals | general journals | $ per 1,000 papers |
|---|---:|---:|---:|
| paper-classifier-0.8B | 64% | 53% | $0.004 |
| paper-classifier-2B | 62% | 61% | $0.009 |
| paper-classifier-2B, 3 option orders | 68% | 61% | $0.027 |
| Qwen3.8-27B (direct) | 57% | 64% | $0.022 |
| Jev (TypeSafe), flat | 64% | 61% | $0.19 |
| Jev (TypeSafe), field then subfield | 68% | 65% | $0.21 |
| GPT-6.1 Sol | 66% | 69% | $1.40 |
| one human expert vs the others | 63% | 57% | ~$550 (reading the abstract at $50/h) |

![Best-value chart](benchmark.png)

Each number has about ±10 points of uncertainty (100 articles per half). This model (2B): right subfield in its
top 3 for 90% of articles; right broad field (of 22) from its subfield pick 79% / 77% (specialist / general).
Costs of local models: one RTX 3090 at $0.25 an hour (Vast.ai and RunPod, 2026-10-18).

### Classifications it never saw

Schemes and papers held out of training entirely, scored against reference labels:

| | same top pick as the reference |
|---|---:|
| JEL economics codes | 82.2% |
| new classifications written for the test | 79.9% |
| held-out papers on other classifications | 87.2% |

### Topic yes / no

8 topics held out of training (4,800 papers, scored against reference labels): macro average precision 0.80.

## Limitations

- **English** titles and abstracts; other languages were not tested.
- **Order sensitivity** (see tips): average a few orders when it matters.
- **Very broad, oddly named levels** are its weak spot: asked directly for one of 22 broad fields, it was right less
  often than when the field was read off its subfield pick. Prefer the finest level you have and add up.
- Outside the expert benchmark, its results are measured against reference labels, not human experts.
- It judges what a paper is *about* from its title and abstract, not its quality or correctness.

## License

Apache 2.0, like the base model (Qwen3.5, Apache 2.0, see `LICENSE`).

	---
	license: apache-2.0
	base_model: Qwen/Qwen3.5-2B
	language:
	- en
	library_name: transformers
	pipeline_tag: zero-shot-classification
	tags:
	- scientific-papers
	- classification
	- zero-shot-classification
	- field-of-science
	- taxonomy
	- research-classification
	- science-of-science
	---

	# paper-classifier-2B

	Give it a scientific paper and any list of options: it returns a probability for every option. The options can be
	any classification of research, at any level (OpenAlex fields, arXiv categories, MeSH, your own labels, the groups of
	a map you made), or a single topic: "is this topic a central subject of the paper?" (yes / no).

	- No text is generated. One forward pass; the answer is read from the next-token logits of the option codes.
	- Fast and cheap: about 28,000 papers an hour on one RTX 3090 for a 176-option question ($9 per million papers at
	$0.25 an hour), 0.14 s for one paper on its own; yes/no questions 0.035 s (155,000 an hour).
	- Works on classifications it never saw: tested on a held-out expert benchmark and on held-out schemes (below).
	- 2B parameters, fine-tuned from [Qwen/Qwen3.5-2B](https://huggingface.co/Qwen/Qwen3.5-2B).
	Sibling: [paper-classifier-0.8B](https://huggingface.co/maximerivest/paper-classifier-0.8B)
	(about twice as fast, a little less accurate).
	- Made for [Science Made Readable](https://sciencemadereadable.com). Apache 2.0.

	## Quick start

	```bash
	pip install "transformers>=5.3" torch huggingface_hub
	```

	```python
	from huggingface_hub import hf_hub_download
	import importlib.util, sys
	spec = importlib.util.spec_from_file_location("paper_classifier",
	hf_hub_download("maximerivest/paper-classifier-2B", "paper_classifier.py"))
	pc = importlib.util.module_from_spec(spec); spec.loader.exec_module(pc)

	clf = pc.PaperClassifier("maximerivest/paper-classifier-2B")
	paper = pc.paper_text(
	title="Habitat fragmentation reduces bird diversity in tropical forests",
	abstract="We surveyed bird communities in 40 forest fragments ...",
	venue="Biological Conservation", year=2019) # venue and year are optional

	clf.classify(paper, ["Ecology", "Ornithology", "Particle physics", "Economics"], scheme="Fields of science")
	# {'Ornithology': 0.65, 'Ecology': 0.35, 'Economics': 0.0004, 'Particle physics': 0.0001}

	clf.is_on_topic(paper, "habitat fragmentation") # 1.0
	clf.is_on_topic(paper, "dark matter") # 0.0
	```

	`paper_classifier.py` (in this repo, standard transformers + torch) builds exactly the prompt the model was trained
	on. Use it, or copy its ~40 lines of prompt code, rather than writing your own prompt.

	## The two tasks, as functions

	```
	classify(paper, options, scheme, question) -> {option: probability} # probabilities sum to 1
	paper str "Title: ...\nVenue: ...\nYear: ...\nAbstract: ..." (paper_text(); "(no abstract)" if none)
	options list names, or {"name", "about" (optional), "parent" (optional)}; 2 to 562 of them
	scheme str name of the classification, e.g. "OpenAlex topic classification" or "My lab's reading groups"
	question str e.g. "Which subfield best describes the research content of this paper?"

	is_on_topic(paper, topic, definition=None) -> P(yes)
	topic str e.g. "habitat fragmentation"
	definition str optional: what the topic covers, to settle ambiguous words
	```

	### Exactly what the model sees

	Every question uses the same system message and the Qwen3.5 chat template with thinking off
	(`enable_thinking=False`, so the assistant turn starts with an empty `<think>` block). The paper comes first, the
	question after.

	System message (verbatim, also `SYSTEM` in `paper_classifier.py`):

	```
	You classify scientific papers. You get a paper (its title, and its abstract when available), then a question. For a list of options, answer with the code of the option that best describes the research content of the paper, following the classification's own meaning of each option. For a single topic, answer yes if the topic is a central subject of the paper, else no. On topic: the paper studies the topic itself as a main subject: the phenomenon, its causes, consequences, mechanisms, measurement, methods, or how it is managed, in any system or organism. Not on topic: the topic is absent or only mentioned in passing; or the words mean something else in this paper; or the paper studies a neighbouring subject without studying the topic itself.
	```

	User message, list of options (here 3 of the 176 Science-Metrix subfields; each option gets a two-letter code):

	```
	Title: Combined androgen excess and Western-style diet accelerates adipose tissue dysfunction ...
	Venue: Human Reproduction
	Year: 2017
	Abstract: STUDY QUESTION: What are the separate and combined effects of ...

	## Classification: Science-Metrix classification of science (176 subfields within 22 fields and 6 domains)
	Which subfield best describes the research content of this paper?
	Options:
	AA. Agronomy & Agriculture — in Agriculture, Fisheries & Forestry
	AB. Dairy & Animal Science — in Agriculture, Fisheries & Forestry
	AC. Fisheries — in Agriculture, Fisheries & Forestry
	Answer with the option code only.
	```

	An option line is `CODE. name`, then optionally ` — ` followed by its description and/or `in <parent>`, joined by
	`; `. Codes are `AA, AB, AC, ...` in the order listed: the 562 two-letter codes that are single tokens
	(`classifier.json` → `codes`, with their token ids in `code_ids`).

	User message, one topic:

	```
	Title: ...
	Abstract: ...

	## Topic: habitat fragmentation
	Is this topic a central subject of the paper? Answer yes or no.
	```

	(with a definition: `## Topic: habitat fragmentation (what it covers: ...)`)

	### Reading the answer (no generation)

	Run the prompt once and take the logits at the next position. Keep only the logits of the codes you offered
	(`code_ids[:len(options)]`; for yes/no the `yes` and `no` ids in `classifier.json`) and apply a softmax over those.
	That is the distribution the model was trained to match. Do not sample or parse generated text.

	```python
	logits = model(**inputs, logits_to_keep=1).logits[0, -1] # transformers
	probs = logits[code_ids[:len(options)]].float().softmax(-1)
	```

	## Tips for the best results (closest to training)

	- Use `paper_text()`: title, venue and year when known, then the abstract (the first 1,500 characters; the
	helper cuts the paper part to 700 tokens). Title-only works too (`(no abstract)`), a little less well.
	- Name the classification in `scheme`, and ask about the level by its name: "Which field ...", "Which
	topic ...". The model follows the scheme's own meaning of each option.
	- Short descriptions help when option names are ambiguous (`about`). For a step down a hierarchy, put the
	parent in the scheme: `scheme="OpenAlex topic classification — within: Ecology"` (what `classify_tree` does), or
	give each option its parent (`parent`).
	- Up to 562 options in one question. Bigger schemes: ask level by level (`classify_tree`). For lists that fit,
	one flat question was better than going level by level on the expert benchmark.
	- The order of the options matters somewhat (shuffling them keeps the same top pick 76% of the time).
	For important decisions use `classify(..., orders=3)`: it averages 3 orders, at 3x the cost.
	- "None of these": `none_option=True` adds an option for papers outside the list.
	- Probabilities are fairly well calibrated (expected calibration error 0.053 on the expert benchmark): use them,
	not just the top pick.

	## Fast bulk scoring with vLLM

	vLLM's text generation path reads at most 128 token probabilities per request and re-reads the prompt for more.
	`vllm_fast.py` in this repo uses vLLM's pooling runner instead: it returns the hidden state at the answer position and
	computes the option scores with the model's output layer, one request per question, any number of options. Same
	results as transformers (same top pick, probabilities within about 0.02). Tested with vLLM 0.29 on an RTX 3090:
	throughput was the same from 16 to 256 questions at a time (the GPU is busy computing the prompt), CUDA graphs on
	gave +13%.

	## Results

	### Expert benchmark: 176 subfields of science, a classification it never trained on

	200 articles, each classified by 5 or 6 experts into the 176 Science-Metrix subfields (Rivest, Vignola-Gagné &
	Archambault 2021, PLOS ONE, [doi:10.1371/journal.pone.0251493](https://doi.org/10.1371/journal.pone.0251493),
	"Benchmark 2"). Score: agreement with the experts' most common first choice. The Science-Metrix scheme and these
	articles were excluded from training (checked: no training question used the scheme; none of the 200 titles is in
	any training data). One question per article, with title, venue, year and abstract.

	\| \| specialist journals \| general journals \| $ per 1,000 papers \|
	\|---\|---:\|---:\|---:\|
	\| paper-classifier-0.8B \| 64% \| 53% \| $0.004 \|
	\| paper-classifier-2B \| 62% \| 61% \| $0.009 \|
	\| paper-classifier-2B, 3 option orders \| 68% \| 61% \| $0.027 \|
	\| Qwen3.8-27B (direct) \| 57% \| 64% \| $0.022 \|
	\| Jev (TypeSafe), flat \| 64% \| 61% \| $0.19 \|
	\| Jev (TypeSafe), field then subfield \| 68% \| 65% \| $0.21 \|
	\| GPT-6.1 Sol \| 66% \| 69% \| $1.40 \|
	\| one human expert vs the others \| 63% \| 57% \| ~$550 (reading the abstract at $50/h) \|

	![Best-value chart](benchmark.png)

	Each number has about ±10 points of uncertainty (100 articles per half). This model (2B): right subfield in its
	top 3 for 90% of articles; right broad field (of 22) from its subfield pick 79% / 77% (specialist / general).
	Costs of local models: one RTX 3090 at $0.25 an hour (Vast.ai and RunPod, 2026-10-18).

	### Classifications it never saw

	Schemes and papers held out of training entirely, scored against reference labels:

	\| \| same top pick as the reference \|
	\|---\|---:\|
	\| JEL economics codes \| 82.2% \|
	\| new classifications written for the test \| 79.9% \|
	\| held-out papers on other classifications \| 87.2% \|

	### Topic yes / no

	8 topics held out of training (4,800 papers, scored against reference labels): macro average precision 0.80.

	## Limitations

	- English titles and abstracts; other languages were not tested.
	- Order sensitivity (see tips): average a few orders when it matters.
	- Very broad, oddly named levels are its weak spot: asked directly for one of 22 broad fields, it was right less
	often than when the field was read off its subfield pick. Prefer the finest level you have and add up.
	- Outside the expert benchmark, its results are measured against reference labels, not human experts.
	- It judges what a paper is about from its title and abstract, not its quality or correctness.

	## License

	Apache 2.0, like the base model (Qwen3.5, Apache 2.0, see `LICENSE`).