---
license: apache-2.0
library_name: pytorch
base_model: answerdotai/ModernBERT-base
pipeline_tag: token-classification
tags:
  - bibr
  - scientific-references
  - sequence-labeling
  - crf
---

# bibr-segmenter-v1

ModernBERT-base + CRF reference segmenter for the
[bibr](https://github.com/scienceverse/bibr) scientific-paper extraction
pipeline. Takes the references section of a paper and emits per-character
BIO labels (`O`, `B-REF`, `I-REF`) to split it into individual references.

- **Architecture:** `answerdotai/ModernBERT-base` encoder + linear head +
  `torchcrf.CRF` decode with BIO transition constraints.
- **Input:** the raw concatenated references-section text, one logical
  reference per line (newlines are the segmenter's training-time
  boundary marker — internal column-wrap whitespace should be collapsed
  per row first; see `bibr/extract/extractor.py:_extract_references_ner`).
- **Output:** a list of reference strings.
- **Inference:** sliding-window with `window=2048`, `stride=1536`,
  overlap-merge via per-position trust score.
- **Training corpus:** gold-anchored, judge-corrected reference lines
  from psych250 + Directorate-General Economics + MDPI Social Sciences
  (~440 papers, 2026-05-09/10).

## Held-out validation (2026-05-10)

- 87% of papers within ±10% of gold reference count.
- 99% within ±20%.
- Exact ref-count match on the 3-paper psych APA holdout.

## Loading

```python
from bibr.ner.segmenter import RefSegmenter
seg = RefSegmenter("scienceverse/bibr-segmenter-v1")
refs = seg.segment(references_section_text)
```

The `RefSegmenter` resolver downloads the checkpoint via
`huggingface_hub.hf_hub_download` (filename: `seg_v3_gold.pt`).

## Caveats

- In-distribution: APA-style social-science references.
- Out-of-distribution: arXiv-style bracketed-numbered refs (`[1] Author...`)
  and IEEE conference proceedings — paired with `bibr-parser-v1` these
  recover ~94% of LLM ref count but container-field accuracy drops.
- Acknowledgement lines that leak into the references-section input
  occasionally pass through as junk "refs"; bibr's downstream
  completeness filter (`title or authors`) catches most.