| --- |
| pipeline_tag: text-ranking |
| library_name: transformers |
| license: other |
| tags: |
| - bert |
| - modernbert |
| - government |
| - relevance |
| - text-relevance |
| - govrelbench |
| --- |
| |
| # GovRelBERT: A Model for Government Domain Relevance |
|
|
| **GovRelBERT** is a specialized model designed for evaluating the core capabilities of Large Language Models (LLMs) in the government domain, particularly focusing on text relevance. It was introduced as part of the [GovRelBench: A Benchmark for Government Domain Relevance](https://huggingface.co/papers/2507.21419) paper. |
|
|
| Built upon the **ModernBERT** architecture, GovRelBERT is trained using the **SoftGovScore** method. This innovative method converts hard labels into soft scores, enabling the model to accurately compute a text's government domain relevance score. GovRelBERT serves as a dedicated evaluation tool within the GovRelBench framework, aiming to improve the assessment of LLMs in government-related research and practical applications. |
|
|
| The code and dataset are available as part of the GovRelBench project on GitHub: [https://github.com/pansysy/GovRelBench](https://github.com/pansysy/GovRelBench). |
|
|
| ## How to use |
|
|
| You can use `GovRelBERT` with the Hugging Face `transformers` library to compute a government domain relevance score for a given text. |
|
|
| ```python |
| from transformers import AutoTokenizer, AutoModelForSequenceClassification |
| import torch |
| |
| model_name = "pansysy/GovRelBERT" |
| tokenizer = AutoTokenizer.from_pretrained(model_name) |
| model = AutoModelForSequenceClassification.from_pretrained(model_name) |
| |
| # Example 1: Text highly relevant to the government domain |
| text_relevant = "The new policy initiative aims to streamline public services for citizens." |
| inputs_relevant = tokenizer(text_relevant, return_tensors="pt") |
| |
| with torch.no_grad(): |
| outputs_relevant = model(**inputs_relevant) |
| relevance_score_relevant = outputs_relevant.logits.item() |
| |
| print(f"Text: '{text_relevant}'") |
| print(f"Government relevance score: {relevance_score_relevant:.4f}") |
| |
| print("-" * 50) |
| |
| # Example 2: Text less relevant to the government domain |
| text_less_relevant = "How to train your pet dog to fetch a ball." |
| inputs_less_relevant = tokenizer(text_less_relevant, return_tensors="pt") |
| |
| with torch.no_grad(): |
| outputs_less_relevant = model(**inputs_less_relevant) |
| relevance_score_less_relevant = outputs_less_relevant.logits.item() |
| |
| print(f"Text: '{text_less_relevant}'") |
| print(f"Government relevance score: {relevance_score_less_relevant:.4f}") |
| ``` |
|
|
| ## Citation |
|
|
| If you find this work helpful, please cite the original paper: |
|
|
| ```bibtex |
| @misc{liu2025govrelbench, |
| title={GovRelBench:A Benchmark for Government Domain Relevance}, |
| author={Yizhuo Liu and Siyi Pan and Ruohong Han and Yu Hong and Bojin Wang and Mingyang Li and Jianxun Tang}, |
| year={2025}, |
| eprint={2507.21419}, |
| archivePrefix={arXiv}, |
| primaryClass={cs.CL}, |
| url={https://arxiv.org/abs/2507.21419}, |
| } |
| ``` |
