| --- |
| license: cc-by-4.0 |
| language: [ru, en, tr, zh] |
| library_name: gliner |
| pipeline_tag: token-classification |
| tags: [gliner, ner, toponym, geonames, multilingual] |
| base_model: urchade/gliner_multi-v2.1 |
| --- |
| |
| # geosearch-ner |
|
|
| Multilingual toponym NER, fine-tuned from [`urchade/gliner_multi-v2.1`](https://huggingface.co/urchade/gliner_multi-v2.1). |
| Labels: `CITY`, `REGION`, `STATE`, `COUNTRY`. |
|
|
| **Span F1 0.763** (P 0.759 / R 0.767) on 47 held-out queries at threshold 0.4. |
|
|
| This is the first stage of a multilingual toponym search pipeline: |
|
|
| ``` |
| text |
| -> GLiNER (this model) |
| -> char-n-gram BM25 retrieval |
| -> CatBoost reranker |
| -> ranked GeoNames places |
| ``` |
|
|
| ## Serving it: the word-splitter contract |
|
|
| **This checkpoint requires a per-ideograph word splitter, and serving it without |
| one loses every Chinese span with no error of any kind.** |
|
|
| GLiNER classifies *token spans*, and its default whitespace splitter treats a |
| run of Han characters as one token — so `莫斯科` inside `莫斯科新闻` is not a |
| span the model can even express, let alone predict. This model was trained with |
| one token per Han ideograph (`src/ner/tokenizer.py::CjkAwareSplitter`), which |
| `gliner_config.json` cannot record: `words_splitter_type` names only GLiNER's |
| built-in kinds. |
|
|
| So the requirement is recorded in **`ner_meta.json`**, shipped beside the |
| weights in this repo: |
| |
| ```json |
| {"words_splitter": "src.ner.tokenizer.CjkAwareSplitter"} |
| ``` |
| |
| The serving code reads that file and refuses to start when the configured |
| splitter disagrees with it. If you load this model yourself, re-attach an |
| equivalent splitter — measured effect of getting it wrong: 56/56 Chinese spans |
| found versus 0. |
| |
| ## Decision threshold |
| |
| Served at **0.3**, below GLiNER's own 0.5, and |
| that is a property of *serving* rather than of the model: retrieval is |
| recall-hungry, because a city span never extracted can never be retrieved, |
| while a spurious span only adds a candidate the reranker can demote. The value |
| is derived from the sweep in `make ner-eval --sweep`, not asserted. |
| |
| ## Results |
| |
| ### Overall |
| |
| | bucket | precision | recall | F1 | gold | predicted | |
| |---|---|---|---|---|---| |
| | overall | 0.759 | 0.767 | 0.763 | 86 | 87 | |
| |
| ### By label |
| |
| | bucket | precision | recall | F1 | gold | predicted | |
| |---|---|---|---|---|---| |
| | CITY | 0.855 | 0.887 | 0.870 | 53 | 55 | |
| | REGION | 0.800 | 0.267 | 0.400 | 15 | 5 | |
| | STATE | 0.333 | 0.833 | 0.476 | 6 | 15 | |
| | COUNTRY | 0.833 | 0.833 | 0.833 | 12 | 12 | |
| |
| ### By language |
| |
| | bucket | precision | recall | F1 | gold | predicted | |
| |---|---|---|---|---|---| |
| | ru | 0.818 | 0.818 | 0.818 | 22 | 22 | |
| | en | 0.809 | 0.809 | 0.809 | 21 | 21 | |
| | tr | 0.708 | 0.708 | 0.708 | 24 | 24 | |
| | zh | 0.700 | 0.737 | 0.718 | 19 | 20 | |
| |
| ### Versus the zero-shot baseline |
| |
| Zero-shot `urchade/gliner_multi-v2.1`, measured with the **same** word splitter (otherwise the Chinese delta would be an artefact of segmentation rather than of fine-tuning): |
| |
| | bucket | precision | recall | F1 | gold | predicted | |
| |---|---|---|---|---|---| |
| | overall | 0.726 | 0.616 | 0.667 | 86 | 73 | |
| |
| Metrics are micro-averaged over **spans**, not macro over queries: a query |
| naming three cities feeds three names into retrieval, so each one is a unit of |
| work the pipeline either gets right or does not. |
| |
| ## Training |
| |
| Selected by **validation span F1**, not `eval_loss` — the checkpoint saved is |
| whichever epoch scored best when called the way the engine calls it |
| (`predict_entities` on raw text). |
| |
| | | | |
| |---|---| |
| | best epoch | 12 | |
| | best val span F1 | 0.763 | |
| | epochs | 15 | |
| | batch size | 8 | |
| | learning rate | 5e-06 | |
| | others LR | 1e-05 | |
| | weight decay | 0.01 | |
| | warmup ratio | 0.1 | |
| | seed | 42 | |
| | trained at | 2026-08-07T08:49:14+00:00 | |
| |
| Trained on synthetic queries generated for this project — see |
| [`mki0809/geosearch-queries`](https://huggingface.co/datasets/mki0809/geosearch-queries). |
| |
| ## Data licence and attribution |
| |
| Derived from [GeoNames](https://www.geonames.org/), licensed |
| [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). |
| |
| **Modifications made to the source data:** |
| |
| - filtered to populated places only (`feature_class = 'P'`) |
| - restricted to RU, US, TR, CN |
| - restricted to name variants in ru, en, tr, zh |
| - dropped feature codes PPLH, PPLQ, PPLW, PPLX |
| - name variants grouped per place and joined into single document strings |
| |
| |
| The fine-tune inherits any licence conditions of its base model; check |
| [`urchade/gliner_multi-v2.1`](https://huggingface.co/urchade/gliner_multi-v2.1) |
| before commercial use. |
| |
