gliner_ko

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+pytorch_model.bin filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,3 +1,98 @@
 ---
-license: cc-by-nc-3.0
+license: cc-by-nc-4.0
+language:
+- korean
+pipeline_tag: token-classification
+library_name: gliner
 ---
+
+
+# Model Card for GLiNER-ko
+
+GLiNER is a Named Entity Recognition (NER) model capable of identifying any entity type using a bidirectional transformer encoder (BERT-like). It provides a practical alternative to traditional NER models, which are limited to predefined entities, and Large Language Models (LLMs) that, despite their flexibility, are costly and large for resource-constrained scenarios.
+
+This version has been trained on the **various Korean NER** dataset (Research purpose). Commercially permission versions are available (**urchade/gliner_smallv2**, **urchade/gliner_mediumv2**, **urchade/gliner_largev2**)
+
+## Links
+
+* Paper: https://arxiv.org/abs/2311.08526
+* Repository: https://github.com/urchade/GLiNER
+
+## Installation
+To use this model, you must install the Korean fork of GLiNER Python library and mecab-ko:
+```
+!pip install git+https://github.com/taeminlee/GLiNER
+!pip install python-mecab-ko
+```
+
+## Usage
+Once you've downloaded the GLiNER library, you can import the GLiNER class. You can then load this model using `GLiNER.from_pretrained` and predict entities with `predict_entities`.
+
+```python
+from gliner import GLiNER
+
+model = GLiNER.from_pretrained("taeminlee/gliner_ko")
+
+text = """
+피터 잭슨 경(, 1961년 10월 31일 ~ )은 뉴질랜드의 영화 감독, 각본가, 영화 프로듀서이다. J. R. R. 톨킨의 소설을 원작으로 한 《반지의 제왕 영화 3부작》(2001년~2003년)의 감독으로 가장 유명하다. 2005년에는 1933년작 킹콩의 리메이크작 《킹콩(2005)》의 감독을 맡았다.
+"""
+
+tta_labels = ["ARTIFACTS", "ANIMAL", "CIVILIZATION", "DATE", "EVENT", "STUDY_FIELD", "LOCATION", "MATERIAL", "ORGANIZATION", "PERSON", "PLANT", "QUANTITY", "TIME", "TERM", "THEORY"]
+
+entities = model.predict_entities(text, labels)
+
+for entity in entities:
+    print(entity["text"], "=>", entity["label"])
+```
+
+```
+피터 잭슨 경 => PERSON
+1961년 10월 31일 ~ => DATE
+뉴질랜드 => LOCATION
+영화 감독 => OCCUPATION
+각본가 => OCCUPATION
+영화 => OCCUPATION
+프로듀서 => OCCUPATION
+J. R. R. 톨킨 => PERSON
+3부작 => QUANTITY
+2001년~2003년 => DATE
+감독 => OCCUPATION
+2005년 => DATE
+1933년작 => DATE
+킹콩 => ARTIFACTS
+킹콩 => ARTIFACTS
+2005 => DATE
+감독 => OCCUPATION
+```
+
+
+## Named Entity Recognition benchmark result
+
+Evaluate with the [konne dev set](https://github.com/korean-named-entity/konne)
+
+| Model            | Precision (P) | Recall (R) | F1 |
+|------------------|-----------|-----------|--------|
+| Gliner-ko (t=0.5)       | **72.51%**    | **79.82%**    | **75.99%** |
+| Gliner Large-v2 (t=0.5)  | 34.33%    | 19.50%    | 24.87% |
+| Gliner Multi (t=0.5)  | 40.94%    | 34.18%    | 37.26% |
+| Pororo           | 70.25%    | 57.94%    | 63.50% |
+
+## Model Authors
+The model authors are:
+* [Taemin Lee](http://tmkor.com)
+* [Urchade Zaratiana](https://huggingface.co/urchade)
+* Nadi Tomeh
+* Pierre Holat
+* Thierry Charnois
+
+## Citation
+```bibtex
+@misc{zaratiana2023gliner,
+      title={GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer}, 
+      author={Urchade Zaratiana and Nadi Tomeh and Pierre Holat and Thierry Charnois},
+      year={2023},
+      eprint={2311.08526},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}
+```

gliner_config.json

@@ -0,0 +1,23 @@
+{
+  "lr_encoder": "1e-5",
+  "lr_others": "5e-5",
+  "warmup_ratio": 0.1,
+  "max_width": 12,
+  "model_name": "lighthouse/mdeberta-v3-base-kor-further",
+  "fine_tune": true,
+  "subtoken_pooling": "first",
+  "hidden_size": 768,
+  "span_mode": "markerV0",
+  "dropout": 0.4,
+  "root_dir": "ablation_backbone",
+  "prev_path": "none",
+  "size_sup": -1,
+  "max_types": 25,
+  "shuffle_types": true,
+  "random_drop": true,
+  "max_neg_type_ratio": 1,
+  "max_len": 384,
+  "name": "largev2_ko_m",
+  "log_dir": "logs",
+  "tokenizer": "mecab-ko"
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e9263ace777fa9929306d62f1c81378556657f5f3a9fcd2a6ceb6ba7e3e8a636
+size 1209405350