chuuhtetnaing commited on
Commit
407a296
·
verified ·
1 Parent(s): dca60b1

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +12 -12
README.md CHANGED
@@ -97,25 +97,25 @@ Fine-tuned [FacebookAI/xlm-roberta-base](https://huggingface.co/FacebookAI/xlm-r
97
 
98
  ## Usage
99
 
100
- ```python
101
- from transformers import AutoModelForTokenClassification, AutoTokenizer, pipeline
102
 
103
- model = AutoModelForTokenClassification.from_pretrained("chuuhtetnaing/myanmar_text_segmentation_model")
104
- tokenizer = AutoTokenizer.from_pretrained("chuuhtetnaing/myanmar_text_segmentation_model")
105
 
106
- # Using pipeline
107
- nlp = pipeline("token-classification", model=model, tokenizer=tokenizer)
108
- tokens = nlp("အချစ်ဆိုတာလူတွေရှင်သန်ဖို့သဘာဝကပေးတဲ့လက်နက်လား၊ဒါမှမဟုတ်ယဉ်ကျေးမှုအရတီထွင်ထားတဲ့စိတ်ကူးယဉ်မှုသက်သက်လား။")
109
 
110
  segmented_text = []
111
- for item in tokens:
112
- if item["entity_group"] == "B":
113
- segmented_text.append(item["word"])
114
  else: # 'I' - append to previous word
115
- segmented_text[-1] += item["word"]
116
  segmented_text = " ".join(segmented_text)
117
 
118
- return segmented_text
 
 
119
  ```
120
 
121
  ## Label Mapping
 
97
 
98
  ## Usage
99
 
100
+ ### Using Pipeline
 
101
 
102
+ ```python
103
+ from transformers import pipeline
104
 
105
+ nlp = pipeline("token-classification", model="chuuhtetnaing/myanmar-text-segmentation-model", grouped_entities=True)
106
+ segments = nlp("အချစ်ဆိုတာလူတွေရှင်သန်ဖို့သဘာဝကပေးတဲ့လက်နက်လား၊ဒါမှမဟုတ်ယဉ်ကျေးမှုအရတီထွင်ထားတဲ့စိတ်ကူးယဉ်မှုသက်သက်လား။")
 
107
 
108
  segmented_text = []
109
+ for segment in segments:
110
+ if segment["entity_group"] == "B":
111
+ segmented_text.append(segment["word"])
112
  else: # 'I' - append to previous word
113
+ segmented_text[-1] += segment["word"]
114
  segmented_text = " ".join(segmented_text)
115
 
116
+ print(segmented_text)
117
+
118
+ # အချစ်ဆိုတာ လူတွေရှင်သန်ဖို့ သဘာဝကပေးတဲ့လက်နက်လား၊ ဒါမှမဟုတ် ယဉ်ကျေးမှုအရ တီထွင်ထားတဲ့ စိတ်ကူးယဉ်မှုသက်သက်လား။
119
  ```
120
 
121
  ## Label Mapping