skywolf46 commited on
Commit
24fc67f
ยท
verified ยท
1 Parent(s): 963212f

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +67 -1
README.md CHANGED
@@ -1,3 +1,69 @@
1
  ---
2
- license: apache-2.0
 
 
 
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ license: mit
3
+ language:
4
+ - ko
5
+ - en
6
+ base_model: jhu-clsp/mmBERT-small
7
+ base_model_relation: finetune
8
+ tags:
9
+ - pytorch
10
+ - safetensors
11
+ - text-segmentation
12
+ - semantic-chunking
13
+ - modernbert
14
+ - prototype
15
  ---
16
+
17
+
18
+ # Backfire Borderline (Rev 232)
19
+ ๋งž๋ถˆ ๋ฐฉํ™”์„  ๋ชจ๋ธ์€ ํ›„์† ๋ชจ๋ธ์— ์ „๋‹ฌํ•˜๊ธฐ ์šฉ์ดํ•˜๋„๋ก ์ฃผ์ œ๋ฅผ ๊ธฐ๋ฐ˜์œผ๋กœ ์ ๋‹นํžˆ ๋ณธ๋ฌธ์„ ๋ถ„ํ• ํ•˜๋Š” ๊ฒƒ์„ ๋ชฉํ‘œ๋กœ ํ•ฉ๋‹ˆ๋‹ค.
20
+
21
+ ๊ธฐ์กด์˜ ๋ชจ๋ธ ํ˜•ํƒœ์™€๋Š” ๋‹ค๋ฅด๊ฒŒ KEEP / CLOSE ํ˜•ํƒœ๋กœ ์ถœ๋ ฅํ•˜๋ฉฐ, ์ด๋ฅผ ํ†ตํ•ด ํ˜„์žฌ ๋ถ€๋ถ„์—์„œ ๋ณธ๋ฌธ์„ ๋Š์–ด์•ผ ํ• ์ง€, ์•„๋‹ˆ๋ฉด ๊ณ„์† ์ด์–ด๊ฐ€์•ผํ• ์ง€๋ฅผ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค.
22
+
23
+ ์ด ํ”„๋กœ์ ํŠธ๋Š” **๋ฐœํ™” ํŠน์ด์ ** ํ”„๋กœ์ ํŠธ์˜ ์ผ๋ถ€์ž…๋‹ˆ๋‹ค.
24
+
25
+ ## ๋น ๋ฅธ ์‹œ์ž‘
26
+
27
+ Python 3.11 ์ด์ƒ์„ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. ๋ชจ๋ธ ์ €์žฅ์†Œ๋ฅผ ๋กœ์ปฌ ํด๋”๋กœ ๋‹ค์šด๋กœ๋“œํ•˜๊ฑฐ๋‚˜ ๋ณต์ œํ•œ ๋’ค,
28
+ ์‚ฌ์šฉํ•  CPUยทGPU ํ™˜๊ฒฝ์— ๋งž๋Š” PyTorch ํŒจํ‚ค์ง€๋ฅผ ์„ค์น˜ํ•˜๊ณ  ๋‹ค์Œ ๋ช…๋ น์„ ์‹คํ–‰ํ•ฉ๋‹ˆ๋‹ค.
29
+
30
+ ```bash
31
+ cd bfb232
32
+ python -m pip install -r requirements.txt
33
+ python bfb232.py --model . --language ko --input input.txt --output result.json
34
+ ```
35
+
36
+ Python์—์„œ๋Š” ์ด ๋ ˆํฌ์ง€ํ† ๋ฆฌ๋ฅผ ๊ฐ™์€ ๊ฒฝ๋กœ์— ๋‘” ์ƒํƒœ์—์„œ ๋‹ค์Œ๊ณผ ๊ฐ™์ด ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค.
37
+
38
+ ```python
39
+ from bfb232 import BFB232
40
+
41
+ model = BFB232.from_pretrained(".", device="auto")
42
+ text = "์—ฌ์„ฏ ์‚ด ์ ์— ๋‚˜๋Š” "์ฒดํ—˜ํ•œ ์ด์•ผ๊ธฐ"๋ผ๋Š” ์ œ๋ชฉ์˜, ์›์‹œ๋ฆผ์— ๊ด€ํ•œ ์ฑ…์—์„œ ๊ธฐ๋ง‰ํžŒ ๊ทธ๋ฆผ ํ•˜๋‚˜๋ฅผ ๋ณธ ์ ์ด ์žˆ๋‹ค. ๋งน์ˆ˜๋ฅผ ์ง‘์–ด์‚ผํ‚ค๊ณ  ์žˆ๋Š” ๋ณด์•„ ๊ตฌ๋ ์ด ๊ทธ๋ฆผ์ด์—ˆ๋‹ค. ์œ„์˜ ๊ทธ๋ฆผ์€ ๊ทธ๊ฒƒ์„ ์˜ฎ๊ฒจ ๊ทธ๋ฆฐ ๊ฒƒ์ด๋‹ค..."
43
+ result = model.segment(text, language="ko")
44
+ for segment in result["segments"]:
45
+ print(segment["start"], segment["end"], repr(segment["text"]))
46
+ assert "".join(s["text"] for s in result["segments"]) == text
47
+ ```
48
+
49
+ `language`์—๋Š” ํ•œ๊ตญ์–ด `ko` ๋˜๋Š” ์˜์–ด `en`์„ ์ง€์ •ํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค.
50
+ ์–ธ์–ด ์ž๋™ ๊ฐ์ง€๋‚˜ ํ˜ผํ•ฉ ์–ธ์–ด ํŒ๋ณ„์€ ์ œ๊ณตํ•˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค - ์ž…๋ ฅ๋œ ๋ฐ์ดํ„ฐ๋ฅผ ๊ธฐ๋ฐ˜์œผ๋กœ ํŒ๋ณ„ํ•จ์„ ์ถ”์ฒœํ•ฉ๋‹ˆ๋‹ค.
51
+
52
+ ## ๋ฐฐํฌ ๊ตฌ์„ฑ
53
+
54
+ | ํ•ญ๋ชฉ | ๋‚ด์šฉ |
55
+ |---|---|
56
+ | ๊ธฐ๋ฐ˜ ๋ชจ๋ธ | ํŒŒ์ธ ํŠœ๋‹๋œ `jhu-clsp/mmBERT-small` |
57
+ | ์ธ์ฝ”๋” ํŒŒ๋ผ๋ฏธํ„ฐ ์ˆ˜ | 140,493,696 |
58
+ | ์ฒดํฌํฌ์ธํŠธ | 232๋ฒˆ์งธ ์‹คํ—˜๋ณธ (์‹œ๋“œ 1026914109) |
59
+ | ๋ถ„ํ•  ํŒ์ • | `close_logit - keep_logit >= ์ž„๊ณ„๊ฐ’`์ด๋ฉด `CLOSE` |
60
+ | ์ž„๊ณ„๊ฐ’ | ํ•œ๊ตญ์–ด โˆ’1.234375, ์˜์–ด โˆ’1.4140625 |
61
+
62
+
63
+ ## ์ž…๋ ฅยท์ถœ๋ ฅ ํฌ๋งท
64
+ - CRLFยทCR ์ค„๋ฐ”๊ฟˆ์€ ๋‚ด๋ถ€์ ์œผ๋กœ LF๋กœ ์ •๊ทœํ™”ํ•˜๋ฉฐ, ๋ฐ˜ํ™˜ํ•˜๋Š” ๊ตฌ๊ฐ„์—๋Š” ์›๋ฌธ์˜ ์ค„๋ฐ”๊ฟˆ๊ณผ ๊ณต๋ฐฑ์„ ๋ณด์กดํ•ฉ๋‹ˆ๋‹ค.
65
+ - ๋ถ„ํ•  ํ›„๋ณด ์œ„์น˜๋งˆ๋‹ค KEEP(์ด์–ด ๋ถ™์ด๊ธฐ) / CLOSE(๊ตฌ๊ฐ„ ๋๋‚ด๊ธฐ)๋ฅผ ํŒ๋‹จํ•˜๊ณ , ์›๋ฌธ์„ ๋ณด์กดํ•œ ๊ตฌ๊ฐ„์„ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค.
66
+
67
+ ## ์ฃผ์˜์‚ฌํ•ญ
68
+ - ํ•ด๋‹น ํ”„๋กœ์ ํŠธ๋Š” ์ž‘์—…์˜ ์†๋„๋ฅผ ์œ„ํ•ด ChatGPT 6 Astra๋ฅผ ์ด์šฉํ•˜์—ฌ ์ฝ”๋“œ๊ฐ€ ์ž‘์„ฑ๋˜์—ˆ์Šต๋‹ˆ๋‹ค.
69
+ ์ผ๋ถ€ ์ฝ”๋“œ์—์„œ ๋ถˆํ•„์š”ํ•œ ํ˜ธ์ถœ์ด ๋ฐœ์ƒํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.