Otzria MLM V2
诪讜讚诇 砖驻讛 注讘专讬 讛诪讘讜住住 注诇 AlephBERT 讜注讘专 讗讬诪讜谉 谞讜住祝 (Domain Adaptation) 注诇 讟拽住讟讬诐 转讜专谞讬讬诐 诪转讜讱 驻专讜讬拽讟 讗讜爪专讬讗.
讛诪讜讚诇 诪讜转讗诐 讘诪讬讜讞讚 诇讟拽住讟讬诐 讻讙讜谉:
- 转诇诪讜讚
- 住驻专讜转 专讗砖讜谞讬诐 讜讗讞专讜谞讬诐
- 讛诇讻讛
- 诪讚专砖讬诐
- 住驻专讜转 转讜专谞讬转 拽诇讗住讬转
Base Model
讛诪讜讚诇 诪讘讜住住 注诇
onlplab/alephbert-base
讗砖专 驻讜转讞 注诇 讬讚讬
ONLP Lab
Architecture
- Model type: BERT
- Task: Masked Language Modeling (MLM)
- Hidden size: 768
- Layers: 12
- Attention heads: 12
Vocabulary
讛诪讜讚诇 诪砖转诪砖 讘讟讜拽谞讬讬讝专 诪讜专讞讘 注诐 讗讜爪专 诪讬诇讬诐 讞讚砖:
Vocabulary size: {vocab_size}
讛专讞讘转 讛-Vocabulary 谞讜注讚讛 诇砖驻专 讘讬爪讜注讬诐 注诇:
- 诪讜谞讞讬诐 转诇诪讜讚讬讬诐
- 诇砖讜谉 讞讝"诇
- 砖诪讜转 住驻专讬诐 讜讞讻诪讬诐
- 讘讬讟讜讬讬诐 转讜专谞讬讬诐 谞驻讜爪讬诐
Training Details
Training Objective
Masked Language Modeling (MLM)
Special Training Strategy
讛讗讬诪讜谉 讛砖转诪砖 讘讟讻谞讬拽讛 砖诇 Weighted MLM:
- 讛住转讘专讜转 masking 专讙讬诇讛: 15%
- 讛住转讘专讜转 masking 诇讟讜拽谞讬诐 讞讚砖讬诐: 45%
- 诪砖拽诇 loss 诇讟讜拽谞讬诐 讞讚砖讬诐: 脳4
讛诪讟专讛 讛讬讬转讛 诇讙专讜诐 诇诪讜讚诇 诇诇诪讜讚 讘诪讛讬专讜转 讗转 讛讟讜拽谞讬诐 讛讞讚砖讬诐 砖谞讜住驻讜 诇-Vocabulary.
Training Configuration
| Parameter | Value |
|---|---|
| Base model | onlplab/alephbert-base |
| Epochs | 1 |
| Batch size | 32 |
| Learning rate | 3e-5 |
| Weight decay | 0.01 |
| Warmup ratio | 0.03 |
Fine-tuned Layers
讛讗讬诪讜谉 讘讜爪注 注诇:
- Embedding layer
- 4 砖讻讘讜转 encoder 讗讞专讜谞讜转
- MLM head
砖讗专 讛砖讻讘讜转 讛讜拽驻讗讜.
Dataset
讛讗讬诪讜谉 讘讜爪注 注诇 拽讜专驻讜住 讟拽住讟讬诐 转讜专谞讬讬诐 诪转讜讱 驻专讜讬拽讟 讗讜爪专讬讗.
讛拽讜专驻讜住 讻讜诇诇 讟拽住讟讬诐 诪转拽讜驻讜转 砖讜谞讜转 砖诇 讛住驻专讜转 讛转讜专谞讬转, 讻讙讜谉:
- 转诇诪讜讚
- 专讗砖讜谞讬诐
- 讗讞专讜谞讬诐
- 住驻专讜转 讛诇讻转讬转
Usage
from transformers import pipeline
fill_mask = pipeline(
\"fill-mask\",
model=\"ArieLLL123/otzria-mlm-V2\"
)
results = fill_mask(\"诪驻谞讬 诪讛 [MASK] 讞专砖 砖讜讟讛 讜拽讟谉 诇讗?\")
for r in results:
print(r[\"sequence\"], r[\"score\"])
Example
Input:
诪驻谞讬 诪讛 [MASK] 讞专砖 砖讜讟讛 讜拽讟谉 诇讗?
Output (example):
诪驻谞讬 诪讛 讗诪专 讞专砖 砖讜讟讛 讜拽讟谉 诇讗?
诪驻谞讬 诪讛 讗诪专讜 讞专砖 砖讜讟讛 讜拽讟谉 诇讗?
Intended Use
讛诪讜讚诇 诪讬讜注讚 诇:
- 诪讞拽专 NLP 讘注讘专讬转 转讜专谞讬转
- 谞讬转讜讞 讟拽住讟讬诐 转讜专谞讬讬诐
- 讞讬驻讜砖 住诪谞讟讬
- 讛砖诇诪转 讟拽住讟讬诐
- preprocessing 诇诪讜讚诇讬诐 讙讚讜诇讬诐 讬讜转专
Limitations
- 讛诪讜讚诇 讗讜诪谉 讘注讬拽专 注诇 讟拽住讟讬诐 转讜专谞讬讬诐 讜诇讻谉 讘讬爪讜注讬讜 注诇 注讘专讬转 诪讜讚专谞讬转 注诇讜诇讬诐 诇讛讬讜转 驻讞讜转 讟讜讘讬诐.
- 诪讜讚诇 MLM 讗讬谞讜 诪讬讜注讚 诇-generation 诪诇讗 砖诇 讟拽住讟.
License
AGPL-3.0
诪砖诪注讜转 讛讚讘专 讛讬讗 砖讻诇 砖讬诪讜砖 讘诪讜讚诇 讘驻专讜讬拽讟 谞讙讝专 诪讞讬讬讘 驻专住讜诐 讛拽讜讚 砖诇 讛驻专讜讬拽讟 转讞转 讗讜转讜 专讬砖讬讜谉.
Acknowledgments
诪讜讚诇 讝讛 诪讘讜住住 注诇 AlephBERT 讜注诇 注讘讜讚转 Bar-Ilan University 讘转讞讜诐 NLP 诇注讘专讬转.
- Downloads last month
- 47