Update README.md
Browse files
README.md
CHANGED
|
@@ -63,8 +63,8 @@ Note that this model is primarily aimed at being fine-tuned on tasks that use th
|
|
| 63 |
from transformers import AutoModelForMaskedLM, AutoTokenizer
|
| 64 |
import torch
|
| 65 |
|
| 66 |
-
model = AutoModelForMaskedLM.from_pretrained("
|
| 67 |
-
tokenizer = AutoTokenizer.from_pretrained("
|
| 68 |
|
| 69 |
text = "The key to effective communication is to [MASK] clearly and listen actively."
|
| 70 |
inputs = tokenizer(text, return_tensors="pt")
|
|
@@ -80,4 +80,44 @@ top_5_tokens = torch.topk(mask_token_logits, 5, dim=1).indices[0].tolist()
|
|
| 80 |
|
| 81 |
for token in top_5_tokens:
|
| 82 |
print(f"{tokenizer.decode([token])}")
|
| 83 |
-
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 63 |
from transformers import AutoModelForMaskedLM, AutoTokenizer
|
| 64 |
import torch
|
| 65 |
|
| 66 |
+
model = AutoModelForMaskedLM.from_pretrained("appleroll/coberta-base")
|
| 67 |
+
tokenizer = AutoTokenizer.from_pretrained("appleroll/coberta-base")
|
| 68 |
|
| 69 |
text = "The key to effective communication is to [MASK] clearly and listen actively."
|
| 70 |
inputs = tokenizer(text, return_tensors="pt")
|
|
|
|
| 80 |
|
| 81 |
for token in top_5_tokens:
|
| 82 |
print(f"{tokenizer.decode([token])}")
|
| 83 |
+
```
|
| 84 |
+
|
| 85 |
+
### Citation
|
| 86 |
+
|
| 87 |
+
If you use CoBERTa in your research, please cite:
|
| 88 |
+
```bibtex
|
| 89 |
+
@misc{coberta,
|
| 90 |
+
title = {CoBERTa: Training Domain-Expert Language Models on Consumer Hardware},
|
| 91 |
+
url = {https://huggingface.co/appleroll/coberta-base},
|
| 92 |
+
author = {Zhang, Ethan},
|
| 93 |
+
year = {2025}
|
| 94 |
+
}
|
| 95 |
+
```
|
| 96 |
+
|
| 97 |
+
### Additional Information
|
| 98 |
+
Authors:
|
| 99 |
+
- Ethan Zhang (Independent Researcher)
|
| 100 |
+
|
| 101 |
+
### Contact
|
| 102 |
+
|
| 103 |
+
For questions, feedback, or collaboration: ethanzhangyixuan@gmail.com
|
| 104 |
+
|
| 105 |
+
### Acknowledgements
|
| 106 |
+
|
| 107 |
+
- Cosmopedia dataset creators (HuggingfaceTB)
|
| 108 |
+
|
| 109 |
+
- Apple MLX development team
|
| 110 |
+
|
| 111 |
+
- Hugging Face for model hosting
|
| 112 |
+
|
| 113 |
+
- Open-source ML community
|
| 114 |
+
|
| 115 |
+
### Version
|
| 116 |
+
|
| 117 |
+
Current: v0.5 (Experimental Release)
|
| 118 |
+
|
| 119 |
+
Previous: None
|
| 120 |
+
|
| 121 |
+
### License
|
| 122 |
+
|
| 123 |
+
This model is licensed under the MIT License. See the LICENSE file for details.
|