File size: 2,898 Bytes
df4af2d 4268c51 df4af2d 4268c51 df4af2d 67b626f df4af2d 67b626f 4268c51 67b626f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 |
---
language:
- en
license: mit
tags:
- roberta
- mlm
- small-model
- academic-language
- synthetic-data
- resource-efficient
- consumer-hardware
pipeline_tag: fill-mask
datasets:
- HuggingFaceTB/cosmopedia
library_name: transformers
---
# CoBERTa: Consumer-friendly Models that Punch Above Weight
## Model Description
CoBERTa is a **24.5 million parameter** RoBERTa-style masked language model specifically trained on synthetic academic data. It demonstrates that **high-quality synthetic data** can compensate for model scale, enabling domain specialization, formal logic and reasoning on consumer hardware.
It achieves academic language proficiency with 5× fewer parameters than comparable models on certain tasks, trained in around 6 hours on consumer hardware.
It was trained on HuggingfaceTB's Cosmopedia dataset.
## Intended uses & limitations
You can use the raw model for masked language modeling, but it's mostly intended to be fine-tuned on a downstream task. See the model hub to look for fine-tuned versions on a task that interests you.
Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked) to make decisions, such as sequence classification, token classification or question answering. For tasks such as text generation you should look at a model like GPT2.
### Basic Usage
```python
from transformers import AutoModelForMaskedLM, AutoTokenizer
import torch
model = AutoModelForMaskedLM.from_pretrained("appleroll/coberta-base")
tokenizer = AutoTokenizer.from_pretrained("appleroll/coberta-base")
text = "The key to effective communication is to [MASK] clearly and listen actively."
inputs = tokenizer(text, return_tensors="pt")
with torch.no_grad():
outputs = model(**inputs)
predictions = outputs.logits
# Get top predictions for [MASK]
mask_token_index = torch.where(inputs.input_ids == tokenizer.mask_token_id)[1]
mask_token_logits = predictions[0, mask_token_index, :]
top_5_tokens = torch.topk(mask_token_logits, 5, dim=1).indices[0].tolist()
for token in top_5_tokens:
print(f"{tokenizer.decode([token])}")
```
### Citation
If you use CoBERTa in your research, please cite:
```bibtex
@misc{coberta,
title = {CoBERTa: Consumer-friendly Models that Punch Above Weight},
url = {https://huggingface.co/appleroll/coberta-base},
author = {Zhang, Ethan},
year = {2025}
}
```
### Additional Information
Authors:
- Ethan Zhang (Independent Researcher)
### Contact
For questions, feedback, or collaboration: ethanzhangyixuan@gmail.com
### Acknowledgements
- Cosmopedia dataset creators (HuggingfaceTB)
- Apple MLX development team
- Hugging Face for model hosting
- Open-source ML community
### Version
Current: v0.5 (Experimental Release)
Previous: None
### License
This model is licensed under the MIT License. See the LICENSE file for details.
|