Hugging Face's logo

SzegedAI
/
hubertusz-tiny-wiki-seq128

Transformers
PyTorch
TensorFlow
Hungarian
bert
pretraining
generated_from_keras_callback
hubert
Model card Files
xet
 Community
1
How to use from the
Use from the
Transformers library
# Load model directly
from transformers import AutoTokenizer, AutoModelForPreTraining

tokenizer = AutoTokenizer.from_pretrained("SzegedAI/hubertusz-tiny-wiki-seq128")
model = AutoModelForPreTraining.from_pretrained("SzegedAI/hubertusz-tiny-wiki-seq128")
Quick Links

hubert-tiny-wiki-seq128

Fully trained model with the second phase of training is available here: SzegedAI/hubert-tiny-wiki

This model was trained from scratch on the Wikipedia subset of Hungarian Webcorpus 2.0 with MLM and SOP tasks.

Pre-Training Parameters:

  • Training steps: 500.000
  • Sequence length: 128 (the model is capable for 512)
  • Batch size: 1024

Framework versions

  • Transformers 4.21.3
  • TensorFlow 2.10.0
  • Datasets 2.4.0
  • Tokenizers 0.12.1

Acknowledgement

Artificial Intelligence - National Laboratory - Hungary

Downloads last month
6
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for SzegedAI/hubertusz-tiny-wiki-seq128

Quantizations
1 model

Dataset used to train SzegedAI/hubertusz-tiny-wiki-seq128

Collection including SzegedAI/hubertusz-tiny-wiki-seq128