|
|
--- |
|
|
library_name: transformers |
|
|
license: mit |
|
|
base_model: sbintuitions/tiny-lm |
|
|
tags: |
|
|
- generated_from_trainer |
|
|
datasets: |
|
|
- HuggingFaceFW/fineweb |
|
|
model-index: |
|
|
- name: output-tiny-lm-fineweb |
|
|
results: [] |
|
|
language: |
|
|
- en |
|
|
--- |
|
|
|
|
|
<!-- This model card has been generated automatically according to the information the Trainer had access to. You |
|
|
should probably proofread and complete it, then remove this comment. --> |
|
|
|
|
|
# UTF8-LM-tiny |
|
|
|
|
|
This model is a fine-tuned version of [sbintuitions/tiny-lm](https://huggingface.co/sbintuitions/tiny-lm) on the HuggingFaceFW/fineweb dataset. |
|
|
|
|
|
Using [this](https://github.com/sign/utf8-tokenizer/blob/main/experiments/language-modelling/run_clm.py) training script, from [utf8-tokenizer](https://github.com/sign/utf8-tokenizer/tree/main). |
|
|
|
|
|
The repository includes the joined model for ease of use, and the [bit_projection_weights.pt](https://huggingface.co/sign/utf8-lm-tiny/blob/main/bit_projection_weights.pt) for further analysis. |
|
|
|
|
|
## Usage |
|
|
|
|
|
```python |
|
|
from transformers import AutoModelForCausalLM |
|
|
import torch |
|
|
|
|
|
from utf8_tokenizer import UTF8Tokenizer |
|
|
|
|
|
model_id = "sign/utf8-lm-tiny" |
|
|
|
|
|
tokenizer = UTF8Tokenizer() |
|
|
model = AutoModelForCausalLM.from_pretrained(model_id) |
|
|
|
|
|
prompt = "My name is" |
|
|
|
|
|
inputs = tokenizer([prompt], return_tensors="pt", |
|
|
padding=True, |
|
|
add_special_tokens=True) |
|
|
inputs["input_ids"] = inputs["input_ids"].to(torch.long) |
|
|
# We need to remove the EOS token |
|
|
inputs["input_ids"] = inputs["input_ids"][:, :-1] |
|
|
inputs["attention_mask"] = inputs["attention_mask"][:, :-1] |
|
|
|
|
|
|
|
|
with torch.no_grad(): |
|
|
out = model.generate( |
|
|
**inputs, |
|
|
max_new_tokens=64, |
|
|
) |
|
|
|
|
|
print(tokenizer.decode(out[0], skip_special_tokens=False)) |
|
|
``` |
|
|
|
|
|
## Training procedure |
|
|
|
|
|
```shell |
|
|
python run_clm.py \ |
|
|
--use_bit_embeddings True \ |
|
|
--output_dir ./output-tiny-lm-fineweb \ |
|
|
--dataset_name HuggingFaceFW/fineweb \ |
|
|
--streaming True \ |
|
|
--dataloader_num_workers 1 \ |
|
|
--dataloader_prefetch_factor 4 \ |
|
|
--dataloader_pin_memory True \ |
|
|
--dataloader_persistent_workers True \ |
|
|
--do_train True \ |
|
|
--save_strategy steps \ |
|
|
--max_steps 20000 \ |
|
|
--save_steps 1000 \ |
|
|
--save_total_limit 2 \ |
|
|
--logging_steps 100 \ |
|
|
--logging_strategy steps \ |
|
|
--model_name_or_path sbintuitions/tiny-lm \ |
|
|
--per_device_train_batch_size 128 \ |
|
|
--block_size 256 \ |
|
|
--optim adamw_torch_fused \ |
|
|
--learning_rate 3e-4 \ |
|
|
--lr_scheduler_type cosine \ |
|
|
--warmup_ratio 0.01 \ |
|
|
--weight_decay 0.1 \ |
|
|
--adam_beta1 0.9 \ |
|
|
--adam_beta2 0.95 \ |
|
|
--max_grad_norm 1.0 \ |
|
|
--gradient_checkpointing True \ |
|
|
--bf16 True \ |
|
|
--seed 42 \ |
|
|
--report_to wandb \ |
|
|
--include_num_input_tokens_seen True |
|
|
``` |
|
|
|
|
|
|
|
|
### Framework versions |
|
|
|
|
|
- Transformers 4.57.3 |
|
|
- Pytorch 2.9.1+cu130 |
|
|
- Datasets 4.4.1 |
|
|
- Tokenizers 0.22.1 |
