Instructions to use chrvngr/pi-fast-tokenizer with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use chrvngr/pi-fast-tokenizer with Transformers:
# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("chrvngr/pi-fast-tokenizer", dtype="auto") - Notebooks
- Google Colab
- Kaggle
| { | |
| "repo_id": "timo/norm_stack_blue_cube_on_red_cube_137", | |
| "vocab_size": 1024, | |
| "scale": 10.0, | |
| "encoded_dims": "0:6", | |
| "encoded_dim_ranges": [ | |
| [ | |
| 0, | |
| 6 | |
| ] | |
| ], | |
| "total_encoded_dims": 6, | |
| "relative_dims": null, | |
| "relative_dim_list": null, | |
| "use_relative_transform": false, | |
| "state_key": "observation.state", | |
| "normalization_mode": "QUANTILES", | |
| "action_horizon": 10, | |
| "num_training_chunks": 4420, | |
| "compression_stats": { | |
| "compression_ratio": 7.040600797934757, | |
| "mean_token_length": 8.522, | |
| "p99_token_length": 17.0, | |
| "min_token_length": 2.0, | |
| "max_token_length": 38.0 | |
| } | |
| } |