ken-sungmin/propagator-tokenizer

Propagator Tokenizer

This repository contains the tokenizer used with the Propagator Multimodal Pretraining Data.

The tokenizer is a byte-level BPE text tokenizer with a small set of special tokens for conversation boundaries and modality markers. It is intended for projects that need the same text and marker vocabulary used by the Propagator multimodal dataset.

Source Code

• GitHub: https://github.com/KennethanCeyer/propagator
• Related dataset: Propagator Multimodal Pretraining Data

Files

• tokenizer.json: Hugging Face tokenizers JSON file.

Token Space

• Base text BPE vocabulary: 16,000 tokens.
• This repository only contains the text tokenizer. Multimodal numeric ids used by the paired dataset are documented in the dataset card.

Special Tokens

Token id	Token	Purpose
0	[PAD]	Padding token
1	[UNK]	Unknown token
2	[SESSION]	Conversation/session start marker
3	[USER]	User turn marker
4	[MODEL]	Model turn marker
5	[LISTEN]	Listening/continuation marker
6	[USER_END]	User turn boundary
7	[MODEL_END]	Model turn boundary
8	[SESSION_END]	Conversation/session boundary
9	[USER_INTERRUPT]	User interruption marker
10	[AUDIO_IN]	Audio input span marker
11	[AUDIO_OUT]	Audio output span marker
12	[AUDIO_END]	Audio span boundary
13	[SILENCE]	Silence marker
14	[TEXT_IN]	Text input span marker
15	[TEXT_OUT]	Text output span marker
16000	[HYBRID_OUT]	Mixed-modality output marker
16001	[IMAGE_IN]	Image input span marker
16002	[TEXT]	Text modality marker
16003	[AUDIO]	Audio modality marker
16004	[IMAGE]	Image modality marker
16005	[HYBRID]	Mixed-modality marker

Quick Start

from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer

repo_id = "ken-sungmin/propagator-tokenizer"
tokenizer_path = hf_hub_download(repo_id, "tokenizer.json")
tokenizer = Tokenizer.from_file(tokenizer_path)

encoded = tokenizer.encode("[SESSION] [USER] [TEXT_IN] Describe the image. [USER_END]")
print(encoded.ids)

Related Dataset

• Dataset: Propagator Multimodal Pretraining Data

Use the dataset card for source coverage, binary frame layout, and reconstruction notes.