Add special token for table tokenization fa6d40e
AUGUST-GKD commited on
How to use DIAL-TFM/TFM-tokenizer with Transformers:
# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("DIAL-TFM/TFM-tokenizer", dtype="auto")