Instructions to use HauserGroup/ApeTokenizer-SELFIES with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use HauserGroup/ApeTokenizer-SELFIES with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("HauserGroup/ApeTokenizer-SELFIES", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Commit History
-
Harden APE tokenizer: malformed-row tolerance, freq accuracy, perf
Verified signature- Skip malformed sequences during training instead of aborting the run, and map malformed input to <unk> at encode time so it stays detectable via unk_rate rather than raising mid-batch. - Debit both constituents when a merge is applied so *_freq.json reflects accurate post-merge counts; primitive keys are retained for coverage. - Drop the per-merge full copy of pair_counts (never read) and reuse untouched sequences by reference during rebuild. Output is unchanged; merge order stays pinned for reproducibility. - Document that encoding is greedy longest-match, not a replay of learned merge order, and that both phases share it. - Remove dead two-letter metal branches from SMILES_RE (always bracketed in canonical SMILES); verified identical splits on the shipped vocab. - Correct the misleading "unbalanced parens = defect" comment: boundary- crossing SMILES tokens are inherent to APE and match the paper.6625342
-
Add APE SELFIES tokenizer with max length 256
Verified signaturefdd34dc
-
Add APE SELFIES tokenizer with max length 256
Verified signature891e224
-
Add APE SELFIES tokenizer
Verified signatureb08272b -
initial commit
Verified signature0535628