---
license: apache-2.0
---

# Interchangeable Token Embeddings

This repository contains the models for the ICML 2025 paper, [Interchangeable Token Embeddings for Extendable Vocabulary and Alpha-Equivalence](https://arxiv.org/abs/2410.17161).

Links:
1. [arXiv](https://arxiv.org/abs/2410.17161)
2. [Project Page](https://necrashter.github.io/interchangeable-token-embeddings/)
3. [Code](https://github.com/necrashter/interchangeable-token-embeddings)

## Models

Figure 3a (LTL):
| Model | Path |
| ----- | ---- |
| Proposed Method | `models/ltl-5/d005-rop-bn1-fn1-ada1-s42` |
| Baseline | `models/adacos/nondyn-rope` |
| Full-Vocabulary | `models/10ap/adacos-nondyn` |
| Alpha-Renaming | `models/ltl-5/s010-rop-bn1-fn1-ada1-s46` |

Figure 3b (Propositional Logic):
| Model | Path |
| ----- | ---- |
| Proposed Method | `models-prop/5ap/d020-rop-bn1-fn1-ada1-s44` |
| Baseline | `models-prop/5ap/0000-rop-bn0-fn1-ada1-s42` |
| Full-Vocabulary | `models-prop/10ap/0000-rop-bn0-fn1-ada1-s43` |
| Alpha-Renaming | `models-prop/5ap/s010-rop-bn1-fn1-ada1-s44` |

Figure 5a (Ablation - LTL):
| Model | Path |
| ----- | ---- |
| w/o Adacos | `models/ltl-5/d005-rop-bn1-fn1-ada0-s42` |
| w/o $f_{{fn}}$ & AdaCos | `models/ltl-5/d005-rop-bn1-fn0-ada0-s42` |
| w/o $f_{{bn}}$ | `models/ltl-5/d005-rop-bn0-fn1-ada1-s42` |

Figure 5b (Ablation - Propositional Logic):
| Model | Path |
| ----- | ---- |
| w/o Adacos | `models-prop/5ap/d020-rop-bn1-fn1-ada0-s43` |
| w/o $f_{{fn}}$ & AdaCos | `models-prop/5ap/d020-rop-bn1-fn0-ada0-s44` |
| w/o $f_{{bn}}$ | `models-prop/5ap/d020-rop-bn0-fn1-ada1-s42` |

### Perturbation Experiments

Table 2 and 7 (LTL):
| Model | Path |
| ----- | ---- |
| Normal Baseline | `models/adacos/nondyn-rope` |
| Perturbed Baseline | `models/adacos/nondyn-rope-perturbed` |
| Perturbed Alpha-Renaming | `models/adacos/s005-rope-perturbed-s42` |
| Perturbed Proposed | `models/adacos/diagbor4-rope-perturbed` |
| Limited Baseline | `models/adacos/nondyn-rope-limited` |
| Limited Alpha-Renaming | `models/adacos/s005-rope-limited-s42` |
| Limited Proposed | `models/adacos/diagbor4-rope-limited` |

Table 8 (Propositional Logic):
| Model | Path |
| ----- | ---- |
| Normal Baseline | `models-prop/5ap/0000-rop-bn0-fn1-ada1-s42` |
| Perturbed Baseline | `models-prop/5ap/base-rop-bn0-fn1-ada1-s44` |
| Perturbed Alpha-Renaming | `models-prop/5ap/s005-rop-bn1-fn1-ada1-s43` |
| Perturbed Proposed | `models-prop/5ap/d020-rop-bn1-fn1-ada1-s42` |