README.md

metadata

title: AlienLM
emoji: 🛸
colorFrom: purple
colorTo: green
sdk: gradio
sdk_version: 6.19.0
app_file: app.py
short_description: AlienLM privacy layer for black-box LLMs
python_version: '3.12'
startup_duration_timeout: 30m

AlienLM Space

Interactive research demo for AlienLM, using the public dsba-lab/gemma2-9b-it-alienlm-full checkpoint.

The Space uses the official repository's token-ID translation rule and startup compatibility checks. Its inference path matches the SingleAlienTokenizer used for training:

1. Render the natural-language chat with the original Gemma 2 tokenizer, then translate the rendered text into alien text.
2. Encode the alien text with the original tokenizer, matching the SingleAlienTokenizer path used during training.
3. Decode generated token IDs with the unchanged original tokenizer, matching a black-box server that returns text and no token metadata.
4. On the client side, invert the original tokenizer's decoded-piece lattice, keep the minimum-token segmentations, and rank their alien-decoded natural text locally. The server tokenizer is not modified.

The Space includes the alien tokenizer files required by the model. Access to the gated google/gemma-2-9b-it tokenizer is supplied through the Space's HF_TOKEN secret.

Privacy scope

This hosted Space is an interactive pipeline demo: natural-language input first reaches the Space backend, where translation and model inference run in the same process. In the deployment described by the paper, the translator and bijection remain on a trusted client so that only alienized text crosses the black-box LLM API boundary. AlienLM is a research method, not a formal cryptographic privacy guarantee.

The inverse segmentation step is necessarily heuristic when the server returns only a fused string: multiple token-ID sequences can decode to the same text. The demo uses minimum token count as a strong prior and resolves collisions using only tokenizer-internal signals: alien-side canonicality, structural validity, and the original tokenizer's BPE vocabulary rank. It does not use a second language model. Special tokens intentionally removed by the server cannot be reconstructed, but are not part of the visible response content. The search budget is intentionally tuned for the Space's short demo responses; long-form reconstruction is outside this demo's scope.

Links

• Paper
• Project page
• Official code
• Model collection