Functionalizer-100M
Collection
11 items • Updated
How to use from
SGLangUse Docker images
docker run --gpus all \
--shm-size 32g \
-p 30000:30000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--env "HF_TOKEN=<secret>" \
--ipc=host \
lmsysorg/sglang:latest \
python3 -m sglang.launch_server \
--model-path "mrkwanzaa/functionalizer-100M-github-code-python-seed2" \
--host 0.0.0.0 \
--port 30000# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
-H "Content-Type: application/json" \
--data '{
"model": "mrkwanzaa/functionalizer-100M-github-code-python-seed2",
"prompt": "Once upon a time,",
"max_tokens": 512,
"temperature": 0.5
}'Quick Links
Functionalizer 100M model
This is a 100M example model. Each model is trained on the dataset specified in the name for 50000 epochs with the given seed.
The model was trained using the Functionalizer framework. Training code and detailed performance analysis are available: https://github.com/connor-makowski/functionalizer
Paper: https://huggingface.co/papers/2609.15991
Running the model
To run the model, use the custom tokenizer fork available here: https://github.com/connor-makowski/tokenizers/tree/functionalizer
Citation
If you find this model or the Functionalizer framework useful, please cite:
@misc{makowski2026functionalizerlosslessfunctionaldecomposition,
title={The Functionalizer: Lossless Functional Decomposition for Subword Tokenization},
author={Connor Makowski and Willem Guter},
year={2026},
eprint={2609.15991},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2609.15991},
}
- Downloads last month
- 325
Install from pip and serve model
# Install SGLang from pip: pip install sglang# Start the SGLang server: python3 -m sglang.launch_server \ --model-path "mrkwanzaa/functionalizer-100M-github-code-python-seed2" \ --host 0.0.0.0 \ --port 30000# Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "mrkwanzaa/functionalizer-100M-github-code-python-seed2", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'