| tags: | |
| - llaza | |
| - zip2zip | |
| - adaptive-tokenization | |
| library_name: zip2zip | |
| # zip2zip-phi35-ozzdata-v1 | |
| Training checkpoint from [zip2zip-core](https://github.com/epfl-dlab/zip2zip-core). | |
| This is a **candidate** model (not production-ready). | |
| ## Training Config | |
| | Field | Value | | |
| |-------|-------| | |
| | model_config | `Phi3.5-mini` | | |
| | init_from | `microsoft/Phi-3.5-mini-instruct` | | |
| | max_subtokens | 4 | | |
| | max_codebook_size | 2048 | | |
| | seq_len | 2048 | | |
| | lr | 0.0003 | | |
| | max_tokens | 262144000 | | |
| | step | 8000 | | |
| | data | `phi-1B-sft-ozzaligned` | | |
| ## Usage | |
| This is a **training checkpoint** (torchtitan format). To use for inference, | |
| export to HuggingFace format first: | |
| ```bash | |
| python scripts/zip2zip_hf/export_to_zip2zip.py \ | |
| --ckpt_dir <local_path>/step_8000 \ | |
| --output_dir <export_dir> \ | |
| --base_model microsoft/Phi-3.5-mini-instruct \ | |
| --model_config Phi3.5-mini | |
| ``` | |