Mobile deployment question — can LTX-2.3 run on phones?
#59
by 3morixd - opened
This comment has been hidden (marked as Off-Topic)
Hi,
When running on a phone, you have the advantage of the shared memory architecture, but you are most likely bound on compute.
LTX-2.3 is extremely efficient in that sense, given the low number of tokens allow for less compute in attention. That said, in order to make it useable on SnapDragon or A series processors, it's best to invest in refining it.
My goto would be:
- Weight distillation to a smaller model - 22B parameters is still a lot, even after quantization.
- Quantization Aware Training to go to 4 bits, most likely INT4 as the target since it's mobile chips.
- Finetune on the target resolution, as well as step distillation - The model is geared towards 1080p and up, if you want 512x512 it's best to finetune and while you're at it, distill it to 1/2 steps.
Currently, our focus in mobile edge is for Physical AI, so it's not exactly the same as phones.