README.md

---
license: bsd-3-clause-clear
base_model: Qualcomm-AI-Research/Neodragon
tags:
  - text-to-video
  - litert
  - tflite
  - on-device
  - android
library_name: litert
---

# Neodragon — LiteRT (.tflite) conversions for on-device text-to-video

LiteRT conversions of [Qualcomm-AI-Research/Neodragon](https://huggingface.co/Qualcomm-AI-Research/Neodragon),
prepared for [Box](https://github.com/jegly/Box) — an open-source, fully offline
on-device AI app for Android.

These are **converted artifacts only**. The models, research and original weights
are Qualcomm AI Research's work; see the
[upstream repo](https://huggingface.co/Qualcomm-AI-Research/Neodragon) and
[paper](https://qualcomm-ai-research.github.io/neodragon).

Generates a 320×512 clip entirely on-device — no network, no server, no account.

## Licence — please read before use

The weights are released by Qualcomm under **two** licences, and both carry over
to these conversions:

1. **[BSD 3-Clause Clear](https://spdx.org/licenses/BSD-3-Clause-Clear.html)** —
   permits redistribution and derivative works (a conversion is a derivative)
   with notices retained. Note the "Clear" variant grants **no patent licence**.
2. **[Qualcomm Responsible AI License](https://www.qualcomm.com/site/responsible-ai-license)** —
   adds use restrictions. It is not a grant licence; it sits on top of the BSD
   grant and **flows down to you**. Prohibited uses include military use,
   criminal/predictive-policing applications, social scoring, biometric
   categorisation, emotion recognition in workplaces or education, and
   distributing generated content without disclosing that it is machine
   generated. Read the full text before using these files.

Box surfaces these restrictions in a consent screen before download and marks
generated output as AI-generated.

## What's here

Fixed-shape LiteRT graphs. Filenames are deliberately **flat** — Box's downloader
flattens remote paths into a single directory.

### Video pipeline (picture → video)
| file | size | shape |
|---|---|---|
| `dt5_text_encoder.tflite` | 495 MB | `(1,128)` ids + mask → `(1,128,4096)` |
| `context_adapter_dynamic_int8.tflite` | 124 MB | `(1,128,4096)` → `(1,128,1536)` |
| `clip_l_text_encoder.tflite` | 472 MB | `(1,77)` → pooled 768 |
| `clip_g_text_encoder_int8.tflite` | 670 MB | `(1,77)` → pooled 1280 |
| `dit_stage0_10x16_int4.tflite` | 816 MB | pyramid stage 0 |
| `dit_stage1_20x32_int4.tflite` | 817 MB | pyramid stage 1 |
| `dit_stage2_40x64_int4.tflite` | 822 MB | pyramid stage 2 |
| `causal_video_vae_encoder_320x512.tflite` | 419 MB | `(1,3,1,320,512)` → `(1,16,1,40,64)` |
| `causal_video_vae_decoder.tflite` | 38 MB | `(1,16,2,40,64)` → `(1,3,9,320,512)` |

The three DiT graphs are the same 1.5B weights at three input resolutions; the
pipeline calls one per pyramid stage. Each accepts **exactly two** conditioning
latents, which is what limits this set to 9-frame clips.

### First-frame pipeline (text → picture, SSD1B at 640×1024)
| file | size | shape |
|---|---|---|
| `ssd_1b_clip_l.tflite` | 443 MB | `(1,77)` → `(1,77,768)` `hidden_states[-2]` |
| `ssd_1b_clip_g.tflite` | 2.65 GB | `(1,77)` → `(1,1280)` + `(1,77,1280)` |
| `ssd_1b_unet_640x1024.tflite` | 5.07 GB | `(1,4,80,128)` + t + ehs + text_embeds + time_ids |
| `ssd_1b_vae_decoder_640x1024.tflite` | 189 MB | `(1,4,80,128)` → `(1,3,640,1024)` |

### Tokenizer assets
| file | note |
|---|---|
| `clip_vocab.json`, `clip_merges.txt` | CLIP BPE, shared by all four CLIP encoders |
| `t5_spiece.model` | DT5 unigram SentencePiece |

`clip_merges.txt` retains its `#version: 0.2` header line — **skip it when
loading**, or every merge rank shifts by one and tokenization silently corrupts.

## Notes for anyone reusing these

- **CPU only.** The LiteRT GPU delegate supports at most rank-4 tensors, and
  Neodragon's latents are 5D `(b,c,t,h,w)`. On a Tensor G5, `--gpu` places 256 of
  3402 ops on the GPU and then segfaults (`RESHAPE: Tensor dimensions must be
  less than 5`). This is structural, not a configuration problem.
- **Two different CLIP pairs.** `clip_*` are Neodragon's own encoders (for the
  DiT's pooled vector); `ssd_1b_clip_*` are SSD1B's (for SDXL conditioning).
  Their weights differ. The tokenizer vocabularies are identical, so one
  tokenizer serves both.
- **SDXL uses `hidden_states[-2]`**, not `last_hidden_state`, from both CLIP
  encoders. Using the last layer does not crash — it quietly degrades output.
- **The SSD1B UNet is fp32** because int8 quantisation fails in
  `ai_edge_quantizer` 0.8.0 on one layer
  (`CrossAttnDownBlock2D_2/ResnetBlock2D_0/conv2`: *"input operand has more
  dimensions than allowed by the axis remapping"*).
- Measured on a Pixel 10 Pro Fold (Tensor G5), CPU: DiT stages ≈ 1.1 / 3.9 /
  17.3 s per call.

## Credit

All credit for the model and research to **Qualcomm AI Research** — Animesh
Karnewar and the Neodragon authors. These conversions exist only to make that
work runnable on more devices.