TKCAM checkpoints

Pretrained checkpoints for TKCAM: Text and Keyframe to Camera Trajectory Generation (NeurIPS 2026). TKCAM generates a camera trajectory from a text description, optionally conditioned on a first frame or on up to 4 keyframes placed at chosen times.

Paper (arXiv:2610.11105) · Code · Project page · Dataset

mode	conditioning	mask transformer	residual transformer
noframe	text	mtrans_25kfixed_noframe_r4 (epoch 555)	rtrans_25kfixed_noframe_r4 (epoch 40)
firstframe	text + first frame	mtrans_25kfixed_1frame_r4 (epoch 480)	rtrans_25kfixed_1frame_r4 (epoch 600)
xframe	text + up to 4 keyframes	mtrans_25kfixed_xframe_r4 (epoch 600)	rtrans_25kfixed_xframe_r4 (epoch 125)

All modes share the residual VQ-VAE rvq_w128_nq4_cb256_25kfixed_60ep (4 quantizers, 256 codes, window 128; epoch 25). Text is encoded with a frozen pretrained t5-base and images with a frozen pretrained CLIP ViT-B/32 vision encoder (openai/clip-vit-base-patch32).

Usage

git clone https://github.com/linearalgebrayhz/TKCAM.git && cd TKCAM
pip install -r requirements.txt && pip install -e .     # or: conda env create -f environment.yml
hf download yhz-linearalgebra/TKCAM --local-dir checkpoints

# Text only
python scripts/generate.py --mode noframe --duration 6 \
  --text "The camera dollies forward slowly, revealing the living room."
# Text + first frame
python scripts/generate.py --mode firstframe --duration 5 --first_frame room.jpg \
  --text "The camera pans left smoothly across the kitchen."
# Text + keyframes at chosen times (image@seconds, up to 4)
python scripts/generate.py --mode xframe --duration 8 \
  --keyframe hall.jpg@0 --keyframe stairs.jpg@4.5 \
  --text "The camera moves forward through the hallway, then tilts up toward the stairs."

Each sample is saved as sample{i}_9d.npy ((T, 9): [rot6d(6), x, y, z]), sample{i}_c2w.npy ((T, 4, 4)), an MP4 animation and meta.json. Trajectories are camera-to-world, relative to the first frame, with OpenGL camera axes (+X right, +Y up, −Z forward), at 30 fps. Durations are clamped to 3–11 s; the training clips are about 4.0–9.3 s. See the code repository for batch inference over a prompts file and for training.

The T5 and CLIP encoder weights are not stored in these files: they are the unmodified public weights and are downloaded from the Hugging Face Hub on first use.

Files

The folder mirrors the layout the TKCAM code expects under checkpoints/:

realestate10k_rotmat/
  rvq_w128_nq4_cb256_25kfixed_60ep/{opt.txt, meta/mean.npy, meta/std.npy, model/net_best_recon.tar}
  {mtrans,rtrans}_25kfixed_{noframe,1frame,xframe}_r4/{opt.txt, net_best_fid.tar}
MANIFEST.json      # epochs, source checkpoint, parameter counts, sizes, sha256

Each .tar holds the model state_dict and the epoch (no optimizer state). Every transformer file is named net_best_fid.tar, the name the TKCAM scripts load by default: it is the checkpoint selected on validation FID (see Training); the epochs are in the table above and in MANIFEST.json. opt.txt is the training configuration; local paths in it were replaced with placeholders.

Training

• Data: the train split of RealEstate10K-Cap (21,250 clips), the released version of the dataset.
• Recipe: MoMask three-stage pipeline (residual VQ-VAE → masked transformer → residual transformer), 600 epochs per transformer, learning rate 5e-5. Visual conditioning uses the first frame (firstframe) or 0–4 randomly sampled keyframes per clip (xframe).
• Checkpoint selection: the VQ-VAE by validation reconstruction error. For each transformer, the saved checkpoints of the run (best validation FID during training, best accuracy, best loss, and the last epoch) were re-scored by FID on the 2,500-clip validation split with a CLaTr evaluator whose training data excludes the RealEstate10K-Cap validation and test clips; the best one is released as net_best_fid.tar.

Validation split (2,500 clips), CLaTr evaluator; R@k = text-to-trajectory retrieval over the whole split, in %:

mode	FID ↓	R@1	R@3	R@10	matching
noframe	0.042	2.1	5.2	13.5	0.289
firstframe	0.045	2.5	5.6	13.6	0.279
xframe	0.045	2.6	5.7	14.7	0.288

These checkpoints were retrained after the paper submission and are not the models behind the paper's tables. Compared with those models, they are trained on the released dataset version and the VQ-VAE checkpoint is selected by validation reconstruction error.

License

MIT, as the code. TKCAM builds on MoMask (MIT).

Citation

@misc{yang2026tkcam,
  title={TKCAM: Text and Keyframe to Camera Trajectory Generation},
  author={Yang, Haozhe and Dou, Zhiyang and Gu, Zekai and Lin, Cheng and Wang, Wenping and Liu, Yuan and Komura, Taku},
  year={2026},
  eprint={2610.11105},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2610.11105}
}