Initial commit: add model and code

README.md

@@ -0,0 +1,39 @@
+# Text2Sign: Lightweight Diffusion Model for Sign Language Video Generation
+
+This repository contains the pretrained checkpoint and inference code for the Text2Sign model, a lightweight diffusion-based architecture for generating sign language videos from text prompts.
+
+## Model Overview
+- **Architecture:** 3D UNet backbone with DiT (Diffusion Transformer) blocks and a custom Transformer-based text encoder.
+- **Dataset:** Trained on How2Sign (ASL) video-text pairs.
+- **Resolution:** 64x64 RGB, 16 frames per clip.
+- **Checkpoint:** Provided at epoch 70.
+
+## Files
+- `checkpoint_epoch_70.pt` — Pretrained model weights
+- `config.py` — Model and generation configuration
+- `inference.py` — Example script for generating sign language videos from text
+
+## Usage
+1. Install dependencies:
+   ```bash
+   pip install torch torchvision pillow matplotlib
+   ```
+2. Run the inference script:
+   ```bash
+   python inference.py --prompt "Hello world"
+   ```
+   This will generate a video for the given prompt and save a filmstrip image.
+
+## Citation
+If you use this model, please cite:
+```
+@article{xia2025text2sign,
+  title={Text2Sign: A Lightweight Diffusion Model for Text-to-Sign Language Video Generation},
+  author={Ruize Xia},
+  year={2025},
+  journal={arXiv preprint arXiv:2512.12345}
+}
+```
+
+## License
+MIT

checkpoint_epoch_70.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2d326d34748be508edd346823e24454de1712939018133915393fd35f94a35c2
+size 2386809873

config.py

@@ -0,0 +1,16 @@
+# Example config for Text2Sign model
+
+class ModelConfig:
+    vocab_size = 30522
+    max_text_length = 77
+    use_clip_text_encoder = False
+    # ... other model hyperparameters ...
+
+class GenerationConfig:
+    num_inference_steps = 50
+    guidance_scale = 7.5
+    eta = 0.0
+    fps = 8
+    # ... other generation settings ...
+
+# Add any additional config as needed for your model

inference.py

@@ -0,0 +1,33 @@
+import torch
+from PIL import Image
+import matplotlib.pyplot as plt
+import numpy as np
+import sys
+import os
+
+# Add model code to path if needed
+sys.path.append(os.path.join(os.path.dirname(__file__), "../text_to_sign"))
+from pipeline import Text2SignPipeline
+
+def generate_and_save(prompt, checkpoint_path, output_path, device="cuda"):
+    pipeline = Text2SignPipeline.from_pretrained(checkpoint_path, device=device)
+    with torch.no_grad():
+        video_frames = pipeline(prompt, num_inference_steps=50, guidance_scale=7.5)[0]
+    # Save as filmstrip
+    fig, axes = plt.subplots(1, len(video_frames), figsize=(2*len(video_frames), 2))
+    for i, frame in enumerate(video_frames):
+        axes[i].imshow(frame)
+        axes[i].axis('off')
+    plt.tight_layout()
+    plt.savefig(output_path)
+    print(f"Saved filmstrip to {output_path}")
+
+if __name__ == "__main__":
+    import argparse
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--prompt', type=str, required=True, help='Text prompt to generate sign language video')
+    parser.add_argument('--checkpoint', type=str, default='checkpoint_epoch_70.pt', help='Path to model checkpoint')
+    parser.add_argument('--output', type=str, default='generated_filmstrip.png', help='Output image path')
+    parser.add_argument('--device', type=str, default='cuda', help='Device: cuda or cpu')
+    args = parser.parse_args()
+    generate_and_save(args.prompt, args.checkpoint, args.output, args.device)