Upload 6 files

LICENSE.txt

@@ -0,0 +1,180 @@
+ModelGo Attribution-NonCommercial-ResponsibleAI License
+Version 2.0, May 2025
+
+By exercising the rights granted in Section 2.1, You acknowledge and agree that You have 
+read, understood, and agree to be bound by the terms and conditions of this License. If You 
+do not agree to any terms and/or conditions of this License, then the Licensor grants You no 
+rights under this License.
+
+1.	DEFINITIONS
+
+"Complementary Materials" means the source code and scripts used to define, run, load, 
+benchmark and/or evaluate the Model, and prepare data for the purpose of training, pretraining, 
+fine-tuning and/or evaluation of the Model, and any tutorials, operating manuals, user guides 
+and/or documentation that guide users in using, operating, implementing and/or customising 
+the Model.
+
+"Derivative Materials" means all improvements, modifications or derivative works to the 
+Licensed Material or any part thereof, which are created or developed by You (either by 
+Yourself or jointly with other third parties), including any derivative model developed by 
+transferring patterns of weights, parameters, activations and/or Output from the Model, such as 
+through distillation methods or synthetic data generation techniques, in order to replicate, 
+approximate, or otherwise achieve functional behavior that is similar to the Model.
+
+"Distribution" (or "Distribute") means any transmission, publication, public performance, 
+sharing, or other methods of making the Licensed Materials, Derivative Materials and/or Output 
+available to a third party, including providing the Licensed Materials or any part thereof as a 
+hosted service or remotely accessible service, such as API-based access or web access.
+
+"Licensor" means the rights owner that is granting the License.
+
+"Licensed Materials" means the Model and Complementary Materials that are Distributed by 
+the Licensor under this License. The Licensed Materials do not include any datasets used for 
+pretraining, training, adapting, or evaluating the Model, which may or may not be made 
+available under a separate license.
+
+"Model" means the machine-learning constructs and/or assemblies licensed by the Licensor 
+under this License, including any checkpoints, learned weights, parameters (including optimizer 
+states) and the model architecture.
+
+"NonCommercial" means not primarily intended for or directed towards commercial advantage 
+or monetary compensation.
+
+"Output" means any information, data, and/or content, including but not limited to images, text, 
+text effects, audio files, and/or video files, generated through operation of the Model, or through 
+operation of any new model developed by transferring patterns from the Model.
+
+"You" (or "Your") means you, or any other person or entity you are entering into this license on 
+behalf of, provided you have the legal authority to bind such person or entity.
+
+2.	LICENSE RIGHTS
+
+2.1	Grant of Rights
+
+(a)	Subject to the conditions in Section 2.2 of this License, the Licensor hereby grants to 
+You a non-exclusive, non-sublicensable, revocable, royalty-free, worldwide license under 
+the relevant copyrights, patent rights, database rights, and any other intellectual property 
+rights to:
+
+(i)	make, have made, use, reproduce and Distribute the Licensed Materials;
+
+(ii)	use the Licensed Materials to create Derivative Materials; and
+
+(iii)	make, have made, use, reproduce and Distribute Derivative Materials;
+
+for NonCommercial purposes only.
+
+2.2	Conditions
+
+(a)	If You Distribute any of the Licensed Materials or Derivative Materials, You shall:
+
+(i)	provide a copy of this License with the Licensed Material or Derivative Materials;
+
+(ii)	in the case of Distributing Derivative Materials, provide as part of the Distribution, 
+prominent notices stating that You have modified the Licensed Materials and the 
+relevant date of such modification;
+
+(iii)	retain as part of the Distribution any existing copyright notice, attribution notice, 
+and/or any notice identifying the authors of the Licensed Materials and any 
+Derivative Materials.
+
+(b)	If You Distribute a collection of Output as a dataset generated wholly or partially using 
+the Licensed Materials and/or Derivative Materials, You shall provide a prominent notice 
+in the Distribution stating that the dataset is generated using the Licensed Materials 
+and/or Derivative Materials provided under this License.
+
+(c)	You agree not to use the Licensed Materials or any Derivative Materials:
+
+(i)	for any illegal or unlawful purposes, including in any way that violates any 
+applicable national, federal, state, local, or international law or regulation;
+
+(ii)	for the purpose of exploiting, harming, or attempting to exploit or harm minors in 
+any way;
+
+(iii)	for the purpose of generating, transmitting and/or disseminating computer code, 
+files, scripts and programs, including any malware and/or software, that is 
+intended or known to be harmful, destructive, disabling or which assists in or 
+enables theft, alteration, denial of service, unauthorised disclosure or destruction 
+or corruption of data, including viruses, worms, spyware, adware, keyloggers, 
+trojans, ransomware and any other type of threats;
+(iv)	for any fraudulent purpose, such as generating, transmitting and/or disseminating 
+verifiably false information and/or content;
+
+(v)	for the purpose of generating, transmitting and/or disseminating personal data with 
+the purpose of harming others;
+
+(vi)	for the purpose of generating, transmitting and/or disseminating information 
+(including, but not limited to, images, code, posts, articles), and placing the 
+information in any public context (including, but not limited to, bot generating 
+tweets) without expressly and intelligibly disclaiming that the information and/or 
+content is machine generated;
+
+(vii)	for any libelous, defamatory, obscene or threatening purposes;
+
+(viii)	for the purpose of impersonating or attempting to impersonate human beings for 
+the purposes of deception;
+
+(ix)	for the purpose of enabling and/or facilitating fully automated decision-making that 
+adversely impacts an individual’s legal rights or otherwise creates or modifies a 
+binding, enforceable obligation without expressly and intelligibly disclaiming that 
+the creation or modification of the obligation is machine generated;
+
+(x)	for the purpose of enabling and/or facilitating discrimination and/or harm against 
+individuals or groups based on online or offline social behavior or known or 
+predicted personal or personality characteristics;
+
+(xi)	for the purpose of exploiting any of the vulnerabilities of a specific group of 
+persons based on their age, social, physical, or mental characteristics, in order to 
+materially distort the behavior of a person pertaining to that group in a manner that 
+causes or is likely to cause that person or another person physical or 
+psychological harm;
+
+(xii)	for the purpose of enabling and/or facilitating discrimination against individuals or 
+groups based on legally protected characteristics or categories;
+
+(xiii)	for the purpose of providing medical advice or medical results interpretation that is 
+intended to be a substitute for professional medical advice, diagnosis, or treatment; 
+and
+
+(xiv)	for the purpose of enabling and/or facilitating fully automated decision-making in 
+the administration of justice, law enforcement, immigration, or asylum processes.
+
+2.3	Reservation of Rights
+
+You shall not refer to the Licensor or use the Licensor's trademarks, trade names, and service 
+marks, for any publicity, advertising or marketing purposes, without the Licensor's prior written 
+consent.
+
+3.	DISCLAIMER
+
+To the maximum extent permissible under applicable law, the Licensed Materials are provided 
+on an "as is" and "as available" basis without any representation or warranty of any kind 
+(whether express, implied, statutory or otherwise), including of merchantability, satisfactory 
+quality, fitness for a particular purpose, title, accuracy, correctness, absence of error, reliability, 
+timeliness, non-infringement of or compliance with any laws, regulations and/or third-party 
+rights, all of which are expressly disclaimed by the Licensor.
+
+4.	LIMITATION OF LIABILITY
+
+To the maximum extent permissible under applicable law, the Licensor shall not be liable for 
+any claims, damages, or any other liabilities, in any way arising out of or in connection with the 
+Licensed Materials, Derivative Materials, and/or Output, whether on an action or claim in 
+contract, tort (including negligence), breach of statutory duty or otherwise, even if the Licensor 
+has been advised of the possibility of such damages.
+
+5.	TERMINATION
+
+This License shall terminate immediately if You breach any material term and/or condition of 
+this License, or if You initiate any legal action against the Licensor alleging that the Licensed 
+Materials and/or Derivative Materials infringe any patent worldwide. Sections 3, 4 and 6 shall 
+survive the termination of this License.
+
+6.	MODIFICATION OF THIS LICENSE
+
+This License is Copyright © 2024 National University of Singapore. Permission is granted to 
+copy, distribute, or communicate this License without modification. Nothing in this License 
+permits You to modify this License as applied to the Licensed Materials. However, You may 
+modify the text of this License and copy, distribute or communicate Your modified version and 
+apply it to other original works of authorship, provided that You do not use the name "ModelGo" 
+for your version of the license and furnish a readable notice describing Your modifications to 
+this License.

README.md

@@ -0,0 +1,15 @@
+---
+title: UniGaze
+emoji: 🐠
+colorFrom: gray
+colorTo: yellow
+sdk: gradio
+sdk_version: 5.43.1
+app_file: app.py
+pinned: false
+license: other
+short_description: Online Demo of UniGaze
+---
+This is demo of the approach described in the paper ["UniGaze: Towards Universal Gaze Estimation via Large-scale Pre-Training"](https://arxiv.org/abs/2502.02307)
+
+Check out the project page at https://ut-vision.github.io/UniGaze

app.py

@@ -0,0 +1,210 @@
+# app.py
+import os
+import sys
+import time
+from pathlib import Path
+from typing import List, Optional, Tuple
+
+import gradio as gr
+import numpy as np
+from huggingface_hub import hf_hub_download, HfApi
+from unigaze.infer_runtime import UniGazeRuntime  # in-process runtime (no subprocess)
+
+# --------------------------------------------------------------------------------------
+# Defaults
+# --------------------------------------------------------------------------------------
+DEFAULT_HF_REPO = "xucongzhang/UniGaze-models"
+DEFAULT_CKPT_FILE = [
+    "unigaze_h14_joint.pth.tar",
+    "unigaze_l16_joint.pth.tar",
+    "unigaze_b16_joint.pth.tar",
+]
+DEFAULT_REVISION = "main"
+
+DEFAULT_CFGS = [
+    "unigaze/configs/model/mae_h_14_gaze.yaml",
+    "unigaze/configs/model/mae_L_16_gaze.yaml",
+    "unigaze/configs/model/mae_b_16_gaze.yaml",
+]
+
+TITLE = "UniGaze Demo (Video + Image)"
+DESC = """
+Upload a short video or a single image. The app downloads a checkpoint from the Hub,
+runs UniGaze in-process (no subprocess, no permanent writes), and returns results.
+"""
+
+# Ensure imports of local packages work
+sys.path.append(os.path.dirname(__file__))
+
+# --------------------------------------------------------------------------------------
+# Helpers
+# --------------------------------------------------------------------------------------
+def resolve_cfg_abs(cfg_str: str) -> Path:
+    """Return an absolute path to the YAML config."""
+    p = Path(cfg_str)
+    if p.is_absolute():
+        if p.exists():
+            return p
+        raise FileNotFoundError(f"Config not found: {p}")
+
+    p2 = (Path.cwd() / p).resolve()
+    if p2.exists():
+        return p2
+
+    if str(p).startswith("configs/"):
+        p3 = (Path.cwd() / "unigaze" / p).resolve()
+        if p3.exists():
+            return p3
+
+    raise FileNotFoundError(f"Config not found. Tried: {p2}")
+
+def list_weight_files(repo_id: str, revision: str = "main") -> List[str]:
+    try:
+        api = HfApi()
+        files = api.list_repo_files(repo_id=repo_id, repo_type="model", revision=revision)
+        return [f for f in files if f.lower().endswith((".pth", ".pt", ".safetensors", ".tar", ".pth.tar"))]
+    except Exception:
+        return []
+
+def get_ckpt_path(repo_id: str, filename: str, revision: str = "main") -> str:
+    files = list_weight_files(repo_id, revision)
+    if files and filename not in files:
+        raise FileNotFoundError(
+            f"File '{filename}' not found in model repo '{repo_id}' at rev '{revision}'. "
+            f"Available weights: {files}"
+        )
+    return hf_hub_download(
+        repo_id=repo_id,
+        filename=filename,
+        revision=revision,
+        repo_type="model",
+    )
+
+# Cache the runtime so we load model/FA only once
+from functools import lru_cache
+@lru_cache(maxsize=3)
+def get_runtime(cfg_abs_str: str, ckpt_path: str, device: str = "cpu") -> UniGazeRuntime:
+    return UniGazeRuntime(cfg_abs_str, ckpt_path, device=device)
+
+# --------------------------------------------------------------------------------------
+# Runners (in-process)
+# --------------------------------------------------------------------------------------
+def run_unigaze_on_video(
+    video_path: str,
+    hf_repo: str,
+    ckpt_filename: str,
+    cfg_path_user: str,
+    extra_args: str = "",
+) -> Tuple[Optional[np.ndarray], Optional[str], Optional[str], str]:
+    logs: List[str] = []
+    t0 = time.time()
+
+    try:
+        ckpt_path = get_ckpt_path(hf_repo, ckpt_filename, revision=DEFAULT_REVISION)
+        logs.append(f"[hub] downloaded: {ckpt_path}")
+    except Exception as e:
+        return None, None, None, f"[hub] ERROR: {e}"
+
+    try:
+        cfg_abs = resolve_cfg_abs(cfg_path_user)
+    except Exception as e:
+        return None, None, None, f"[cfg] {e}"
+
+    rt = get_runtime(str(cfg_abs), ckpt_path, device="cpu")
+    mp4_path, last_rgb, run_sec = rt.predict_video(video_path)
+    logs.append(f"[time] total runtime: {run_sec:.2f} seconds")
+
+    return (last_rgb if last_rgb is not None else None), mp4_path, None, "\n".join(logs)
+
+def run_unigaze_on_image(
+    image_array: np.ndarray,
+    hf_repo: str,
+    ckpt_filename: str,
+    cfg_path_user: str,
+    extra_args: str = "",
+) -> Tuple[Optional[np.ndarray], str]:
+    logs: List[str] = []
+    t0 = time.time()
+
+    try:
+        ckpt_path = get_ckpt_path(hf_repo, ckpt_filename, revision=DEFAULT_REVISION)
+        logs.append(f"[hub] downloaded: {ckpt_path}")
+    except Exception as e:
+        return None, f"[hub] ERROR: {e}"
+
+    try:
+        cfg_abs = resolve_cfg_abs(cfg_path_user)
+    except Exception as e:
+        return None, f"[cfg] {e}"
+
+    rt = get_runtime(str(cfg_abs), ckpt_path, device="cpu")
+    out_rgb = rt.predict_image(image_array)
+    logs.append(f"[time] total runtime: {time.time() - t0:.2f} seconds")
+
+    return out_rgb, "\n".join(logs)
+
+# --------------------------------------------------------------------------------------
+# UI
+# --------------------------------------------------------------------------------------
+with gr.Blocks(title=TITLE) as demo:
+    gr.Markdown(f"# {TITLE}\n{DESC}")
+
+    with gr.Row():
+        ckpt_file = gr.Dropdown(choices=DEFAULT_CKPT_FILE, value=DEFAULT_CKPT_FILE[0], label="Checkpoint filename")
+        cfg_choice = gr.Dropdown(choices=DEFAULT_CFGS, value=DEFAULT_CFGS[0], label="Model config")
+
+    # IMAGE TAB
+    with gr.Tab("Image"):
+        in_img = gr.Image(type="numpy", label="Input image")
+        run_img = gr.Button("Run on Image", variant="primary")
+        out_img = gr.Image(label="Output image")
+        out_logs = gr.Textbox(label="Logs", interactive=False, lines=18)
+
+        def ui_predict_image(image, ckpt, cfg_use):
+            return run_unigaze_on_image(
+                image_array=image,
+                hf_repo=DEFAULT_HF_REPO,
+                ckpt_filename=ckpt,
+                cfg_path_user=cfg_use,
+            )
+
+        run_img.click(
+            fn=ui_predict_image,
+            inputs=[in_img, ckpt_file, cfg_choice],
+            outputs=[out_img, out_logs],
+        )
+
+        # Example image
+        gr.Examples(
+            examples=[["examples/The_Night_Watch_Frans_Banninck_Cocq.png", DEFAULT_CKPT_FILE[0], DEFAULT_CFGS[0]]],
+            inputs=[in_img, ckpt_file, cfg_choice],
+            outputs=[out_img, out_logs],
+            fn=ui_predict_image,
+            cache_examples=False,
+        )
+
+    # VIDEO TAB
+    with gr.Tab("Video"):
+        in_vid = gr.Video(label="Input video", sources=["upload"])
+        run_vid = gr.Button("Run on Video", variant="primary")
+        out_img_v = gr.Image(label="Annotated image (last frame)")
+        out_vid_v = gr.Video(label="Output video")
+        out_zip_v = gr.File(label="All artifacts as ZIP")  # always None now
+        out_logs_v = gr.Textbox(label="Logs", interactive=False, lines=18)
+
+        def ui_predict_video(video, ckpt, cfg_use):
+            return run_unigaze_on_video(
+                video_path=video,
+                hf_repo=DEFAULT_HF_REPO,
+                ckpt_filename=ckpt,
+                cfg_path_user=cfg_use,
+            )
+
+        run_vid.click(
+            fn=ui_predict_video,
+            inputs=[in_vid, ckpt_file, cfg_choice],
+            outputs=[out_img_v, out_vid_v, out_zip_v, out_logs_v],
+        )
+
+if __name__ == "__main__":
+    demo.launch()

gitattributes

@@ -0,0 +1,38 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+example/The_Night_Watch_Frans_Banninck_Cocq.png filter=lfs diff=lfs merge=lfs -text
+examples/The_Night_Watch_Frans_Banninck_Cocq.png filter=lfs diff=lfs merge=lfs -text
+examples/De_Nachtwacht.png filter=lfs diff=lfs merge=lfs -text

gitignore

@@ -0,0 +1,78 @@
+.DS_Store
+*.h5
+*.json
+.vscode
+*.pth
+result/
+*.npy
+*.pdf
+cache/
+*.txt
+!requirements.txt
+*.drawio
+
+*.ipynb
+!load_gaze_model.ipynb
+.ipynb_checkpoints
+*.npy
+*.dat
+*.bz2
+
+figures/
+*.csv
+prototype/
+__pycache__/
+results/
+*.xlsx
+run_on_server/
+
+memo/
+ckpt/
+
+
+
+sig.sh
+*.pkl
+
+*.pt
+*.pth
+
+*.pyc
+
+# src/clip
+# src/taming-transformers
+
+*.ckpt
+*.pth.tar
+checkpoints/*
+
+!checkpoints/pre_trained_ckpt.md
+!checkpoints/*.md
+
+logs/*
+
+*.jpg
+*.png
+*.JPG
+
+*.PNG
+*.gif
+__pycache__/
+
+*.log
+*.sh
+
+configs/data_path.yaml
+
+!assets/*.jpg
+!configs/trainer/figs/*.jpg
+!configs/trainer/figs/*.png
+!assets/figs/*/*
+!data/face_model.txt
+batch_log*/
+
+scripts/
+*out
+*_nodeinfo
+
+outputs/

requirements.txt

@@ -0,0 +1,52 @@
+gradio>=4.0.0
+huggingface_hub>=0.23.0
+
+rich
+wandb
+glob2
+h5py
+omegaconf
+matplotlib
+imageio
+imageio-ffmpeg
+seaborn
+torch-summary
+torchfile
+torchmetrics
+
+transformers
+accelerate
+albumentations
+chumpy
+cmake
+coloredlogs
+colorlog
+cython
+diffusers
+easydict
+einops
+lpips
+
+opencv-python
+optuna
+pandas
+protobuf
+pyopengl
+pytorch-fid
+pytorch-lightning
+pytorch-metric-learning
+scikit-image
+scikit-learn
+scipy
+
+tensorboard
+tensorboardx
+
+face-alignment
+
+numpy==1.24.4
+hydra-core
+torch==2.0.1
+timm>=0.6.13
+torchvision==0.15.2
+