Delete WiLoR

WiLoR/README.md

@@ -1,93 +0,0 @@
-<div align="center">
-
-# WiLoR: End-to-end 3D hand localization and reconstruction in-the-wild
-
-[Rolandos Alexandros Potamias](https://rolpotamias.github.io)<sup>1</sup> &emsp; [Jinglei Zhang]()<sup>2</sup> &emsp; [Jiankang Deng](https://jiankangdeng.github.io/)<sup>1</sup> &emsp; [Stefanos Zafeiriou](https://www.imperial.ac.uk/people/s.zafeiriou)<sup>1</sup>  
-
-<sup>1</sup>Imperial College London, UK <br>
-<sup>2</sup>Shanghai Jiao Tong University, China
-
-<font color="blue"><strong>CVPR 2025</strong></font> 
-
-<a href='https://rolpotamias.github.io/WiLoR/'><img src='https://img.shields.io/badge/Project-Page-blue'></a>
-<a href='https://arxiv.org/abs/2409.12259'><img src='https://img.shields.io/badge/Paper-arXiv-red'></a>
-<a href='https://huggingface.co/spaces/rolpotamias/WiLoR'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Demo-green'></a>
-<a href='https://colab.research.google.com/drive/1bNnYFECmJbbvCNZAKtQcxJGxf0DZppsB?usp=sharing'><img src='https://colab.research.google.com/assets/colab-badge.svg'></a>
-</div>
-
-<div align="center">
-
-[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/wilor-end-to-end-3d-hand-localization-and/3d-hand-pose-estimation-on-freihand)](https://paperswithcode.com/sota/3d-hand-pose-estimation-on-freihand?p=wilor-end-to-end-3d-hand-localization-and)
-[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/wilor-end-to-end-3d-hand-localization-and/3d-hand-pose-estimation-on-ho-3d)](https://paperswithcode.com/sota/3d-hand-pose-estimation-on-ho-3d?p=wilor-end-to-end-3d-hand-localization-and)
-
-</div>
-
-This is the official implementation of **[WiLoR](https://rolpotamias.github.io/WiLoR/)**, an state-of-the-art hand localization and reconstruction model:
-
-![teaser](assets/teaser.png)
-
-## Installation
-### [Update] Quick Installation
-Thanks to [@warmshao](https://github.com/warmshao) WiLoR can now be installed using a single pip command:  
-```
-pip install git+https://github.com/warmshao/WiLoR-mini
-```
-Please head to [WiLoR-mini](https://github.com/warmshao/WiLoR-mini) for additional details. 
-
-**Note:** the above code is a simplified version of WiLoR and can be used for demo only. 
-If you wish to use WiLoR for other tasks it is suggested to follow the original installation instructued bellow: 
-### Original Installation
-```
-git clone --recursive https://github.com/rolpotamias/WiLoR.git
-cd WiLoR
-```
-
-The code has been tested with PyTorch 2.0.0 and CUDA 11.7. It is suggested to use an anaconda environment to install the the required dependencies:
-```bash
-conda create --name wilor python=3.10
-conda activate wilor
-
-pip install torch torchvision --index-url https://download.pytorch.org/whl/cu117
-# Install requirements
-pip install -r requirements.txt
-```
-Download the pretrained models using: 
-```bash
-wget https://huggingface.co/spaces/rolpotamias/WiLoR/resolve/main/pretrained_models/detector.pt -P ./pretrained_models/
-wget https://huggingface.co/spaces/rolpotamias/WiLoR/resolve/main/pretrained_models/wilor_final.ckpt -P ./pretrained_models/
-```
-It is also required to download MANO model from [MANO website](https://mano.is.tue.mpg.de). 
-Create an account by clicking Sign Up and download the models (mano_v*_*.zip). Unzip and place the right hand model `MANO_RIGHT.pkl` under the `mano_data/` folder. 
-Note that MANO model falls under the [MANO license](https://mano.is.tue.mpg.de/license.html).
-## Demo
-```bash
-python demo.py --img_folder demo_img --out_folder demo_out --save_mesh 
-```
-## Start a local gradio demo
-You can start a local demo for inference by running:
-```bash
-python gradio_demo.py
-```
-## WHIM Dataset
-To download WHIM dataset please follow the instructions [here](./whim/Dataset_instructions.md)
-
-## Acknowledgements
-Parts of the code are taken or adapted from the following repos:
-- [HaMeR](https://github.com/geopavlakos/hamer/)
-- [Ultralytics](https://github.com/ultralytics/ultralytics)
-
-## License 
-WiLoR models fall under the [CC-BY-NC--ND License](./license.txt). This repository depends also on [Ultralytics library](https://github.com/ultralytics/ultralytics) and [MANO Model](https://mano.is.tue.mpg.de/license.html), which are fall under their own licenses. By using this repository, you must also comply with the terms of these external licenses.
-## Citing
-If you find WiLoR useful for your research, please consider citing our paper:
-
-```bibtex
-@misc{potamias2024wilor,
-    title={WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wild},
-    author={Rolandos Alexandros Potamias and Jinglei Zhang and Jiankang Deng and Stefanos Zafeiriou},
-    year={2024},
-    eprint={2409.12259},
-    archivePrefix={arXiv},
-    primaryClass={cs.CV}
-}
-```

WiLoR/demo.py

@@ -1,142 +0,0 @@
-from pathlib import Path
-import torch
-import argparse
-import os
-import cv2
-import numpy as np
-import json
-from typing import Dict, Optional
-
-from wilor.models import WiLoR, load_wilor
-from wilor.utils import recursive_to
-from wilor.datasets.vitdet_dataset import ViTDetDataset, DEFAULT_MEAN, DEFAULT_STD
-from wilor.utils.renderer import Renderer, cam_crop_to_full
-from ultralytics import YOLO 
-LIGHT_PURPLE=(0.25098039,  0.274117647,  0.65882353)
-
-def main():
-    parser = argparse.ArgumentParser(description='WiLoR demo code')
-    parser.add_argument('--img_folder', type=str, default='images', help='Folder with input images')
-    parser.add_argument('--out_folder', type=str, default='out_demo', help='Output folder to save rendered results')
-    parser.add_argument('--save_mesh', dest='save_mesh', action='store_true', default=False, help='If set, save meshes to disk also')
-    parser.add_argument('--rescale_factor', type=float, default=2.0, help='Factor for padding the bbox')
-    parser.add_argument('--file_type', nargs='+', default=['*.jpg', '*.png', '*.jpeg'], help='List of file extensions to consider')
-
-    args = parser.parse_args()
-
-    # Download and load checkpoints
-    model, model_cfg = load_wilor(checkpoint_path = './pretrained_models/wilor_final.ckpt' , cfg_path= './pretrained_models/model_config.yaml')
-    detector = YOLO('./pretrained_models/detector.pt')
-    # Setup the renderer
-    renderer = Renderer(model_cfg, faces=model.mano.faces)
-    renderer_side = Renderer(model_cfg, faces=model.mano.faces)
-    
-    device   = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')
-    model    = model.to(device)
-    detector = detector.to(device)
-    model.eval()
-
-    # Make output directory if it does not exist
-    os.makedirs(args.out_folder, exist_ok=True)
-
-    # Get all demo images ends with .jpg or .png
-    img_paths = [img for end in args.file_type for img in Path(args.img_folder).glob(end)]
-    # Iterate over all images in folder
-    for img_path in img_paths:
-        img_cv2 = cv2.imread(str(img_path))
-        detections = detector(img_cv2, conf = 0.3, verbose=False)[0]
-        bboxes    = []
-        is_right  = []
-        for det in detections: 
-            Bbox = det.boxes.data.cpu().detach().squeeze().numpy()
-            is_right.append(det.boxes.cls.cpu().detach().squeeze().item())
-            bboxes.append(Bbox[:4].tolist())
-        
-        if len(bboxes) == 0:
-            continue
-        boxes = np.stack(bboxes)
-        right = np.stack(is_right)
-        dataset = ViTDetDataset(model_cfg, img_cv2, boxes, right, rescale_factor=args.rescale_factor)
-        dataloader = torch.utils.data.DataLoader(dataset, batch_size=16, shuffle=False, num_workers=0)
-
-        all_verts = []
-        all_cam_t = []
-        all_right = []
-        all_joints= []
-        all_kpts  = []
-        
-        for batch in dataloader: 
-            batch = recursive_to(batch, device)
-    
-            with torch.no_grad():
-                out = model(batch) 
-                
-            multiplier    = (2*batch['right']-1)
-            pred_cam      = out['pred_cam']
-            pred_cam[:,1] = multiplier*pred_cam[:,1]
-            box_center    = batch["box_center"].float()
-            box_size      = batch["box_size"].float()
-            img_size      = batch["img_size"].float()
-            scaled_focal_length = model_cfg.EXTRA.FOCAL_LENGTH / model_cfg.MODEL.IMAGE_SIZE * img_size.max()
-            pred_cam_t_full     = cam_crop_to_full(pred_cam, box_center, box_size, img_size, scaled_focal_length).detach().cpu().numpy()
-
-            
-            # Render the result
-            batch_size = batch['img'].shape[0]
-            for n in range(batch_size):
-                # Get filename from path img_path
-                img_fn, _ = os.path.splitext(os.path.basename(img_path))
-                
-                verts  = out['pred_vertices'][n].detach().cpu().numpy()
-                joints = out['pred_keypoints_3d'][n].detach().cpu().numpy()
-                
-                is_right    = batch['right'][n].cpu().numpy()
-                verts[:,0]  = (2*is_right-1)*verts[:,0]
-                joints[:,0] = (2*is_right-1)*joints[:,0]
-                cam_t = pred_cam_t_full[n]
-                kpts_2d = project_full_img(verts, cam_t, scaled_focal_length, img_size[n])
-                
-                all_verts.append(verts)
-                all_cam_t.append(cam_t)
-                all_right.append(is_right)
-                all_joints.append(joints)
-                all_kpts.append(kpts_2d)
-                
-                
-                # Save all meshes to disk
-                if args.save_mesh:
-                    camera_translation = cam_t.copy()
-                    tmesh = renderer.vertices_to_trimesh(verts, camera_translation, LIGHT_PURPLE, is_right=is_right)
-                    tmesh.export(os.path.join(args.out_folder, f'{img_fn}_{n}.obj'))
-
-        # Render front view
-        if len(all_verts) > 0:
-            misc_args = dict(
-                mesh_base_color=LIGHT_PURPLE,
-                scene_bg_color=(1, 1, 1),
-                focal_length=scaled_focal_length,
-            )
-            cam_view = renderer.render_rgba_multiple(all_verts, cam_t=all_cam_t, render_res=img_size[n], is_right=all_right, **misc_args)
-
-            # Overlay image
-            input_img = img_cv2.astype(np.float32)[:,:,::-1]/255.0
-            input_img = np.concatenate([input_img, np.ones_like(input_img[:,:,:1])], axis=2) # Add alpha channel
-            input_img_overlay = input_img[:,:,:3] * (1-cam_view[:,:,3:]) + cam_view[:,:,:3] * cam_view[:,:,3:]
-
-            cv2.imwrite(os.path.join(args.out_folder, f'{img_fn}.jpg'), 255*input_img_overlay[:, :, ::-1])
-
-def project_full_img(points, cam_trans, focal_length, img_res): 
-    camera_center = [img_res[0] / 2., img_res[1] / 2.]
-    K = torch.eye(3) 
-    K[0,0] = focal_length
-    K[1,1] = focal_length
-    K[0,2] = camera_center[0]
-    K[1,2] = camera_center[1]
-    points = points + cam_trans
-    points = points / points[..., -1:] 
-    
-    V_2d = (K @ points.T).T 
-    return V_2d[..., :-1]
-
-if __name__ == '__main__':
-    main()

WiLoR/download_videos.py

@@ -1,58 +0,0 @@
-import os
-import json
-import numpy as np
-import argparse
-from pytubefix import YouTube
-
-parser = argparse.ArgumentParser()
-
-parser.add_argument("--root", type=str, help="Directory of WiLoR")
-parser.add_argument("--mode", type=str, choices=['train', 'test'], default= 'train', help="Train/Test set")
-
-args = parser.parse_args()
-
-with open(os.path.join(args.root, f'./whim/{args.mode}_video_ids.json')) as f:
-    video_dict = json.load(f)
-
-Video_IDs = video_dict.keys()
-failed_IDs = []
-os.makedirs(os.path.join(args.root, 'Videos'), exist_ok=True)  
-
-for Video_ID in Video_IDs:
-    res = video_dict[Video_ID]['res'][0]
-    try:
-        YouTube('https://youtu.be/'+Video_ID).streams.filter(only_video=True, 
-                                                             file_extension='mp4', 
-                                                             res =f'{res}p'
-                                                             ).order_by('resolution').desc().first().download(
-                                                             output_path=os.path.join(args.root, 'Videos') , 
-                                                             filename = Video_ID +'.mp4')
-    except:
-        print(f'Failed {Video_ID}')
-        failed_IDs.append(Video_ID)
-        continue
-        
-        
-    cap = cv2.VideoCapture(os.path.join(args.root, 'Videos', Video_ID + '.mp4'))
-    if (cap.isOpened()== False): 
-        print(f"Error opening video stream {os.path.join(args.root, 'Videos', Video_ID + '.mp4')}")
-
-    VIDEO_LEN = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
-    length = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
-    fps    = cap.get(cv2.CAP_PROP_FPS)
-    
-    fps_org = video_dict[Video_ID]['fps']
-    fps_rate = round(fps / fps_org)
-    
-    all_frames   = os.listdir(os.path.join(args.root, 'WHIM', args.mode, 'anno', Video_ID))
-    
-    for frame in all_frames: 
-        frame_gt = int(frame[:-4])
-        frame_idx = (frame_gt * fps_rate)
-
-        cap.set(cv2.CAP_PROP_POS_FRAMES, frame_idx)
-        ret, img_cv2 = cap.read()
-    
-        cv2.imwrite(os.path.join(args.root, 'WHIM', args.mode, 'anno', Video_ID, frame +'.jpg' ), img_cv2.astype(np.float32))
-                              
-np.save(os.path.join(args.root, 'failed_videos.npy'), failed_IDs)  

WiLoR/gradio_demo.py

@@ -1,192 +0,0 @@
-import os 
-import sys 
-os.environ["PYOPENGL_PLATFORM"] = "egl"
-os.environ["MESA_GL_VERSION_OVERRIDE"] = "4.1"
-# os.system('pip install /home/user/app/pyrender')
-# sys.path.append('/home/user/app/pyrender')
-
-import gradio as gr
-#import spaces
-import cv2 
-import numpy as np 
-import torch 
-from ultralytics import YOLO
-from pathlib import Path
-import argparse
-import json
-from typing import Dict, Optional
-
-from wilor.models import WiLoR, load_wilor
-from wilor.utils import recursive_to
-from wilor.datasets.vitdet_dataset import ViTDetDataset, DEFAULT_MEAN, DEFAULT_STD
-from wilor.utils.renderer import Renderer, cam_crop_to_full
-device = torch.device('cpu') if torch.cuda.is_available() else torch.device('cuda')
-
-LIGHT_PURPLE=(0.25098039,  0.274117647,  0.65882353)
-
-model, model_cfg = load_wilor(checkpoint_path = './pretrained_models/wilor_final.ckpt' , cfg_path= './pretrained_models/model_config.yaml')
-# Setup the renderer
-renderer = Renderer(model_cfg, faces=model.mano.faces)
-model = model.to(device)
-model.eval()
-
-detector = YOLO(f'./pretrained_models/detector.pt').to(device)
-
-def render_reconstruction(image, conf, IoU_threshold=0.3): 
-    input_img, num_dets, reconstructions = run_wilow_model(image, conf, IoU_threshold=0.5)
-    if num_dets> 0: 
-    # Render front view
-    
-        misc_args = dict(
-            mesh_base_color=LIGHT_PURPLE,
-            scene_bg_color=(1, 1, 1),
-            focal_length=reconstructions['focal'],
-        )
-
-        cam_view = renderer.render_rgba_multiple(reconstructions['verts'], 
-                                                 cam_t=reconstructions['cam_t'], 
-                                                 render_res=reconstructions['img_size'], 
-                                                 is_right=reconstructions['right'], **misc_args)
-
-        # Overlay image
-        
-        input_img = np.concatenate([input_img, np.ones_like(input_img[:,:,:1])], axis=2) # Add alpha channel
-        input_img_overlay = input_img[:,:,:3] * (1-cam_view[:,:,3:]) + cam_view[:,:,:3] * cam_view[:,:,3:]
-
-        return input_img_overlay, f'{num_dets} hands detected'  
-    else: 
-        return input_img, f'{num_dets} hands detected' 
-
-#@spaces.GPU()
-def run_wilow_model(image, conf, IoU_threshold=0.5):
-    img_cv2 = image[...,::-1]
-    img_vis = image.copy()
-    
-    detections = detector(img_cv2, conf=conf, verbose=False, iou=IoU_threshold)[0]
-    
-    bboxes    = []
-    is_right  = []
-    for det in detections: 
-        Bbox = det.boxes.data.cpu().detach().squeeze().numpy()
-        Conf = det.boxes.conf.data.cpu().detach()[0].numpy().reshape(-1).astype(np.float16)
-        Side = det.boxes.cls.data.cpu().detach()
-        #Bbox[:2] -= np.int32(0.1 * Bbox[:2])
-        #Bbox[2:] += np.int32(0.1 * Bbox[ 2:])
-        is_right.append(det.boxes.cls.cpu().detach().squeeze().item())
-        bboxes.append(Bbox[:4].tolist())
-        
-        color = (255*0.208, 255*0.647 ,255*0.603 ) if Side==0. else (255*1, 255*0.78039, 255*0.2353)
-        label = f'L - {Conf[0]:.3f}' if Side==0 else f'R - {Conf[0]:.3f}'
-
-        cv2.rectangle(img_vis, (int(Bbox[0]), int(Bbox[1])), (int(Bbox[2]), int(Bbox[3])), color , 3) 
-        (w, h), _ = cv2.getTextSize(label, cv2.FONT_HERSHEY_SIMPLEX, 0.6, 1)
-        cv2.rectangle(img_vis, (int(Bbox[0]), int(Bbox[1]) - 20), (int(Bbox[0]) + w, int(Bbox[1])), color, -1)
-        cv2.putText(img_vis, label, (int(Bbox[0]), int(Bbox[1]) - 5), cv2.FONT_HERSHEY_SIMPLEX, 0.6, (0,0,0), 2)
-          
-    if len(bboxes) != 0: 
-        boxes = np.stack(bboxes)
-        right = np.stack(is_right)
-        dataset = ViTDetDataset(model_cfg, img_cv2, boxes, right, rescale_factor=2.0 )
-        dataloader = torch.utils.data.DataLoader(dataset, batch_size=32, shuffle=False, num_workers=0)
-    
-        all_verts = []
-        all_cam_t = []
-        all_right = []
-        all_joints= []
-    
-        for batch in dataloader: 
-            batch = recursive_to(batch, device)
-    
-            with torch.no_grad():
-                out = model(batch) 
-                
-            multiplier    = (2*batch['right']-1)
-            pred_cam      = out['pred_cam']
-            pred_cam[:,1] = multiplier*pred_cam[:,1]
-            box_center    = batch["box_center"].float()
-            box_size      = batch["box_size"].float()
-            img_size      = batch["img_size"].float()
-            scaled_focal_length = model_cfg.EXTRA.FOCAL_LENGTH / model_cfg.MODEL.IMAGE_SIZE * img_size.max()
-            pred_cam_t_full     = cam_crop_to_full(pred_cam, box_center, box_size, img_size, scaled_focal_length).detach().cpu().numpy()
-
-            
-            batch_size = batch['img'].shape[0]
-            for n in range(batch_size):
-                
-                verts  = out['pred_vertices'][n].detach().cpu().numpy()
-                joints = out['pred_keypoints_3d'][n].detach().cpu().numpy()
-                
-                is_right = batch['right'][n].cpu().numpy()
-                verts[:,0] = (2*is_right-1)*verts[:,0]
-                joints[:,0] = (2*is_right-1)*joints[:,0]
-                
-                cam_t = pred_cam_t_full[n]
-                
-                all_verts.append(verts)
-                all_cam_t.append(cam_t)
-                all_right.append(is_right)
-                all_joints.append(joints)
-
-        reconstructions = {'verts': all_verts, 'cam_t': all_cam_t, 'right': all_right, 'img_size': img_size[n], 'focal': scaled_focal_length}
-        return img_vis.astype(np.float32)/255.0, len(detections), reconstructions
-    else: 
-        return img_vis.astype(np.float32)/255.0, len(detections), None       
-
-
-
-header = ('''
-<div class="embed_hidden" style="text-align: center;">
-    <h1> <b>WiLoR</b>: End-to-end 3D hand localization and reconstruction in-the-wild</h1>
-    <h3>
-        <a href="https://rolpotamias.github.io" target="_blank" rel="noopener noreferrer">Rolandos Alexandros Potamias</a><sup>1</sup>,
-        <a href="" target="_blank" rel="noopener noreferrer">Jinglei Zhang</a><sup>2</sup>,
-        <br>
-        <a href="https://jiankangdeng.github.io/" target="_blank" rel="noopener noreferrer">Jiankang Deng</a><sup>1</sup>,
-        <a href="https://wp.doc.ic.ac.uk/szafeiri/" target="_blank" rel="noopener noreferrer">Stefanos Zafeiriou</a><sup>1</sup>
-    </h3>
-    <h3>
-        <sup>1</sup>Imperial College London;
-        <sup>2</sup>Shanghai Jiao Tong University
-    </h3>
-</div>
-<div style="display:flex; gap: 0.3rem; justify-content: center; align-items: center;" align="center">
-<a href=''><img src='https://img.shields.io/badge/Arxiv-......-A42C25?style=flat&logo=arXiv&logoColor=A42C25'></a> 
-<a href='https://rolpotamias.github.io/pdfs/WiLoR.pdf'><img src='https://img.shields.io/badge/Paper-PDF-yellow?style=flat&logo=arXiv&logoColor=yellow'></a> 
-<a href='https://rolpotamias.github.io/WiLoR/'><img src='https://img.shields.io/badge/Project-Page-%23df5b46?style=flat&logo=Google%20chrome&logoColor=%23df5b46'></a> 
-<a href='https://github.com/rolpotamias/WiLoR'><img src='https://img.shields.io/badge/GitHub-Code-black?style=flat&logo=github&logoColor=white'></a> 
-''')
-
-
-with gr.Blocks(title="WiLoR: End-to-end 3D hand localization and reconstruction in-the-wild", css=".gradio-container") as demo:
-
-    gr.Markdown(header)
-
-    with gr.Row():
-        with gr.Column():
-            input_image = gr.Image(label="Input image", type="numpy")
-            threshold = gr.Slider(value=0.3, minimum=0.05, maximum=0.95, step=0.05, label='Detection Confidence Threshold')
-            #nms = gr.Slider(value=0.5, minimum=0.05, maximum=0.95, step=0.05, label='IoU NMS Threshold')
-            submit = gr.Button("Submit", variant="primary")
-        
-        
-        with gr.Column():
-            reconstruction = gr.Image(label="Reconstructions", type="numpy")
-            hands_detected = gr.Textbox(label="Hands Detected")
-    
-        submit.click(fn=render_reconstruction, inputs=[input_image, threshold], outputs=[reconstruction, hands_detected])
-
-    with gr.Row():
-        example_images = gr.Examples([
-
-            ['./demo_img/test1.jpg'], 
-            ['./demo_img/test2.png'], 
-            ['./demo_img/test3.jpg'], 
-            ['./demo_img/test4.jpg'],
-            ['./demo_img/test5.jpeg'],
-            ['./demo_img/test6.jpg'], 
-            ['./demo_img/test7.jpg'],
-            ['./demo_img/test8.jpg'], 
-            ], 
-            inputs=input_image)
-
-demo.launch()

WiLoR/license.txt

@@ -1,402 +0,0 @@
-Attribution-NonCommercial-NoDerivatives 4.0 International
-
-=======================================================================
-
-Creative Commons Corporation ("Creative Commons") is not a law firm and
-does not provide legal services or legal advice. Distribution of
-Creative Commons public licenses does not create a lawyer-client or
-other relationship. Creative Commons makes its licenses and related
-information available on an "as-is" basis. Creative Commons gives no
-warranties regarding its licenses, any material licensed under their
-terms and conditions, or any related information. Creative Commons
-disclaims all liability for damages resulting from their use to the
-fullest extent possible.
-
-Using Creative Commons Public Licenses
-
-Creative Commons public licenses provide a standard set of terms and
-conditions that creators and other rights holders may use to share
-original works of authorship and other material subject to copyright
-and certain other rights specified in the public license below. The
-following considerations are for informational purposes only, are not
-exhaustive, and do not form part of our licenses.
-
-     Considerations for licensors: Our public licenses are
-     intended for use by those authorized to give the public
-     permission to use material in ways otherwise restricted by
-     copyright and certain other rights. Our licenses are
-     irrevocable. Licensors should read and understand the terms
-     and conditions of the license they choose before applying it.
-     Licensors should also secure all rights necessary before
-     applying our licenses so that the public can reuse the
-     material as expected. Licensors should clearly mark any
-     material not subject to the license. This includes other CC-
-     licensed material, or material used under an exception or
-     limitation to copyright. More considerations for licensors:
-    wiki.creativecommons.org/Considerations_for_licensors
-
-     Considerations for the public: By using one of our public
-     licenses, a licensor grants the public permission to use the
-     licensed material under specified terms and conditions. If
-     the licensor's permission is not necessary for any reason--for
-     example, because of any applicable exception or limitation to
-     copyright--then that use is not regulated by the license. Our
-     licenses grant only permissions under copyright and certain
-     other rights that a licensor has authority to grant. Use of
-     the licensed material may still be restricted for other
-     reasons, including because others have copyright or other
-     rights in the material. A licensor may make special requests,
-     such as asking that all changes be marked or described.
-     Although not required by our licenses, you are encouraged to
-     respect those requests where reasonable. More considerations
-     for the public:
-    wiki.creativecommons.org/Considerations_for_licensees
-
-=======================================================================
-
-Creative Commons Attribution-NonCommercial-NoDerivatives 4.0
-International Public License
-
-By exercising the Licensed Rights (defined below), You accept and agree
-to be bound by the terms and conditions of this Creative Commons
-Attribution-NonCommercial-NoDerivatives 4.0 International Public
-License ("Public License"). To the extent this Public License may be
-interpreted as a contract, You are granted the Licensed Rights in
-consideration of Your acceptance of these terms and conditions, and the
-Licensor grants You such rights in consideration of benefits the
-Licensor receives from making the Licensed Material available under
-these terms and conditions.
-
-
-Section 1 -- Definitions.
-
-  a. Adapted Material means material subject to Copyright and Similar
-     Rights that is derived from or based upon the Licensed Material
-     and in which the Licensed Material is translated, altered,
-     arranged, transformed, or otherwise modified in a manner requiring
-     permission under the Copyright and Similar Rights held by the
-     Licensor. For purposes of this Public License, where the Licensed
-     Material is a musical work, performance, or sound recording,
-     Adapted Material is always produced where the Licensed Material is
-     synched in timed relation with a moving image.
-
-  b. Copyright and Similar Rights means copyright and/or similar rights
-     closely related to copyright including, without limitation,
-     performance, broadcast, sound recording, and Sui Generis Database
-     Rights, without regard to how the rights are labeled or
-     categorized. For purposes of this Public License, the rights
-     specified in Section 2(b)(1)-(2) are not Copyright and Similar
-     Rights.
-
-  c. Effective Technological Measures means those measures that, in the
-     absence of proper authority, may not be circumvented under laws
-     fulfilling obligations under Article 11 of the WIPO Copyright
-     Treaty adopted on December 20, 1996, and/or similar international
-     agreements.
-
-  d. Exceptions and Limitations means fair use, fair dealing, and/or
-     any other exception or limitation to Copyright and Similar Rights
-     that applies to Your use of the Licensed Material.
-
-  e. Licensed Material means the artistic or literary work, database,
-     or other material to which the Licensor applied this Public
-     License.
-
-  f. Licensed Rights means the rights granted to You subject to the
-     terms and conditions of this Public License, which are limited to
-     all Copyright and Similar Rights that apply to Your use of the
-     Licensed Material and that the Licensor has authority to license.
-
-  g. Licensor means the individual(s) or entity(ies) granting rights
-     under this Public License.
-
-  h. NonCommercial means not primarily intended for or directed towards
-     commercial advantage or monetary compensation. For purposes of
-     this Public License, the exchange of the Licensed Material for
-     other material subject to Copyright and Similar Rights by digital
-     file-sharing or similar means is NonCommercial provided there is
-     no payment of monetary compensation in connection with the
-     exchange.
-
-  i. Share means to provide material to the public by any means or
-     process that requires permission under the Licensed Rights, such
-     as reproduction, public display, public performance, distribution,
-     dissemination, communication, or importation, and to make material
-     available to the public including in ways that members of the
-     public may access the material from a place and at a time
-     individually chosen by them.
-
-  j. Sui Generis Database Rights means rights other than copyright
-     resulting from Directive 96/9/EC of the European Parliament and of
-     the Council of 11 March 1996 on the legal protection of databases,
-     as amended and/or succeeded, as well as other essentially
-     equivalent rights anywhere in the world.
-
-  k. You means the individual or entity exercising the Licensed Rights
-     under this Public License. Your has a corresponding meaning.
-
-
-Section 2 -- Scope.
-
-  a. License grant.
-
-       1. Subject to the terms and conditions of this Public License,
-          the Licensor hereby grants You a worldwide, royalty-free,
-          non-sublicensable, non-exclusive, irrevocable license to
-          exercise the Licensed Rights in the Licensed Material to:
-
-            a. reproduce and Share the Licensed Material, in whole or
-               in part, for NonCommercial purposes only; and
-
-            b. produce and reproduce, but not Share, Adapted Material
-               for NonCommercial purposes only.
-
-       2. Exceptions and Limitations. For the avoidance of doubt, where
-          Exceptions and Limitations apply to Your use, this Public
-          License does not apply, and You do not need to comply with
-          its terms and conditions.
-
-       3. Term. The term of this Public License is specified in Section
-          6(a).
-
-       4. Media and formats; technical modifications allowed. The
-          Licensor authorizes You to exercise the Licensed Rights in
-          all media and formats whether now known or hereafter created,
-          and to make technical modifications necessary to do so. The
-          Licensor waives and/or agrees not to assert any right or
-          authority to forbid You from making technical modifications
-          necessary to exercise the Licensed Rights, including
-          technical modifications necessary to circumvent Effective
-          Technological Measures. For purposes of this Public License,
-          simply making modifications authorized by this Section 2(a)
-          (4) never produces Adapted Material.
-
-       5. Downstream recipients.
-
-            a. Offer from the Licensor -- Licensed Material. Every
-               recipient of the Licensed Material automatically
-               receives an offer from the Licensor to exercise the
-               Licensed Rights under the terms and conditions of this
-               Public License.
-
-            b. No downstream restrictions. You may not offer or impose
-               any additional or different terms or conditions on, or
-               apply any Effective Technological Measures to, the
-               Licensed Material if doing so restricts exercise of the
-               Licensed Rights by any recipient of the Licensed
-               Material.
-
-       6. No endorsement. Nothing in this Public License constitutes or
-          may be construed as permission to assert or imply that You
-          are, or that Your use of the Licensed Material is, connected
-          with, or sponsored, endorsed, or granted official status by,
-          the Licensor or others designated to receive attribution as
-          provided in Section 3(a)(1)(A)(i).
-
-  b. Other rights.
-
-       1. Moral rights, such as the right of integrity, are not
-          licensed under this Public License, nor are publicity,
-          privacy, and/or other similar personality rights; however, to
-          the extent possible, the Licensor waives and/or agrees not to
-          assert any such rights held by the Licensor to the limited
-          extent necessary to allow You to exercise the Licensed
-          Rights, but not otherwise.
-
-       2. Patent and trademark rights are not licensed under this
-          Public License.
-
-       3. To the extent possible, the Licensor waives any right to
-          collect royalties from You for the exercise of the Licensed
-          Rights, whether directly or through a collecting society
-          under any voluntary or waivable statutory or compulsory
-          licensing scheme. In all other cases the Licensor expressly
-          reserves any right to collect such royalties, including when
-          the Licensed Material is used other than for NonCommercial
-          purposes.
-
-
-Section 3 -- License Conditions.
-
-Your exercise of the Licensed Rights is expressly made subject to the
-following conditions.
-
-  a. Attribution.
-
-       1. If You Share the Licensed Material, You must:
-
-            a. retain the following if it is supplied by the Licensor
-               with the Licensed Material:
-
-                 i. identification of the creator(s) of the Licensed
-                    Material and any others designated to receive
-                    attribution, in any reasonable manner requested by
-                    the Licensor (including by pseudonym if
-                    designated);
-
-                ii. a copyright notice;
-
-               iii. a notice that refers to this Public License;
-
-                iv. a notice that refers to the disclaimer of
-                    warranties;
-
-                 v. a URI or hyperlink to the Licensed Material to the
-                    extent reasonably practicable;
-
-            b. indicate if You modified the Licensed Material and
-               retain an indication of any previous modifications; and
-
-            c. indicate the Licensed Material is licensed under this
-               Public License, and include the text of, or the URI or
-               hyperlink to, this Public License.
-
-          For the avoidance of doubt, You do not have permission under
-          this Public License to Share Adapted Material.
-
-       2. You may satisfy the conditions in Section 3(a)(1) in any
-          reasonable manner based on the medium, means, and context in
-          which You Share the Licensed Material. For example, it may be
-          reasonable to satisfy the conditions by providing a URI or
-          hyperlink to a resource that includes the required
-          information.
-
-       3. If requested by the Licensor, You must remove any of the
-          information required by Section 3(a)(1)(A) to the extent
-          reasonably practicable.
-
-
-Section 4 -- Sui Generis Database Rights.
-
-Where the Licensed Rights include Sui Generis Database Rights that
-apply to Your use of the Licensed Material:
-
-  a. for the avoidance of doubt, Section 2(a)(1) grants You the right
-     to extract, reuse, reproduce, and Share all or a substantial
-     portion of the contents of the database for NonCommercial purposes
-     only and provided You do not Share Adapted Material;
-
-  b. if You include all or a substantial portion of the database
-     contents in a database in which You have Sui Generis Database
-     Rights, then the database in which You have Sui Generis Database
-     Rights (but not its individual contents) is Adapted Material; and
-
-  c. You must comply with the conditions in Section 3(a) if You Share
-     all or a substantial portion of the contents of the database.
-
-For the avoidance of doubt, this Section 4 supplements and does not
-replace Your obligations under this Public License where the Licensed
-Rights include other Copyright and Similar Rights.
-
-
-Section 5 -- Disclaimer of Warranties and Limitation of Liability.
-
-  a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE
-     EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS
-     AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF
-     ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS,
-     IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION,
-     WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR
-     PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS,
-     ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT
-     KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT
-     ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU.
-
-  b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE
-     TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION,
-     NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT,
-     INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES,
-     COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR
-     USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN
-     ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR
-     DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR
-     IN PART, THIS LIMITATION MAY NOT APPLY TO YOU.
-
-  c. The disclaimer of warranties and limitation of liability provided
-     above shall be interpreted in a manner that, to the extent
-     possible, most closely approximates an absolute disclaimer and
-     waiver of all liability.
-
-
-Section 6 -- Term and Termination.
-
-  a. This Public License applies for the term of the Copyright and
-     Similar Rights licensed here. However, if You fail to comply with
-     this Public License, then Your rights under this Public License
-     terminate automatically.
-
-  b. Where Your right to use the Licensed Material has terminated under
-     Section 6(a), it reinstates:
-
-       1. automatically as of the date the violation is cured, provided
-          it is cured within 30 days of Your discovery of the
-          violation; or
-
-       2. upon express reinstatement by the Licensor.
-
-     For the avoidance of doubt, this Section 6(b) does not affect any
-     right the Licensor may have to seek remedies for Your violations
-     of this Public License.
-
-  c. For the avoidance of doubt, the Licensor may also offer the
-     Licensed Material under separate terms or conditions or stop
-     distributing the Licensed Material at any time; however, doing so
-     will not terminate this Public License.
-
-  d. Sections 1, 5, 6, 7, and 8 survive termination of this Public
-     License.
-
-
-Section 7 -- Other Terms and Conditions.
-
-  a. The Licensor shall not be bound by any additional or different
-     terms or conditions communicated by You unless expressly agreed.
-
-  b. Any arrangements, understandings, or agreements regarding the
-     Licensed Material not stated herein are separate from and
-     independent of the terms and conditions of this Public License.
-
-
-Section 8 -- Interpretation.
-
-  a. For the avoidance of doubt, this Public License does not, and
-     shall not be interpreted to, reduce, limit, restrict, or impose
-     conditions on any use of the Licensed Material that could lawfully
-     be made without permission under this Public License.
-
-  b. To the extent possible, if any provision of this Public License is
-     deemed unenforceable, it shall be automatically reformed to the
-     minimum extent necessary to make it enforceable. If the provision
-     cannot be reformed, it shall be severed from this Public License
-     without affecting the enforceability of the remaining terms and
-     conditions.
-
-  c. No term or condition of this Public License will be waived and no
-     failure to comply consented to unless expressly agreed to by the
-     Licensor.
-
-  d. Nothing in this Public License constitutes or may be interpreted
-     as a limitation upon, or waiver of, any privileges and immunities
-     that apply to the Licensor or You, including from the legal
-     processes of any jurisdiction or authority.
-
-=======================================================================
-
-Creative Commons is not a party to its public
-licenses. Notwithstanding, Creative Commons may elect to apply one of
-its public licenses to material it publishes and in those instances
-will be considered the “Licensor.” The text of the Creative Commons
-public licenses is dedicated to the public domain under the CC0 Public
-Domain Dedication. Except for the limited purpose of indicating that
-material is shared under a Creative Commons public license or as
-otherwise permitted by the Creative Commons policies published at
-creativecommons.org/policies, Creative Commons does not authorize the
-use of the trademark "Creative Commons" or any other trademark or logo
-of Creative Commons without its prior written consent including,
-without limitation, in connection with any unauthorized modifications
-to any of its public licenses or any other arrangements,
-understandings, or agreements concerning use of licensed material. For
-the avoidance of doubt, this paragraph does not form part of the
-public licenses.
-
-Creative Commons may be contacted at creativecommons.org.

WiLoR/mano_data/mano_mean_params.npz

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:efc0ec58e4a5cef78f3abfb4e8f91623b8950be9eff8b8e0dbb0d036ebc63988
-size 1178

WiLoR/pretrained_models/dataset_config.yaml

@@ -1,58 +0,0 @@
-ARCTIC-TRAIN:
-  TYPE: ImageDataset
-  URLS: wilor_training_data/dataset_tars/arctic-train/{000000..000176}.tar
-  epoch_size: 177000
-BEDLAM-TRAIN:
-  TYPE: ImageDataset
-  URLS: wilor_training_data/dataset_tars/bedlam-train/{000000..000300}.tar
-  epoch_size: 301000
-COCOW-TRAIN:
-  TYPE: ImageDataset
-  URLS: wilor_training_data/dataset_tars/cocow-train/{000000..000036}.tar
-  epoch_size: 78666
-DEX-TRAIN:
-  TYPE: ImageDataset
-  URLS: wilor_training_data/dataset_tars/dex-train/{000000..000406}.tar
-  epoch_size: 406888
-FREIHAND-MOCAP:
-  DATASET_FILE: wilor_training_data/freihand_mocap.npz
-FREIHAND-TRAIN:
-  TYPE: ImageDataset
-  URLS: wilor_training_data/dataset_tars/freihand-train/{000000..000130}.tar
-  epoch_size: 130240
-H2O3D-TRAIN:
-  TYPE: ImageDataset
-  URLS: wilor_training_data/dataset_tars/h2o3d-train/{000000..000060}.tar
-  epoch_size: 121996
-HALPE-TRAIN:
-  TYPE: ImageDataset
-  URLS: wilor_training_data/dataset_tars/halpe-train/{000000..000022}.tar
-  epoch_size: 34289
-HO3D-TRAIN:
-  TYPE: ImageDataset
-  URLS: wilor_training_data/dataset_tars/ho3d-train/{000000..000083}.tar
-  epoch_size: 83325
-HOT3D-TRAIN:
-  TYPE: ImageDataset
-  URLS: wilor_training_data/dataset_tars/hot3d-train/{000000..000571}.tar
-  epoch_size: 572000
-INTERHAND26M-TRAIN:
-  TYPE: ImageDataset
-  URLS: wilor_training_data/dataset_tars/interhand26m-train/{000000..001056}.tar
-  epoch_size: 1424632
-MPIINZSL-TRAIN:
-  TYPE: ImageDataset
-  URLS: wilor_training_data/dataset_tars/mpiinzsl-train/{000000..000015}.tar
-  epoch_size: 15184
-MTC-TRAIN:
-  TYPE: ImageDataset
-  URLS: wilor_training_data/dataset_tars/mtc-train/{000000..000306}.tar
-  epoch_size: 363947
-REINTER-TRAIN:
-  TYPE: ImageDataset
-  URLS: wilor_training_data/dataset_tars/reinter-train/{000000..000418}.tar
-  epoch_size: 419000
-RHD-TRAIN:
-  TYPE: ImageDataset
-  URLS: wilor_training_data/dataset_tars/rhd-train/{000000..000041}.tar
-  epoch_size: 61705

WiLoR/pretrained_models/model_config.yaml

@@ -1,119 +0,0 @@
-task_name: train
-tags:
-- dev
-train: true
-test: false
-ckpt_path: null
-seed: null
-DATASETS:
-  TRAIN:
-    FREIHAND-TRAIN:
-      WEIGHT: 0.2
-    INTERHAND26M-TRAIN:
-      WEIGHT: 0.1
-    MTC-TRAIN:
-      WEIGHT: 0.05
-    RHD-TRAIN:
-      WEIGHT: 0.05
-    COCOW-TRAIN:
-      WEIGHT: 0.05
-    HALPE-TRAIN:
-      WEIGHT: 0.05
-    MPIINZSL-TRAIN:
-      WEIGHT: 0.05
-    HO3D-TRAIN:
-      WEIGHT: 0.05
-    H2O3D-TRAIN:
-      WEIGHT: 0.05
-    DEX-TRAIN:
-      WEIGHT: 0.05
-    BEDLAM-TRAIN:
-      WEIGHT: 0.05
-    REINTER-TRAIN:
-      WEIGHT: 0.1
-    HOT3D-TRAIN:
-      WEIGHT: 0.05
-    ARCTIC-TRAIN:
-      WEIGHT: 0.1
-  VAL:
-    FREIHAND-TRAIN:
-      WEIGHT: 1.0
-  MOCAP: FREIHAND-MOCAP
-  BETAS_REG: true
-  CONFIG:
-    SCALE_FACTOR: 0.3
-    ROT_FACTOR: 30
-    TRANS_FACTOR: 0.02
-    COLOR_SCALE: 0.2
-    ROT_AUG_RATE: 0.6
-    TRANS_AUG_RATE: 0.5
-    DO_FLIP: false
-    FLIP_AUG_RATE: 0.0
-    EXTREME_CROP_AUG_RATE: 0.0
-    EXTREME_CROP_AUG_LEVEL: 1
-extras:
-  ignore_warnings: false
-  enforce_tags: true
-  print_config: true
-exp_name: WiLoR
-MANO:
-  DATA_DIR: mano_data
-  MODEL_PATH: ${MANO.DATA_DIR}
-  GENDER: neutral
-  NUM_HAND_JOINTS: 15
-  MEAN_PARAMS: ${MANO.DATA_DIR}/mano_mean_params.npz
-  CREATE_BODY_POSE: false
-EXTRA:
-  FOCAL_LENGTH: 5000
-  NUM_LOG_IMAGES: 4
-  NUM_LOG_SAMPLES_PER_IMAGE: 8
-  PELVIS_IND: 0
-GENERAL:
-  TOTAL_STEPS: 1000000
-  LOG_STEPS: 1000
-  VAL_STEPS: 1000
-  CHECKPOINT_STEPS: 1000
-  CHECKPOINT_SAVE_TOP_K: 1
-  NUM_WORKERS: 8
-  PREFETCH_FACTOR: 2
-TRAIN:
-  LR: 1.0e-05
-  WEIGHT_DECAY: 0.0001
-  BATCH_SIZE: 32
-  LOSS_REDUCTION: mean
-  NUM_TRAIN_SAMPLES: 2
-  NUM_TEST_SAMPLES: 64
-  POSE_2D_NOISE_RATIO: 0.01
-  SMPL_PARAM_NOISE_RATIO: 0.005
-MODEL:
-  IMAGE_SIZE: 256
-  IMAGE_MEAN:
-  - 0.485
-  - 0.456
-  - 0.406
-  IMAGE_STD:
-  - 0.229
-  - 0.224
-  - 0.225
-  BACKBONE:
-    TYPE: vit
-    PRETRAINED_WEIGHTS: training_data/vitpose_backbone.pth
-  MANO_HEAD:
-    TYPE: transformer_decoder
-    IN_CHANNELS: 2048
-    TRANSFORMER_DECODER:
-      depth: 6
-      heads: 8
-      mlp_dim: 1024
-      dim_head: 64
-      dropout: 0.0
-      emb_dropout: 0.0
-      norm: layer
-      context_dim: 1280
-LOSS_WEIGHTS:
-  KEYPOINTS_3D: 0.05
-  KEYPOINTS_2D: 0.01
-  GLOBAL_ORIENT: 0.001
-  HAND_POSE: 0.001
-  BETAS: 0.0005
-  ADVERSARIAL: 0.0005

WiLoR/requirements.txt

@@ -1,20 +0,0 @@
-numpy
-opencv-python
-pyrender
-pytorch-lightning
-scikit-image
-smplx==0.1.28
-yacs
-chumpy @ git+https://github.com/mattloper/chumpy
-timm
-einops
-xtcocotools
-pandas
-hydra-core
-hydra-submitit-launcher
-hydra-colorlog
-pyrootutils
-rich
-webdataset
-gradio
-ultralytics==8.1.34

WiLoR/whim/Dataset_instructions.md

@@ -1,31 +0,0 @@
-## WHIM Dataset
-
-**Annotations** 
-
-The image annotations can be downloaded from the following Drive:
-
-```
-https://drive.google.com/drive/folders/1d9Fw7LfnF5oJuA6yE8T3xA-u9p6H5ObZ
-```
-
-**[Alternative]**: The image annotations can be also downloaded from Hugging Face:
-```
-https://huggingface.co/datasets/rolpotamias/WHIM
-```
-If you are using Hugging Face you might need to merge the training zip files into a single file before uncompressing: 
-```
-cat train_split.zip* > ~/train_split.zip
-```
-
-**Images** 
-
-To download the corresponding images you need to first download the YouTube videos and extract the specific frames.
-You will need to install ''pytubefix'' or any similar package to download YouTube videos: 
-```
-pip install -Iv pytubefix==8.12.2
-```
-You can then run the following command to download the corresponding train/test images: 
-```
-python download_videos.py --mode {train/test}
-```
-Please make sure that the data are downloaded in the same directory. 

WiLoR/whim/test_video_ids.json

@@ -1 +0,0 @@
-{"YynYZyoETto": {"res": [360, 480], "length": 4678, "fps": 29.97002997002997}, "_iirwC_DvJ0": {"res": [480, 854], "length": 7994, "fps": 29.97002997002997}, "ZMnb9TTsx98": {"res": [1080, 1920], "length": 8109, "fps": 29.97002997002997}, "IrSIHJ0-AaU": {"res": [360, 640], "length": 880, "fps": 30.0}, "w2ULyzWkZ3k": {"res": [1080, 1920], "length": 17032, "fps": 29.97002997002997}, "ivyqQreoVQA": {"res": [1080, 1440], "length": 9610, "fps": 29.97002997002997}, "R07f8kg1h8o": {"res": [1080, 1920], "length": 10726, "fps": 23.976023976023978}, "7S9q1kAVmc0": {"res": [720, 1280], "length": 53757, "fps": 25.0}, "_Ce7G35GIqA": {"res": [720, 1280], "length": 1620, "fps": 30.0}, "lhHkJ3InQOE": {"res": [240, 320], "length": 1600, "fps": 11.988011988011989}, "NXRHcCScubA": {"res": [1080, 1920], "length": 9785, "fps": 29.97002997002997}, "DjFX4idkS3o": {"res": [720, 1280], "length": 5046, "fps": 29.97002997002997}, "06kKvQp4SfM": {"res": [720, 1280], "length": 2661, "fps": 30.0}, "8NqJiAu9W3Y": {"res": [720, 1280], "length": 4738, "fps": 29.97002997002997}, "nN5Y--biYv4": {"res": [720, 1280], "length": 38380, "fps": 29.97}, "OiAlJIaWOBg": {"res": [720, 1280], "length": 10944, "fps": 30.0}, "nJa_omJBzoU": {"res": [720, 1280], "length": 4311, "fps": 29.97002997002997}, "ff_xcsFJ8Pw": {"res": [720, 1280], "length": 5631, "fps": 29.97}, "Y1mNu5iFwMg": {"res": [720, 1280], "length": 7060, "fps": 30.0}, "Ipe9xJCfuTM": {"res": [1080, 1920], "length": 52419, "fps": 29.97002997002997}, "vRkcw9SRems": {"res": [1080, 1920], "length": 10282, "fps": 23.976023976023978}, "ChIJjJyBjQ0": {"res": [1080, 1920], "length": 20228, "fps": 29.97002997002997}, "bxZtXdVvfpc": {"res": [1080, 1920], "length": 2369, "fps": 23.976023976023978}, "MPeXy2U4yJM": {"res": [1080, 1920], "length": 6760, "fps": 24.0}, "wnKnoui3THA": {"res": [1080, 1920], "length": 7934, "fps": 25.0}, "gnArvcWaH6I": {"res": [480, 720], "length": 6864, "fps": 29.97002997002997}}

WiLoR/wilor/configs/__init__.py

@@ -1,114 +0,0 @@
-import os
-from typing import Dict
-from yacs.config import CfgNode as CN
-
-CACHE_DIR_PRETRAINED = "./pretrained_models/"
-
-def to_lower(x: Dict) -> Dict:
-    """
-    Convert all dictionary keys to lowercase
-    Args:
-      x (dict): Input dictionary
-    Returns:
-      dict: Output dictionary with all keys converted to lowercase
-    """
-    return {k.lower(): v for k, v in x.items()}
-
-_C = CN(new_allowed=True)
-
-_C.GENERAL = CN(new_allowed=True)
-_C.GENERAL.RESUME = True
-_C.GENERAL.TIME_TO_RUN = 3300
-_C.GENERAL.VAL_STEPS = 100
-_C.GENERAL.LOG_STEPS = 100
-_C.GENERAL.CHECKPOINT_STEPS = 20000
-_C.GENERAL.CHECKPOINT_DIR = "checkpoints"
-_C.GENERAL.SUMMARY_DIR = "tensorboard"
-_C.GENERAL.NUM_GPUS = 1
-_C.GENERAL.NUM_WORKERS = 4
-_C.GENERAL.MIXED_PRECISION = True
-_C.GENERAL.ALLOW_CUDA = True
-_C.GENERAL.PIN_MEMORY = False
-_C.GENERAL.DISTRIBUTED = False
-_C.GENERAL.LOCAL_RANK = 0
-_C.GENERAL.USE_SYNCBN = False
-_C.GENERAL.WORLD_SIZE = 1
-
-_C.TRAIN = CN(new_allowed=True)
-_C.TRAIN.NUM_EPOCHS = 100
-_C.TRAIN.BATCH_SIZE = 32
-_C.TRAIN.SHUFFLE = True
-_C.TRAIN.WARMUP = False
-_C.TRAIN.NORMALIZE_PER_IMAGE = False
-_C.TRAIN.CLIP_GRAD = False
-_C.TRAIN.CLIP_GRAD_VALUE = 1.0
-_C.LOSS_WEIGHTS = CN(new_allowed=True)
-
-_C.DATASETS = CN(new_allowed=True)
-
-_C.MODEL = CN(new_allowed=True)
-_C.MODEL.IMAGE_SIZE = 224
-
-_C.EXTRA = CN(new_allowed=True)
-_C.EXTRA.FOCAL_LENGTH = 5000
-
-_C.DATASETS.CONFIG = CN(new_allowed=True)
-_C.DATASETS.CONFIG.SCALE_FACTOR = 0.3
-_C.DATASETS.CONFIG.ROT_FACTOR = 30
-_C.DATASETS.CONFIG.TRANS_FACTOR = 0.02
-_C.DATASETS.CONFIG.COLOR_SCALE = 0.2
-_C.DATASETS.CONFIG.ROT_AUG_RATE = 0.6
-_C.DATASETS.CONFIG.TRANS_AUG_RATE = 0.5
-_C.DATASETS.CONFIG.DO_FLIP = False
-_C.DATASETS.CONFIG.FLIP_AUG_RATE = 0.5
-_C.DATASETS.CONFIG.EXTREME_CROP_AUG_RATE = 0.10
-
-def default_config() -> CN:
-    """
-    Get a yacs CfgNode object with the default config values.
-    """
-    # Return a clone so that the defaults will not be altered
-    # This is for the "local variable" use pattern
-    return _C.clone()
-
-def dataset_config(name='datasets_tar.yaml') -> CN:
-    """
-    Get dataset config file
-    Returns:
-      CfgNode: Dataset config as a yacs CfgNode object.
-    """
-    cfg = CN(new_allowed=True)
-    config_file = os.path.join(os.path.dirname(os.path.realpath(__file__)), name)
-    cfg.merge_from_file(config_file)
-    cfg.freeze()
-    return cfg
-
-def dataset_eval_config() -> CN:
-    return dataset_config('datasets_eval.yaml')
-
-def get_config(config_file: str, merge: bool = True, update_cachedir: bool = False) -> CN:
-    """
-    Read a config file and optionally merge it with the default config file.
-    Args:
-      config_file (str): Path to config file.
-      merge (bool): Whether to merge with the default config or not.
-    Returns:
-      CfgNode: Config as a yacs CfgNode object.
-    """
-    if merge:
-      cfg = default_config()
-    else:
-      cfg = CN(new_allowed=True)
-    cfg.merge_from_file(config_file)
-
-    if update_cachedir:
-      def update_path(path: str) -> str:
-        if os.path.isabs(path):
-          return path
-        return os.path.join(CACHE_DIR_PRETRAINED, path)
-
-      cfg.MANO.MODEL_PATH = update_path(cfg.MANO.MODEL_PATH)
-      cfg.MANO.MEAN_PARAMS = update_path(cfg.MANO.MEAN_PARAMS)
-
-    cfg.freeze()
-    return cfg

WiLoR/wilor/datasets/utils.py

@@ -1,994 +0,0 @@
-"""
-Parts of the code are taken or adapted from
-https://github.com/mkocabas/EpipolarPose/blob/master/lib/utils/img_utils.py
-"""
-import torch
-import numpy as np
-from skimage.transform import rotate, resize
-from skimage.filters import gaussian
-import random
-import cv2
-from typing import List, Dict, Tuple
-from yacs.config import CfgNode
-
-def expand_to_aspect_ratio(input_shape, target_aspect_ratio=None):
-    """Increase the size of the bounding box to match the target shape."""
-    if target_aspect_ratio is None:
-        return input_shape
-
-    try:
-        w , h = input_shape
-    except (ValueError, TypeError):
-        return input_shape
-
-    w_t, h_t = target_aspect_ratio
-    if h / w < h_t / w_t:
-        h_new = max(w * h_t / w_t, h)
-        w_new = w
-    else:
-        h_new = h
-        w_new = max(h * w_t / h_t, w)
-    if h_new < h or w_new < w:
-        breakpoint()
-    return np.array([w_new, h_new])
-
-def do_augmentation(aug_config: CfgNode) -> Tuple:
-    """
-    Compute random augmentation parameters.
-    Args:
-        aug_config (CfgNode): Config containing augmentation parameters.
-    Returns:
-        scale (float): Box rescaling factor.
-        rot (float): Random image rotation.
-        do_flip (bool): Whether to flip image or not.
-        do_extreme_crop (bool): Whether to apply extreme cropping (as proposed in EFT).
-        color_scale (List): Color rescaling factor
-        tx (float): Random translation along the x axis.
-        ty (float): Random translation along the y axis. 
-    """
-
-    tx = np.clip(np.random.randn(), -1.0, 1.0) * aug_config.TRANS_FACTOR
-    ty = np.clip(np.random.randn(), -1.0, 1.0) * aug_config.TRANS_FACTOR
-    scale = np.clip(np.random.randn(), -1.0, 1.0) * aug_config.SCALE_FACTOR + 1.0
-    rot = np.clip(np.random.randn(), -2.0,
-                  2.0) * aug_config.ROT_FACTOR if random.random() <= aug_config.ROT_AUG_RATE else 0
-    do_flip = aug_config.DO_FLIP and random.random() <= aug_config.FLIP_AUG_RATE
-    do_extreme_crop = random.random() <= aug_config.EXTREME_CROP_AUG_RATE
-    extreme_crop_lvl = aug_config.get('EXTREME_CROP_AUG_LEVEL', 0)
-    # extreme_crop_lvl = 0
-    c_up = 1.0 + aug_config.COLOR_SCALE
-    c_low = 1.0 - aug_config.COLOR_SCALE
-    color_scale = [random.uniform(c_low, c_up), random.uniform(c_low, c_up), random.uniform(c_low, c_up)]
-    return scale, rot, do_flip, do_extreme_crop, extreme_crop_lvl, color_scale, tx, ty
-
-def rotate_2d(pt_2d: np.array, rot_rad: float) -> np.array:
-    """
-    Rotate a 2D point on the x-y plane.
-    Args:
-        pt_2d (np.array): Input 2D point with shape (2,).
-        rot_rad (float): Rotation angle
-    Returns:
-        np.array: Rotated 2D point.
-    """
-    x = pt_2d[0]
-    y = pt_2d[1]
-    sn, cs = np.sin(rot_rad), np.cos(rot_rad)
-    xx = x * cs - y * sn
-    yy = x * sn + y * cs
-    return np.array([xx, yy], dtype=np.float32)
-
-
-def gen_trans_from_patch_cv(c_x: float, c_y: float,
-                            src_width: float, src_height: float,
-                            dst_width: float, dst_height: float,
-                            scale: float, rot: float) -> np.array:
-    """
-    Create transformation matrix for the bounding box crop.
-    Args:
-        c_x (float): Bounding box center x coordinate in the original image.
-        c_y (float): Bounding box center y coordinate in the original image.
-        src_width (float): Bounding box width.
-        src_height (float): Bounding box height.
-        dst_width (float): Output box width.
-        dst_height (float): Output box height.
-        scale (float): Rescaling factor for the bounding box (augmentation).
-        rot (float): Random rotation applied to the box.
-    Returns:
-        trans (np.array): Target geometric transformation.
-    """
-    # augment size with scale
-    src_w = src_width * scale
-    src_h = src_height * scale
-    src_center = np.zeros(2)
-    src_center[0] = c_x
-    src_center[1] = c_y
-    # augment rotation
-    rot_rad = np.pi * rot / 180
-    src_downdir = rotate_2d(np.array([0, src_h * 0.5], dtype=np.float32), rot_rad)
-    src_rightdir = rotate_2d(np.array([src_w * 0.5, 0], dtype=np.float32), rot_rad)
-
-    dst_w = dst_width
-    dst_h = dst_height
-    dst_center = np.array([dst_w * 0.5, dst_h * 0.5], dtype=np.float32)
-    dst_downdir = np.array([0, dst_h * 0.5], dtype=np.float32)
-    dst_rightdir = np.array([dst_w * 0.5, 0], dtype=np.float32)
-
-    src = np.zeros((3, 2), dtype=np.float32)
-    src[0, :] = src_center
-    src[1, :] = src_center + src_downdir
-    src[2, :] = src_center + src_rightdir
-
-    dst = np.zeros((3, 2), dtype=np.float32)
-    dst[0, :] = dst_center
-    dst[1, :] = dst_center + dst_downdir
-    dst[2, :] = dst_center + dst_rightdir
-
-    trans = cv2.getAffineTransform(np.float32(src), np.float32(dst))
-
-    return trans
-
-
-def trans_point2d(pt_2d: np.array, trans: np.array):
-    """
-    Transform a 2D point using translation matrix trans.
-    Args:
-        pt_2d (np.array): Input 2D point with shape (2,).
-        trans (np.array): Transformation matrix.
-    Returns:
-        np.array: Transformed 2D point.
-    """
-    src_pt = np.array([pt_2d[0], pt_2d[1], 1.]).T
-    dst_pt = np.dot(trans, src_pt)
-    return dst_pt[0:2]
-
-def get_transform(center, scale, res, rot=0):
-    """Generate transformation matrix."""
-    """Taken from PARE: https://github.com/mkocabas/PARE/blob/6e0caca86c6ab49ff80014b661350958e5b72fd8/pare/utils/image_utils.py"""
-    h = 200 * scale
-    t = np.zeros((3, 3))
-    t[0, 0] = float(res[1]) / h
-    t[1, 1] = float(res[0]) / h
-    t[0, 2] = res[1] * (-float(center[0]) / h + .5)
-    t[1, 2] = res[0] * (-float(center[1]) / h + .5)
-    t[2, 2] = 1
-    if not rot == 0:
-        rot = -rot  # To match direction of rotation from cropping
-        rot_mat = np.zeros((3, 3))
-        rot_rad = rot * np.pi / 180
-        sn, cs = np.sin(rot_rad), np.cos(rot_rad)
-        rot_mat[0, :2] = [cs, -sn]
-        rot_mat[1, :2] = [sn, cs]
-        rot_mat[2, 2] = 1
-        # Need to rotate around center
-        t_mat = np.eye(3)
-        t_mat[0, 2] = -res[1] / 2
-        t_mat[1, 2] = -res[0] / 2
-        t_inv = t_mat.copy()
-        t_inv[:2, 2] *= -1
-        t = np.dot(t_inv, np.dot(rot_mat, np.dot(t_mat, t)))
-    return t
-
-
-def transform(pt, center, scale, res, invert=0, rot=0, as_int=True):
-    """Transform pixel location to different reference."""
-    """Taken from PARE: https://github.com/mkocabas/PARE/blob/6e0caca86c6ab49ff80014b661350958e5b72fd8/pare/utils/image_utils.py"""
-    t = get_transform(center, scale, res, rot=rot)
-    if invert:
-        t = np.linalg.inv(t)
-    new_pt = np.array([pt[0] - 1, pt[1] - 1, 1.]).T
-    new_pt = np.dot(t, new_pt)
-    if as_int:
-        new_pt = new_pt.astype(int)
-    return new_pt[:2] + 1
-
-def crop_img(img, ul, br, border_mode=cv2.BORDER_CONSTANT, border_value=0):
-    c_x = (ul[0] + br[0])/2
-    c_y = (ul[1] + br[1])/2
-    bb_width = patch_width = br[0] - ul[0]
-    bb_height = patch_height = br[1] - ul[1]
-    trans = gen_trans_from_patch_cv(c_x, c_y, bb_width, bb_height, patch_width, patch_height, 1.0, 0)
-    img_patch = cv2.warpAffine(img, trans, (int(patch_width), int(patch_height)), 
-                                flags=cv2.INTER_LINEAR, 
-                                borderMode=border_mode,
-                                borderValue=border_value
-                        )
-    
-    # Force borderValue=cv2.BORDER_CONSTANT for alpha channel
-    if (img.shape[2] == 4) and (border_mode != cv2.BORDER_CONSTANT):
-        img_patch[:,:,3] = cv2.warpAffine(img[:,:,3], trans, (int(patch_width), int(patch_height)), 
-                                            flags=cv2.INTER_LINEAR, 
-                                            borderMode=cv2.BORDER_CONSTANT,
-                            )
-
-    return img_patch
-
-def generate_image_patch_skimage(img: np.array, c_x: float, c_y: float,
-                                 bb_width: float, bb_height: float,
-                                 patch_width: float, patch_height: float,
-                                 do_flip: bool, scale: float, rot: float,
-                                 border_mode=cv2.BORDER_CONSTANT, border_value=0) -> Tuple[np.array, np.array]:
-    """
-    Crop image according to the supplied bounding box.
-    Args:
-        img (np.array): Input image of shape (H, W, 3)
-        c_x (float): Bounding box center x coordinate in the original image.
-        c_y (float): Bounding box center y coordinate in the original image.
-        bb_width (float): Bounding box width.
-        bb_height (float): Bounding box height.
-        patch_width (float): Output box width.
-        patch_height (float): Output box height.
-        do_flip (bool): Whether to flip image or not.
-        scale (float): Rescaling factor for the bounding box (augmentation).
-        rot (float): Random rotation applied to the box.
-    Returns:
-        img_patch (np.array): Cropped image patch of shape (patch_height, patch_height, 3)
-        trans (np.array): Transformation matrix.
-    """
-    
-    img_height, img_width, img_channels = img.shape
-    if do_flip:
-       img = img[:, ::-1, :]
-       c_x = img_width - c_x - 1
-
-    trans = gen_trans_from_patch_cv(c_x, c_y, bb_width, bb_height, patch_width, patch_height, scale, rot)
-
-    #img_patch = cv2.warpAffine(img, trans, (int(patch_width), int(patch_height)), flags=cv2.INTER_LINEAR)
-
-    # skimage
-    center = np.zeros(2)
-    center[0] = c_x
-    center[1] = c_y
-    res = np.zeros(2)
-    res[0] = patch_width
-    res[1] = patch_height
-    # assumes bb_width = bb_height
-    # assumes patch_width = patch_height
-    assert bb_width == bb_height, f'{bb_width=} != {bb_height=}'
-    assert patch_width == patch_height, f'{patch_width=} != {patch_height=}'
-    scale1 = scale*bb_width/200.
-    
-    # Upper left point
-    ul = np.array(transform([1, 1], center, scale1, res, invert=1, as_int=False)) - 1
-    # Bottom right point
-    br = np.array(transform([res[0] + 1,
-                             res[1] + 1], center, scale1, res, invert=1, as_int=False)) - 1
-
-    # Padding so that when rotated proper amount of context is included
-    try:
-        pad = int(np.linalg.norm(br - ul) / 2 - float(br[1] - ul[1]) / 2) + 1
-    except:
-        breakpoint()
-    if not rot == 0:
-        ul -= pad
-        br += pad
-
-
-    if False:
-        # Old way of cropping image
-        ul_int = ul.astype(int)
-        br_int = br.astype(int)
-        new_shape = [br_int[1] - ul_int[1], br_int[0] - ul_int[0]]
-        if len(img.shape) > 2:
-            new_shape += [img.shape[2]]
-        new_img = np.zeros(new_shape)
-
-        # Range to fill new array
-        new_x = max(0, -ul_int[0]), min(br_int[0], len(img[0])) - ul_int[0]
-        new_y = max(0, -ul_int[1]), min(br_int[1], len(img)) - ul_int[1]
-        # Range to sample from original image
-        old_x = max(0, ul_int[0]), min(len(img[0]), br_int[0])
-        old_y = max(0, ul_int[1]), min(len(img), br_int[1])
-        new_img[new_y[0]:new_y[1], new_x[0]:new_x[1]] = img[old_y[0]:old_y[1],
-                                                        old_x[0]:old_x[1]]
-
-    # New way of cropping image
-    new_img = crop_img(img, ul, br, border_mode=border_mode, border_value=border_value).astype(np.float32)
-
-    # print(f'{new_img.shape=}')
-    # print(f'{new_img1.shape=}')
-    # print(f'{np.allclose(new_img, new_img1)=}')
-    # print(f'{img.dtype=}')
-
-
-    if not rot == 0:
-        # Remove padding
-
-        new_img = rotate(new_img, rot) # scipy.misc.imrotate(new_img, rot)
-        new_img = new_img[pad:-pad, pad:-pad]
-
-    if new_img.shape[0] < 1 or new_img.shape[1] < 1:
-        print(f'{img.shape=}')
-        print(f'{new_img.shape=}')
-        print(f'{ul=}')
-        print(f'{br=}')
-        print(f'{pad=}')
-        print(f'{rot=}')
-
-        breakpoint()
-
-    # resize image
-    new_img = resize(new_img, res) # scipy.misc.imresize(new_img, res)
-    
-    new_img = np.clip(new_img, 0, 255).astype(np.uint8)
-
-    return new_img, trans
-
-
-def generate_image_patch_cv2(img: np.array, c_x: float, c_y: float,
-                             bb_width: float, bb_height: float,
-                             patch_width: float, patch_height: float,
-                             do_flip: bool, scale: float, rot: float,
-                             border_mode=cv2.BORDER_CONSTANT, border_value=0) -> Tuple[np.array, np.array]:
-    """
-    Crop the input image and return the crop and the corresponding transformation matrix.
-    Args:
-        img (np.array): Input image of shape (H, W, 3)
-        c_x (float): Bounding box center x coordinate in the original image.
-        c_y (float): Bounding box center y coordinate in the original image.
-        bb_width (float): Bounding box width.
-        bb_height (float): Bounding box height.
-        patch_width (float): Output box width.
-        patch_height (float): Output box height.
-        do_flip (bool): Whether to flip image or not.
-        scale (float): Rescaling factor for the bounding box (augmentation).
-        rot (float): Random rotation applied to the box.
-    Returns:
-        img_patch (np.array): Cropped image patch of shape (patch_height, patch_height, 3)
-        trans (np.array): Transformation matrix.
-    """
-
-    img_height, img_width, img_channels = img.shape
-    if do_flip:
-        img = img[:, ::-1, :]
-        c_x = img_width - c_x - 1
-
-
-    trans = gen_trans_from_patch_cv(c_x, c_y, bb_width, bb_height, patch_width, patch_height, scale, rot)
-
-    img_patch = cv2.warpAffine(img, trans, (int(patch_width), int(patch_height)), 
-                        flags=cv2.INTER_LINEAR, 
-                        borderMode=border_mode,
-                        borderValue=border_value,
-                )
-    # Force borderValue=cv2.BORDER_CONSTANT for alpha channel
-    if (img.shape[2] == 4) and (border_mode != cv2.BORDER_CONSTANT):
-        img_patch[:,:,3] = cv2.warpAffine(img[:,:,3], trans, (int(patch_width), int(patch_height)), 
-                                            flags=cv2.INTER_LINEAR, 
-                                            borderMode=cv2.BORDER_CONSTANT,
-                            )
-
-    return img_patch, trans
-
-
-def convert_cvimg_to_tensor(cvimg: np.array):
-    """
-    Convert image from HWC to CHW format.
-    Args:
-        cvimg (np.array): Image of shape (H, W, 3) as loaded by OpenCV.
-    Returns:
-        np.array: Output image of shape (3, H, W).
-    """
-    # from h,w,c(OpenCV) to c,h,w
-    img = cvimg.copy()
-    img = np.transpose(img, (2, 0, 1))
-    # from int to float
-    img = img.astype(np.float32)
-    return img
-
-def fliplr_params(mano_params: Dict, has_mano_params: Dict) -> Tuple[Dict, Dict]:
-    """
-    Flip MANO parameters when flipping the image.
-    Args:
-        mano_params (Dict): MANO parameter annotations.
-        has_mano_params (Dict): Whether MANO annotations are valid.
-    Returns:
-        Dict, Dict: Flipped MANO parameters and valid flags.
-    """
-    global_orient = mano_params['global_orient'].copy()
-    hand_pose = mano_params['hand_pose'].copy()
-    betas = mano_params['betas'].copy()
-    has_global_orient = has_mano_params['global_orient'].copy()
-    has_hand_pose = has_mano_params['hand_pose'].copy()
-    has_betas = has_mano_params['betas'].copy()
-
-    global_orient[1::3] *= -1
-    global_orient[2::3] *= -1
-    hand_pose[1::3] *= -1
-    hand_pose[2::3] *= -1
-
-    mano_params = {'global_orient': global_orient.astype(np.float32),
-                   'hand_pose': hand_pose.astype(np.float32),
-                   'betas': betas.astype(np.float32)
-                  }
-
-    has_mano_params = {'global_orient': has_global_orient,
-                       'hand_pose': has_hand_pose,
-                       'betas': has_betas
-                      }
-
-    return mano_params, has_mano_params
-
-
-def fliplr_keypoints(joints: np.array, width: float, flip_permutation: List[int]) -> np.array:
-    """
-    Flip 2D or 3D keypoints.
-    Args:
-        joints (np.array): Array of shape (N, 3) or (N, 4) containing 2D or 3D keypoint locations and confidence.
-        flip_permutation (List): Permutation to apply after flipping.
-    Returns:
-        np.array: Flipped 2D or 3D keypoints with shape (N, 3) or (N, 4) respectively.
-    """
-    joints = joints.copy()
-    # Flip horizontal
-    joints[:, 0] = width - joints[:, 0] - 1
-    joints = joints[flip_permutation, :]
-
-    return joints
-
-def keypoint_3d_processing(keypoints_3d: np.array, flip_permutation: List[int], rot: float, do_flip: float) -> np.array:
-    """
-    Process 3D keypoints (rotation/flipping).
-    Args:
-        keypoints_3d (np.array): Input array of shape (N, 4) containing the 3D keypoints and confidence.
-        flip_permutation (List): Permutation to apply after flipping.
-        rot (float): Random rotation applied to the keypoints.
-        do_flip (bool): Whether to flip keypoints or not.
-    Returns:
-        np.array: Transformed 3D keypoints with shape (N, 4).
-    """
-    if do_flip:
-        keypoints_3d = fliplr_keypoints(keypoints_3d, 1, flip_permutation)
-    # in-plane rotation
-    rot_mat = np.eye(3)
-    if not rot == 0:
-        rot_rad = -rot * np.pi / 180
-        sn,cs = np.sin(rot_rad), np.cos(rot_rad)
-        rot_mat[0,:2] = [cs, -sn]
-        rot_mat[1,:2] = [sn, cs]
-    keypoints_3d[:, :-1] = np.einsum('ij,kj->ki', rot_mat, keypoints_3d[:, :-1])
-    # flip the x coordinates
-    keypoints_3d = keypoints_3d.astype('float32')
-    return keypoints_3d
-
-def rot_aa(aa: np.array, rot: float) -> np.array:
-    """
-    Rotate axis angle parameters.
-    Args:
-        aa (np.array): Axis-angle vector of shape (3,).
-        rot (np.array): Rotation angle in degrees.
-    Returns:
-        np.array: Rotated axis-angle vector.
-    """
-    # pose parameters
-    R = np.array([[np.cos(np.deg2rad(-rot)), -np.sin(np.deg2rad(-rot)), 0],
-                  [np.sin(np.deg2rad(-rot)), np.cos(np.deg2rad(-rot)), 0],
-                  [0, 0, 1]])
-    # find the rotation of the hand in camera frame
-    per_rdg, _ = cv2.Rodrigues(aa)
-    # apply the global rotation to the global orientation
-    resrot, _ = cv2.Rodrigues(np.dot(R,per_rdg))
-    aa = (resrot.T)[0]
-    return aa.astype(np.float32)
-
-def mano_param_processing(mano_params: Dict, has_mano_params: Dict, rot: float, do_flip: bool) -> Tuple[Dict, Dict]:
-    """
-    Apply random augmentations to the MANO parameters.
-    Args:
-        mano_params (Dict): MANO parameter annotations.
-        has_mano_params (Dict): Whether mano annotations are valid.
-        rot (float): Random rotation applied to the keypoints.
-        do_flip (bool): Whether to flip keypoints or not.
-    Returns:
-        Dict, Dict: Transformed MANO parameters and valid flags.
-    """
-    if do_flip:
-        mano_params, has_mano_params = fliplr_params(mano_params, has_mano_params)
-    mano_params['global_orient'] = rot_aa(mano_params['global_orient'], rot)
-    return mano_params, has_mano_params
-
-
-
-def get_example(img_path: str|np.ndarray, center_x: float, center_y: float,
-                width: float, height: float,
-                keypoints_2d: np.array, keypoints_3d: np.array,
-                mano_params: Dict, has_mano_params: Dict,
-                flip_kp_permutation: List[int],
-                patch_width: int, patch_height: int,
-                mean: np.array, std: np.array,
-                do_augment: bool, is_right: bool, augm_config: CfgNode,
-                is_bgr: bool = True,
-                use_skimage_antialias: bool = False,
-                border_mode: int = cv2.BORDER_CONSTANT,
-                return_trans: bool = False) -> Tuple:
-    """
-    Get an example from the dataset and (possibly) apply random augmentations.
-    Args:
-        img_path (str): Image filename
-        center_x (float): Bounding box center x coordinate in the original image.
-        center_y (float): Bounding box center y coordinate in the original image.
-        width (float): Bounding box width.
-        height (float): Bounding box height.
-        keypoints_2d (np.array): Array with shape (N,3) containing the 2D keypoints in the original image coordinates.
-        keypoints_3d (np.array): Array with shape (N,4) containing the 3D keypoints.
-        mano_params (Dict): MANO parameter annotations.
-        has_mano_params (Dict): Whether MANO annotations are valid.
-        flip_kp_permutation (List): Permutation to apply to the keypoints after flipping.
-        patch_width (float): Output box width.
-        patch_height (float): Output box height.
-        mean (np.array): Array of shape (3,) containing the mean for normalizing the input image.
-        std (np.array): Array of shape (3,) containing the std for normalizing the input image.
-        do_augment (bool): Whether to apply data augmentation or not.
-        aug_config (CfgNode): Config containing augmentation parameters.
-    Returns:
-        return img_patch, keypoints_2d, keypoints_3d, mano_params, has_mano_params, img_size
-        img_patch (np.array): Cropped image patch of shape (3, patch_height, patch_height)
-        keypoints_2d (np.array): Array with shape (N,3) containing the transformed 2D keypoints.
-        keypoints_3d (np.array): Array with shape (N,4) containing the transformed 3D keypoints.
-        mano_params (Dict): Transformed MANO parameters.
-        has_mano_params (Dict): Valid flag for transformed MANO parameters.
-        img_size (np.array): Image size of the original image.
-        """
-    if isinstance(img_path, str):
-        # 1. load image
-        cvimg = cv2.imread(img_path, cv2.IMREAD_COLOR | cv2.IMREAD_IGNORE_ORIENTATION)
-        if not isinstance(cvimg, np.ndarray):
-            raise IOError("Fail to read %s" % img_path)
-    elif isinstance(img_path, np.ndarray):
-        cvimg = img_path
-    else:
-        raise TypeError('img_path must be either a string or a numpy array')
-    img_height, img_width, img_channels = cvimg.shape
-
-    img_size = np.array([img_height, img_width])
-
-    # 2. get augmentation params
-    if do_augment:
-        scale, rot, do_flip, do_extreme_crop, extreme_crop_lvl, color_scale, tx, ty = do_augmentation(augm_config)
-    else:
-        scale, rot, do_flip, do_extreme_crop, extreme_crop_lvl, color_scale, tx, ty = 1.0, 0, False, False, 0, [1.0, 1.0, 1.0], 0., 0.
-
-    # if it's a left hand, we flip
-    if not is_right:
-        do_flip = True
-
-    if width < 1 or height < 1:
-        breakpoint()
-
-    if do_extreme_crop:
-        if extreme_crop_lvl == 0:
-            center_x1, center_y1, width1, height1 = extreme_cropping(center_x, center_y, width, height, keypoints_2d)
-        elif extreme_crop_lvl == 1:
-            center_x1, center_y1, width1, height1 = extreme_cropping_aggressive(center_x, center_y, width, height, keypoints_2d)
-
-        THRESH = 4
-        if width1 < THRESH or height1 < THRESH:
-            # print(f'{do_extreme_crop=}')
-            # print(f'width: {width}, height: {height}')
-            # print(f'width1: {width1}, height1: {height1}')
-            # print(f'center_x: {center_x}, center_y: {center_y}')
-            # print(f'center_x1: {center_x1}, center_y1: {center_y1}')
-            # print(f'keypoints_2d: {keypoints_2d}')
-            # print(f'\n\n', flush=True)
-            # breakpoint()
-            pass
-            # print(f'skip ==> width1: {width1}, height1: {height1}, width: {width}, height: {height}')
-        else:
-            center_x, center_y, width, height = center_x1, center_y1, width1, height1
-
-    center_x += width * tx
-    center_y += height * ty
-
-    # Process 3D keypoints
-    keypoints_3d = keypoint_3d_processing(keypoints_3d, flip_kp_permutation, rot, do_flip)
-
-    # 3. generate image patch
-    if use_skimage_antialias:
-        # Blur image to avoid aliasing artifacts
-        downsampling_factor = (patch_width / (width*scale))
-        if downsampling_factor > 1.1:
-            cvimg  = gaussian(cvimg, sigma=(downsampling_factor-1)/2, channel_axis=2, preserve_range=True, truncate=3.0)
-
-    img_patch_cv, trans = generate_image_patch_cv2(cvimg,
-                                                    center_x, center_y,
-                                                    width, height,
-                                                    patch_width, patch_height,
-                                                    do_flip, scale, rot, 
-                                                    border_mode=border_mode)
-
-        # img_patch_cv, trans = generate_image_patch_skimage(cvimg,
-        #                                                 center_x, center_y,
-        #                                                 width, height,
-        #                                                 patch_width, patch_height,
-        #                                                 do_flip, scale, rot, 
-        #                                                 border_mode=border_mode)
-
-    image = img_patch_cv.copy()
-    if is_bgr:
-        image = image[:, :, ::-1]
-    img_patch_cv = image.copy()
-    img_patch = convert_cvimg_to_tensor(image)
-
-
-    mano_params, has_mano_params = mano_param_processing(mano_params, has_mano_params, rot, do_flip)
-
-    # apply normalization
-    for n_c in range(min(img_channels, 3)):
-        img_patch[n_c, :, :] = np.clip(img_patch[n_c, :, :] * color_scale[n_c], 0, 255)
-        if mean is not None and std is not None:
-            img_patch[n_c, :, :] = (img_patch[n_c, :, :] - mean[n_c]) / std[n_c]
-    if do_flip:
-        keypoints_2d = fliplr_keypoints(keypoints_2d, img_width, flip_kp_permutation)
-
-
-    for n_jt in range(len(keypoints_2d)):
-        keypoints_2d[n_jt, 0:2] = trans_point2d(keypoints_2d[n_jt, 0:2], trans)
-    keypoints_2d[:, :-1] = keypoints_2d[:, :-1] / patch_width - 0.5
-
-    if not return_trans:
-        return img_patch, keypoints_2d, keypoints_3d, mano_params, has_mano_params, img_size
-    else:
-        return img_patch, keypoints_2d, keypoints_3d, mano_params, has_mano_params, img_size, trans
-
-def crop_to_hips(center_x: float, center_y: float, width: float, height: float, keypoints_2d: np.array) -> Tuple:
-    """
-    Extreme cropping: Crop the box up to the hip locations.
-    Args:
-        center_x (float): x coordinate of the bounding box center.
-        center_y (float): y coordinate of the bounding box center.
-        width (float): Bounding box width.
-        height (float): Bounding box height.
-        keypoints_2d (np.array): Array of shape (N, 3) containing 2D keypoint locations.
-    Returns:
-        center_x (float): x coordinate of the new bounding box center.
-        center_y (float): y coordinate of the new bounding box center.
-        width (float): New bounding box width.
-        height (float): New bounding box height.
-    """
-    keypoints_2d = keypoints_2d.copy()
-    lower_body_keypoints = [10, 11, 13, 14, 19, 20, 21, 22, 23, 24, 25+0, 25+1, 25+4, 25+5]
-    keypoints_2d[lower_body_keypoints, :] = 0
-    if keypoints_2d[:, -1].sum() > 1:
-        center, scale = get_bbox(keypoints_2d)
-        center_x = center[0]
-        center_y = center[1]
-        width = 1.1 * scale[0]
-        height = 1.1 * scale[1]
-    return center_x, center_y, width, height
-
-
-def crop_to_shoulders(center_x: float, center_y: float, width: float, height: float, keypoints_2d: np.array):
-    """
-    Extreme cropping: Crop the box up to the shoulder locations.
-    Args:
-        center_x (float): x coordinate of the bounding box center.
-        center_y (float): y coordinate of the bounding box center.
-        width (float): Bounding box width.
-        height (float): Bounding box height.
-        keypoints_2d (np.array): Array of shape (N, 3) containing 2D keypoint locations.
-    Returns:
-        center_x (float): x coordinate of the new bounding box center.
-        center_y (float): y coordinate of the new bounding box center.
-        width (float): New bounding box width.
-        height (float): New bounding box height.
-    """
-    keypoints_2d = keypoints_2d.copy()
-    lower_body_keypoints = [3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 19, 20, 21, 22, 23, 24] + [25 + i for i in [0, 1, 2, 3, 4, 5, 6, 7, 10, 11, 14, 15, 16]]
-    keypoints_2d[lower_body_keypoints, :] = 0
-    center, scale = get_bbox(keypoints_2d)
-    if keypoints_2d[:, -1].sum() > 1:
-        center, scale = get_bbox(keypoints_2d)
-        center_x = center[0]
-        center_y = center[1]
-        width = 1.2 * scale[0]
-        height = 1.2 * scale[1]
-    return center_x, center_y, width, height
-
-def crop_to_head(center_x: float, center_y: float, width: float, height: float, keypoints_2d: np.array):
-    """
-    Extreme cropping: Crop the box and keep on only the head.
-    Args:
-        center_x (float): x coordinate of the bounding box center.
-        center_y (float): y coordinate of the bounding box center.
-        width (float): Bounding box width.
-        height (float): Bounding box height.
-        keypoints_2d (np.array): Array of shape (N, 3) containing 2D keypoint locations.
-    Returns:
-        center_x (float): x coordinate of the new bounding box center.
-        center_y (float): y coordinate of the new bounding box center.
-        width (float): New bounding box width.
-        height (float): New bounding box height.
-    """
-    keypoints_2d = keypoints_2d.copy()
-    lower_body_keypoints = [3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 19, 20, 21, 22, 23, 24] + [25 + i for i in [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 14, 15, 16]]
-    keypoints_2d[lower_body_keypoints, :] = 0
-    if keypoints_2d[:, -1].sum() > 1:
-        center, scale = get_bbox(keypoints_2d)
-        center_x = center[0]
-        center_y = center[1]
-        width = 1.3 * scale[0]
-        height = 1.3 * scale[1]
-    return center_x, center_y, width, height
-
-def crop_torso_only(center_x: float, center_y: float, width: float, height: float, keypoints_2d: np.array):
-    """
-    Extreme cropping: Crop the box and keep on only the torso.
-    Args:
-        center_x (float): x coordinate of the bounding box center.
-        center_y (float): y coordinate of the bounding box center.
-        width (float): Bounding box width.
-        height (float): Bounding box height.
-        keypoints_2d (np.array): Array of shape (N, 3) containing 2D keypoint locations.
-    Returns:
-        center_x (float): x coordinate of the new bounding box center.
-        center_y (float): y coordinate of the new bounding box center.
-        width (float): New bounding box width.
-        height (float): New bounding box height.
-    """
-    keypoints_2d = keypoints_2d.copy()
-    nontorso_body_keypoints = [0, 3, 4, 6, 7, 10, 11, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24] + [25 + i for i in [0, 1, 4, 5, 6, 7, 10, 11, 13, 17, 18]]
-    keypoints_2d[nontorso_body_keypoints, :] = 0
-    if keypoints_2d[:, -1].sum() > 1:
-        center, scale = get_bbox(keypoints_2d)
-        center_x = center[0]
-        center_y = center[1]
-        width = 1.1 * scale[0]
-        height = 1.1 * scale[1]
-    return center_x, center_y, width, height
-
-def crop_rightarm_only(center_x: float, center_y: float, width: float, height: float, keypoints_2d: np.array):
-    """
-    Extreme cropping: Crop the box and keep on only the right arm.
-    Args:
-        center_x (float): x coordinate of the bounding box center.
-        center_y (float): y coordinate of the bounding box center.
-        width (float): Bounding box width.
-        height (float): Bounding box height.
-        keypoints_2d (np.array): Array of shape (N, 3) containing 2D keypoint locations.
-    Returns:
-        center_x (float): x coordinate of the new bounding box center.
-        center_y (float): y coordinate of the new bounding box center.
-        width (float): New bounding box width.
-        height (float): New bounding box height.
-    """
-    keypoints_2d = keypoints_2d.copy()
-    nonrightarm_body_keypoints = [0, 1, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24] + [25 + i for i in [0, 1, 2, 3, 4, 5, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18]]
-    keypoints_2d[nonrightarm_body_keypoints, :] = 0
-    if keypoints_2d[:, -1].sum() > 1:
-        center, scale = get_bbox(keypoints_2d)
-        center_x = center[0]
-        center_y = center[1]
-        width = 1.1 * scale[0]
-        height = 1.1 * scale[1]
-    return center_x, center_y, width, height
-
-def crop_leftarm_only(center_x: float, center_y: float, width: float, height: float, keypoints_2d: np.array):
-    """
-    Extreme cropping: Crop the box and keep on only the left arm.
-    Args:
-        center_x (float): x coordinate of the bounding box center.
-        center_y (float): y coordinate of the bounding box center.
-        width (float): Bounding box width.
-        height (float): Bounding box height.
-        keypoints_2d (np.array): Array of shape (N, 3) containing 2D keypoint locations.
-    Returns:
-        center_x (float): x coordinate of the new bounding box center.
-        center_y (float): y coordinate of the new bounding box center.
-        width (float): New bounding box width.
-        height (float): New bounding box height.
-    """
-    keypoints_2d = keypoints_2d.copy()
-    nonleftarm_body_keypoints = [0, 1, 2, 3, 4, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24] + [25 + i for i in [0, 1, 2, 3, 4, 5, 6, 7, 8, 12, 13, 14, 15, 16, 17, 18]]
-    keypoints_2d[nonleftarm_body_keypoints, :] = 0
-    if keypoints_2d[:, -1].sum() > 1:
-        center, scale = get_bbox(keypoints_2d)
-        center_x = center[0]
-        center_y = center[1]
-        width = 1.1 * scale[0]
-        height = 1.1 * scale[1]
-    return center_x, center_y, width, height
-
-def crop_legs_only(center_x: float, center_y: float, width: float, height: float, keypoints_2d: np.array):
-    """
-    Extreme cropping: Crop the box and keep on only the legs.
-    Args:
-        center_x (float): x coordinate of the bounding box center.
-        center_y (float): y coordinate of the bounding box center.
-        width (float): Bounding box width.
-        height (float): Bounding box height.
-        keypoints_2d (np.array): Array of shape (N, 3) containing 2D keypoint locations.
-    Returns:
-        center_x (float): x coordinate of the new bounding box center.
-        center_y (float): y coordinate of the new bounding box center.
-        width (float): New bounding box width.
-        height (float): New bounding box height.
-    """
-    keypoints_2d = keypoints_2d.copy()
-    nonlegs_body_keypoints = [0, 1, 2, 3, 4, 5, 6, 7, 15, 16, 17, 18] + [25 + i for i in [6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 17, 18]]
-    keypoints_2d[nonlegs_body_keypoints, :] = 0
-    if keypoints_2d[:, -1].sum() > 1:
-        center, scale = get_bbox(keypoints_2d)
-        center_x = center[0]
-        center_y = center[1]
-        width = 1.1 * scale[0]
-        height = 1.1 * scale[1]
-    return center_x, center_y, width, height
-
-def crop_rightleg_only(center_x: float, center_y: float, width: float, height: float, keypoints_2d: np.array):
-    """
-    Extreme cropping: Crop the box and keep on only the right leg.
-    Args:
-        center_x (float): x coordinate of the bounding box center.
-        center_y (float): y coordinate of the bounding box center.
-        width (float): Bounding box width.
-        height (float): Bounding box height.
-        keypoints_2d (np.array): Array of shape (N, 3) containing 2D keypoint locations.
-    Returns:
-        center_x (float): x coordinate of the new bounding box center.
-        center_y (float): y coordinate of the new bounding box center.
-        width (float): New bounding box width.
-        height (float): New bounding box height.
-    """
-    keypoints_2d = keypoints_2d.copy()
-    nonrightleg_body_keypoints = [0, 1, 2, 3, 4, 5, 6, 7, 8, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21] + [25 + i for i in [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18]]
-    keypoints_2d[nonrightleg_body_keypoints, :] = 0
-    if keypoints_2d[:, -1].sum() > 1:
-        center, scale = get_bbox(keypoints_2d)
-        center_x = center[0]
-        center_y = center[1]
-        width = 1.1 * scale[0]
-        height = 1.1 * scale[1]
-    return center_x, center_y, width, height
-
-def crop_leftleg_only(center_x: float, center_y: float, width: float, height: float, keypoints_2d: np.array):
-    """
-    Extreme cropping: Crop the box and keep on only the left leg.
-    Args:
-        center_x (float): x coordinate of the bounding box center.
-        center_y (float): y coordinate of the bounding box center.
-        width (float): Bounding box width.
-        height (float): Bounding box height.
-        keypoints_2d (np.array): Array of shape (N, 3) containing 2D keypoint locations.
-    Returns:
-        center_x (float): x coordinate of the new bounding box center.
-        center_y (float): y coordinate of the new bounding box center.
-        width (float): New bounding box width.
-        height (float): New bounding box height.
-    """
-    keypoints_2d = keypoints_2d.copy()
-    nonleftleg_body_keypoints = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 15, 16, 17, 18, 22, 23, 24] + [25 + i for i in [0, 1, 2, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18]]
-    keypoints_2d[nonleftleg_body_keypoints, :] = 0
-    if keypoints_2d[:, -1].sum() > 1:
-        center, scale = get_bbox(keypoints_2d)
-        center_x = center[0]
-        center_y = center[1]
-        width = 1.1 * scale[0]
-        height = 1.1 * scale[1]
-    return center_x, center_y, width, height
-
-def full_body(keypoints_2d: np.array) -> bool:
-    """
-    Check if all main body joints are visible.
-    Args:
-        keypoints_2d (np.array): Array of shape (N, 3) containing 2D keypoint locations.
-    Returns:
-        bool: True if all main body joints are visible.
-    """
-
-    body_keypoints_openpose = [2, 3, 4, 5, 6, 7, 10, 11, 13, 14]
-    body_keypoints = [25 + i for i in [8, 7, 6, 9, 10, 11, 1, 0, 4, 5]]
-    return (np.maximum(keypoints_2d[body_keypoints, -1], keypoints_2d[body_keypoints_openpose, -1]) > 0).sum() == len(body_keypoints)
-
-def upper_body(keypoints_2d: np.array):
-    """
-    Check if all upper body joints are visible.
-    Args:
-        keypoints_2d (np.array): Array of shape (N, 3) containing 2D keypoint locations.
-    Returns:
-        bool: True if all main body joints are visible.
-    """
-    lower_body_keypoints_openpose = [10, 11, 13, 14]
-    lower_body_keypoints = [25 + i for i in [1, 0, 4, 5]]
-    upper_body_keypoints_openpose = [0, 1, 15, 16, 17, 18]
-    upper_body_keypoints = [25+8, 25+9, 25+12, 25+13, 25+17, 25+18]
-    return ((keypoints_2d[lower_body_keypoints + lower_body_keypoints_openpose, -1] > 0).sum() == 0)\
-       and ((keypoints_2d[upper_body_keypoints + upper_body_keypoints_openpose, -1] > 0).sum() >= 2)
-
-def get_bbox(keypoints_2d: np.array, rescale: float = 1.2) -> Tuple:
-    """
-    Get center and scale for bounding box from openpose detections.
-    Args:
-        keypoints_2d (np.array): Array of shape (N, 3) containing 2D keypoint locations.
-        rescale (float): Scale factor to rescale bounding boxes computed from the keypoints.
-    Returns:
-        center (np.array): Array of shape (2,) containing the new bounding box center.
-        scale (float): New bounding box scale.
-    """
-    valid = keypoints_2d[:,-1] > 0
-    valid_keypoints = keypoints_2d[valid][:,:-1]
-    center = 0.5 * (valid_keypoints.max(axis=0) + valid_keypoints.min(axis=0))
-    bbox_size = (valid_keypoints.max(axis=0) - valid_keypoints.min(axis=0))
-    # adjust bounding box tightness
-    scale = bbox_size
-    scale *= rescale
-    return center, scale
-
-def extreme_cropping(center_x: float, center_y: float, width: float, height: float, keypoints_2d: np.array) -> Tuple:
-    """
-    Perform extreme cropping
-    Args:
-        center_x (float): x coordinate of bounding box center.
-        center_y (float): y coordinate of bounding box center.
-        width (float): bounding box width.
-        height (float): bounding box height.
-        keypoints_2d (np.array): Array of shape (N, 3) containing 2D keypoint locations.
-        rescale (float): Scale factor to rescale bounding boxes computed from the keypoints.
-    Returns:
-        center_x (float): x coordinate of bounding box center.
-        center_y (float): y coordinate of bounding box center.
-        width (float): bounding box width.
-        height (float): bounding box height.
-    """
-    p = torch.rand(1).item()
-    if full_body(keypoints_2d):
-        if p < 0.7:
-            center_x, center_y, width, height = crop_to_hips(center_x, center_y, width, height, keypoints_2d)
-        elif p < 0.9:
-            center_x, center_y, width, height = crop_to_shoulders(center_x, center_y, width, height, keypoints_2d)
-        else:
-            center_x, center_y, width, height = crop_to_head(center_x, center_y, width, height, keypoints_2d)
-    elif upper_body(keypoints_2d):
-        if p < 0.9:
-            center_x, center_y, width, height = crop_to_shoulders(center_x, center_y, width, height, keypoints_2d)
-        else:
-            center_x, center_y, width, height = crop_to_head(center_x, center_y, width, height, keypoints_2d)
-
-    return center_x, center_y, max(width, height), max(width, height)
-
-def extreme_cropping_aggressive(center_x: float, center_y: float, width: float, height: float, keypoints_2d: np.array) -> Tuple:
-    """
-    Perform aggressive extreme cropping
-    Args:
-        center_x (float): x coordinate of bounding box center.
-        center_y (float): y coordinate of bounding box center.
-        width (float): bounding box width.
-        height (float): bounding box height.
-        keypoints_2d (np.array): Array of shape (N, 3) containing 2D keypoint locations.
-        rescale (float): Scale factor to rescale bounding boxes computed from the keypoints.
-    Returns:
-        center_x (float): x coordinate of bounding box center.
-        center_y (float): y coordinate of bounding box center.
-        width (float): bounding box width.
-        height (float): bounding box height.
-    """
-    p = torch.rand(1).item()
-    if full_body(keypoints_2d):
-        if p < 0.2:
-            center_x, center_y, width, height = crop_to_hips(center_x, center_y, width, height, keypoints_2d)
-        elif p < 0.3:
-            center_x, center_y, width, height = crop_to_shoulders(center_x, center_y, width, height, keypoints_2d)
-        elif p < 0.4:
-            center_x, center_y, width, height = crop_to_head(center_x, center_y, width, height, keypoints_2d)
-        elif p < 0.5:
-            center_x, center_y, width, height = crop_torso_only(center_x, center_y, width, height, keypoints_2d)
-        elif p < 0.6:
-            center_x, center_y, width, height = crop_rightarm_only(center_x, center_y, width, height, keypoints_2d)
-        elif p < 0.7:
-            center_x, center_y, width, height = crop_leftarm_only(center_x, center_y, width, height, keypoints_2d)
-        elif p < 0.8:
-            center_x, center_y, width, height = crop_legs_only(center_x, center_y, width, height, keypoints_2d)
-        elif p < 0.9:
-            center_x, center_y, width, height = crop_rightleg_only(center_x, center_y, width, height, keypoints_2d)
-        else:
-            center_x, center_y, width, height = crop_leftleg_only(center_x, center_y, width, height, keypoints_2d)
-    elif upper_body(keypoints_2d):
-        if p < 0.2:
-            center_x, center_y, width, height = crop_to_shoulders(center_x, center_y, width, height, keypoints_2d)
-        elif p < 0.4:
-            center_x, center_y, width, height = crop_to_head(center_x, center_y, width, height, keypoints_2d)
-        elif p < 0.6:
-            center_x, center_y, width, height = crop_torso_only(center_x, center_y, width, height, keypoints_2d)
-        elif p < 0.8:
-            center_x, center_y, width, height = crop_rightarm_only(center_x, center_y, width, height, keypoints_2d)
-        else:
-            center_x, center_y, width, height = crop_leftarm_only(center_x, center_y, width, height, keypoints_2d)
-    return center_x, center_y, max(width, height), max(width, height)

WiLoR/wilor/datasets/vitdet_dataset.py

@@ -1,95 +0,0 @@
-from typing import Dict
-
-import cv2
-import numpy as np
-from skimage.filters import gaussian
-from yacs.config import CfgNode
-import torch
-
-from .utils import (convert_cvimg_to_tensor,
-                    expand_to_aspect_ratio,
-                    generate_image_patch_cv2)
-
-DEFAULT_MEAN = 255. * np.array([0.485, 0.456, 0.406])
-DEFAULT_STD = 255. * np.array([0.229, 0.224, 0.225])
-
-class ViTDetDataset(torch.utils.data.Dataset):
-
-    def __init__(self,
-                 cfg: CfgNode,
-                 img_cv2: np.array,
-                 boxes: np.array,
-                 right: np.array,
-                 rescale_factor=2.5,
-                 train: bool = False,
-                 **kwargs):
-        super().__init__()
-        self.cfg = cfg
-        self.img_cv2 = img_cv2
-        # self.boxes = boxes
-
-        assert train == False, "ViTDetDataset is only for inference"
-        self.train = train
-        self.img_size = cfg.MODEL.IMAGE_SIZE
-        self.mean = 255. * np.array(self.cfg.MODEL.IMAGE_MEAN)
-        self.std = 255. * np.array(self.cfg.MODEL.IMAGE_STD)
-
-        # Preprocess annotations
-        boxes = boxes.astype(np.float32)
-        self.center = (boxes[:, 2:4] + boxes[:, 0:2]) / 2.0
-        self.scale = rescale_factor * (boxes[:, 2:4] - boxes[:, 0:2]) / 200.0
-        self.personid = np.arange(len(boxes), dtype=np.int32)
-        self.right = right.astype(np.float32)
-
-    def __len__(self) -> int:
-        return len(self.personid)
-
-    def __getitem__(self, idx: int) -> Dict[str, np.array]:
-
-        center = self.center[idx].copy()
-        center_x = center[0]
-        center_y = center[1]
-
-        scale = self.scale[idx]
-        BBOX_SHAPE = self.cfg.MODEL.get('BBOX_SHAPE', None)
-        bbox_size = expand_to_aspect_ratio(scale*200, target_aspect_ratio=BBOX_SHAPE).max()
-
-        patch_width = patch_height = self.img_size
-
-        right = self.right[idx].copy()
-        flip = right == 0
-
-        # 3. generate image patch
-        # if use_skimage_antialias:
-        cvimg = self.img_cv2.copy()
-        if True:
-            # Blur image to avoid aliasing artifacts
-            downsampling_factor = ((bbox_size*1.0) / patch_width)
-            #print(f'{downsampling_factor=}')
-            downsampling_factor = downsampling_factor / 2.0
-            if downsampling_factor > 1.1:
-                cvimg  = gaussian(cvimg, sigma=(downsampling_factor-1)/2, channel_axis=2, preserve_range=True)
-
-
-        img_patch_cv, trans = generate_image_patch_cv2(cvimg,
-                                                    center_x, center_y,
-                                                    bbox_size, bbox_size,
-                                                    patch_width, patch_height,
-                                                    flip, 1.0, 0,
-                                                    border_mode=cv2.BORDER_CONSTANT)
-        img_patch_cv = img_patch_cv[:, :, ::-1]
-        img_patch = convert_cvimg_to_tensor(img_patch_cv)
-
-        # apply normalization
-        for n_c in range(min(self.img_cv2.shape[2], 3)):
-            img_patch[n_c, :, :] = (img_patch[n_c, :, :] - self.mean[n_c]) / self.std[n_c]
-
-        item = {
-            'img': img_patch,
-            'personid': int(self.personid[idx]),
-        }
-        item['box_center'] = self.center[idx].copy()
-        item['box_size'] = bbox_size
-        item['img_size'] = 1.0 * np.array([cvimg.shape[1], cvimg.shape[0]])
-        item['right'] = self.right[idx].copy()
-        return item

WiLoR/wilor/models/__init__.py

@@ -1,36 +0,0 @@
-from .mano_wrapper import MANO
-from .wilor import WiLoR
-
-from .discriminator import Discriminator
-
-def load_wilor(checkpoint_path, cfg_path):
-    from pathlib import Path
-    from wilor.configs import get_config
-    print('Loading ', checkpoint_path)
-    model_cfg = get_config(cfg_path, update_cachedir=True)
-
-    # Override some config values, to crop bbox correctly
-    if ('vit' in model_cfg.MODEL.BACKBONE.TYPE) and ('BBOX_SHAPE' not in model_cfg.MODEL):
-        
-        model_cfg.defrost()
-        assert model_cfg.MODEL.IMAGE_SIZE == 256, f"MODEL.IMAGE_SIZE ({model_cfg.MODEL.IMAGE_SIZE}) should be 256 for ViT backbone"
-        model_cfg.MODEL.BBOX_SHAPE = [192,256]
-        model_cfg.freeze()
-
-    # Update config to be compatible with demo
-    if ('PRETRAINED_WEIGHTS' in model_cfg.MODEL.BACKBONE):
-        model_cfg.defrost()
-        model_cfg.MODEL.BACKBONE.pop('PRETRAINED_WEIGHTS')
-        model_cfg.freeze()
-        
-        # Update config to be compatible with demo
-
-    if ('DATA_DIR' in model_cfg.MANO):
-        model_cfg.defrost()
-        model_cfg.MANO.DATA_DIR    = './mano_data/'
-        model_cfg.MANO.MODEL_PATH  = './mano_data/'
-        model_cfg.MANO.MEAN_PARAMS = './mano_data/mano_mean_params.npz'
-        model_cfg.freeze()
-
-    model = WiLoR.load_from_checkpoint(checkpoint_path, strict=False, cfg=model_cfg)
-    return model, model_cfg

WiLoR/wilor/models/backbones/__init__.py

@@ -1,17 +0,0 @@
-from .vit import vit
-
-def create_backbone(cfg):
-    if cfg.MODEL.BACKBONE.TYPE == 'vit':
-        return vit(cfg)
-    elif cfg.MODEL.BACKBONE.TYPE == 'fast_vit':
-        import torch 
-        import sys 
-        from timm.models import create_model
-        #from models.modules.mobileone import reparameterize_model
-        fast_vit = create_model("fastvit_ma36", drop_path_rate=0.2)
-        checkpoint = torch.load('./pretrained_models/fastvit_ma36.pt')
-        fast_vit.load_state_dict(checkpoint['state_dict'])
-        return fast_vit
-        
-    else:
-        raise NotImplementedError('Backbone type is not implemented')

WiLoR/wilor/models/backbones/vit.py

@@ -1,410 +0,0 @@
-# Copyright (c) OpenMMLab. All rights reserved.
-import math
-import numpy as np
-import torch
-from functools import partial
-import torch.nn as nn
-import torch.nn.functional as F
-import torch.utils.checkpoint as checkpoint
-from ...utils.geometry import rot6d_to_rotmat, aa_to_rotmat
-from timm.models.layers import drop_path, to_2tuple, trunc_normal_
-
-def vit(cfg):
-    return ViT(
-                img_size=(256, 192),
-                patch_size=16,
-                embed_dim=1280,
-                depth=32,
-                num_heads=16,
-                ratio=1,
-                use_checkpoint=False,
-                mlp_ratio=4,
-                qkv_bias=True,
-                drop_path_rate=0.55,
-                cfg = cfg
-            )
-
-def get_abs_pos(abs_pos, h, w, ori_h, ori_w, has_cls_token=True):
-    """
-    Calculate absolute positional embeddings. If needed, resize embeddings and remove cls_token
-        dimension for the original embeddings.
-    Args:
-        abs_pos (Tensor): absolute positional embeddings with (1, num_position, C).
-        has_cls_token (bool): If true, has 1 embedding in abs_pos for cls token.
-        hw (Tuple): size of input image tokens.
-
-    Returns:
-        Absolute positional embeddings after processing with shape (1, H, W, C)
-    """
-    cls_token = None
-    B, L, C = abs_pos.shape
-    if has_cls_token:
-        cls_token = abs_pos[:, 0:1]
-        abs_pos = abs_pos[:, 1:]
-
-    if ori_h != h or ori_w != w:
-        new_abs_pos = F.interpolate(
-            abs_pos.reshape(1, ori_h, ori_w, -1).permute(0, 3, 1, 2),
-            size=(h, w),
-            mode="bicubic",
-            align_corners=False,
-        ).permute(0, 2, 3, 1).reshape(B, -1, C)
-
-    else:
-        new_abs_pos = abs_pos
-    
-    if cls_token is not None:
-        new_abs_pos = torch.cat([cls_token, new_abs_pos], dim=1)
-    return new_abs_pos
-
-class DropPath(nn.Module):
-    """Drop paths (Stochastic Depth) per sample  (when applied in main path of residual blocks).
-    """
-    def __init__(self, drop_prob=None):
-        super(DropPath, self).__init__()
-        self.drop_prob = drop_prob
-
-    def forward(self, x):
-        return drop_path(x, self.drop_prob, self.training)
-    
-    def extra_repr(self):
-        return 'p={}'.format(self.drop_prob)
-
-class Mlp(nn.Module):
-    def __init__(self, in_features, hidden_features=None, out_features=None, act_layer=nn.GELU, drop=0.):
-        super().__init__()
-        out_features = out_features or in_features
-        hidden_features = hidden_features or in_features
-        self.fc1 = nn.Linear(in_features, hidden_features)
-        self.act = act_layer()
-        self.fc2 = nn.Linear(hidden_features, out_features)
-        self.drop = nn.Dropout(drop)
-
-    def forward(self, x):
-        x = self.fc1(x)
-        x = self.act(x)
-        x = self.fc2(x)
-        x = self.drop(x)
-        return x
-
-class Attention(nn.Module):
-    def __init__(
-            self, dim, num_heads=8, qkv_bias=False, qk_scale=None, attn_drop=0.,
-            proj_drop=0., attn_head_dim=None,):
-        super().__init__()
-        self.num_heads = num_heads
-        head_dim = dim // num_heads
-        self.dim = dim
-
-        if attn_head_dim is not None:
-            head_dim = attn_head_dim
-        all_head_dim = head_dim * self.num_heads
-
-        self.scale = qk_scale or head_dim ** -0.5
-
-        self.qkv = nn.Linear(dim, all_head_dim * 3, bias=qkv_bias)
-
-        self.attn_drop = nn.Dropout(attn_drop)
-        self.proj = nn.Linear(all_head_dim, dim)
-        self.proj_drop = nn.Dropout(proj_drop)
-
-    def forward(self, x):
-        B, N, C = x.shape
-        qkv = self.qkv(x)
-        qkv = qkv.reshape(B, N, 3, self.num_heads, -1).permute(2, 0, 3, 1, 4)
-        q, k, v = qkv[0], qkv[1], qkv[2]   # make torchscript happy (cannot use tensor as tuple)
-
-        q = q * self.scale
-        attn = (q @ k.transpose(-2, -1))
-
-        attn = attn.softmax(dim=-1)
-        attn = self.attn_drop(attn)
-
-        x = (attn @ v).transpose(1, 2).reshape(B, N, -1)
-        x = self.proj(x)
-        x = self.proj_drop(x)
-
-        return x
-
-class Block(nn.Module):
-
-    def __init__(self, dim, num_heads, mlp_ratio=4., qkv_bias=False, qk_scale=None, 
-                 drop=0., attn_drop=0., drop_path=0., act_layer=nn.GELU, 
-                 norm_layer=nn.LayerNorm, attn_head_dim=None
-                 ):
-        super().__init__()
-        
-        self.norm1 = norm_layer(dim)
-        self.attn = Attention(
-            dim, num_heads=num_heads, qkv_bias=qkv_bias, qk_scale=qk_scale,
-            attn_drop=attn_drop, proj_drop=drop, attn_head_dim=attn_head_dim
-            )
-
-        # NOTE: drop path for stochastic depth, we shall see if this is better than dropout here
-        self.drop_path = DropPath(drop_path) if drop_path > 0. else nn.Identity()
-        self.norm2 = norm_layer(dim)
-        mlp_hidden_dim = int(dim * mlp_ratio)
-        self.mlp = Mlp(in_features=dim, hidden_features=mlp_hidden_dim, act_layer=act_layer, drop=drop)
-
-    def forward(self, x):
-        x = x + self.drop_path(self.attn(self.norm1(x)))
-        x = x + self.drop_path(self.mlp(self.norm2(x)))
-        return x
-
-
-class PatchEmbed(nn.Module):
-    """ Image to Patch Embedding
-    """
-    def __init__(self, img_size=224, patch_size=16, in_chans=3, embed_dim=768, ratio=1):
-        super().__init__()
-        img_size = to_2tuple(img_size)
-        patch_size = to_2tuple(patch_size)
-        num_patches = (img_size[1] // patch_size[1]) * (img_size[0] // patch_size[0]) * (ratio ** 2)
-        self.patch_shape = (int(img_size[0] // patch_size[0] * ratio), int(img_size[1] // patch_size[1] * ratio))
-        self.origin_patch_shape = (int(img_size[0] // patch_size[0]), int(img_size[1] // patch_size[1]))
-        self.img_size = img_size
-        self.patch_size = patch_size
-        self.num_patches = num_patches
-
-        self.proj = nn.Conv2d(in_chans, embed_dim, kernel_size=patch_size, stride=(patch_size[0] // ratio), padding=4 + 2 * (ratio//2-1))
-
-    def forward(self, x, **kwargs):
-        B, C, H, W = x.shape
-        x = self.proj(x)
-        Hp, Wp = x.shape[2], x.shape[3]
-
-        x = x.flatten(2).transpose(1, 2)
-        return x, (Hp, Wp)
-
-
-class HybridEmbed(nn.Module):
-    """ CNN Feature Map Embedding
-    Extract feature map from CNN, flatten, project to embedding dim.
-    """
-    def __init__(self, backbone, img_size=224, feature_size=None, in_chans=3, embed_dim=768):
-        super().__init__()
-        assert isinstance(backbone, nn.Module)
-        img_size = to_2tuple(img_size)
-        self.img_size = img_size
-        self.backbone = backbone
-        if feature_size is None:
-            with torch.no_grad():
-                training = backbone.training
-                if training:
-                    backbone.eval()
-                o = self.backbone(torch.zeros(1, in_chans, img_size[0], img_size[1]))[-1]
-                feature_size = o.shape[-2:]
-                feature_dim = o.shape[1]
-                backbone.train(training)
-        else:
-            feature_size = to_2tuple(feature_size)
-            feature_dim = self.backbone.feature_info.channels()[-1]
-        self.num_patches = feature_size[0] * feature_size[1]
-        self.proj = nn.Linear(feature_dim, embed_dim)
-
-    def forward(self, x):
-        x = self.backbone(x)[-1]
-        x = x.flatten(2).transpose(1, 2)
-        x = self.proj(x)
-        return x
-
-
-class ViT(nn.Module):
-
-    def __init__(self,
-                 img_size=224, patch_size=16, in_chans=3, num_classes=80, embed_dim=768, depth=12,
-                 num_heads=12, mlp_ratio=4., qkv_bias=False, qk_scale=None, drop_rate=0., attn_drop_rate=0.,
-                 drop_path_rate=0., hybrid_backbone=None, norm_layer=None, use_checkpoint=False, 
-                 frozen_stages=-1, ratio=1, last_norm=True,
-                 patch_padding='pad', freeze_attn=False, freeze_ffn=False,cfg=None, 
-                 ):
-        # Protect mutable default arguments
-        super(ViT, self).__init__()
-        norm_layer = norm_layer or partial(nn.LayerNorm, eps=1e-6)
-        self.num_classes = num_classes
-        self.num_features = self.embed_dim = embed_dim  # num_features for consistency with other models
-        self.frozen_stages = frozen_stages
-        self.use_checkpoint = use_checkpoint
-        self.patch_padding = patch_padding
-        self.freeze_attn = freeze_attn
-        self.freeze_ffn = freeze_ffn
-        self.depth = depth
-
-        if hybrid_backbone is not None:
-            self.patch_embed = HybridEmbed(
-                hybrid_backbone, img_size=img_size, in_chans=in_chans, embed_dim=embed_dim)
-        else:
-            self.patch_embed = PatchEmbed(
-                img_size=img_size, patch_size=patch_size, in_chans=in_chans, embed_dim=embed_dim, ratio=ratio)
-        num_patches = self.patch_embed.num_patches
-        
-        ##########################################
-        self.cfg = cfg
-        self.joint_rep_type = cfg.MODEL.MANO_HEAD.get('JOINT_REP', '6d')
-        self.joint_rep_dim = {'6d': 6, 'aa': 3}[self.joint_rep_type]
-        npose = self.joint_rep_dim * (cfg.MANO.NUM_HAND_JOINTS + 1)
-        self.npose = npose
-        mean_params = np.load(cfg.MANO.MEAN_PARAMS)
-        init_cam = torch.from_numpy(mean_params['cam'].astype(np.float32)).unsqueeze(0)
-        self.register_buffer('init_cam', init_cam)
-        init_hand_pose = torch.from_numpy(mean_params['pose'].astype(np.float32)).unsqueeze(0)
-        init_betas = torch.from_numpy(mean_params['shape'].astype('float32')).unsqueeze(0)
-        self.register_buffer('init_hand_pose', init_hand_pose)
-        self.register_buffer('init_betas', init_betas)
-        
-        self.pose_emb  = nn.Linear(self.joint_rep_dim  , embed_dim)
-        self.shape_emb = nn.Linear(10  , embed_dim)
-        self.cam_emb   = nn.Linear(3 , embed_dim)
-        
-        self.decpose  = nn.Linear(self.num_features, 6)
-        self.decshape = nn.Linear(self.num_features, 10)
-        self.deccam   = nn.Linear(self.num_features, 3)
-        if cfg.MODEL.MANO_HEAD.get('INIT_DECODER_XAVIER', False):
-            # True by default in MLP. False by default in Transformer
-            nn.init.xavier_uniform_(self.decpose.weight, gain=0.01)
-            nn.init.xavier_uniform_(self.decshape.weight, gain=0.01)
-            nn.init.xavier_uniform_(self.deccam.weight, gain=0.01)
-
-            
-        ##########################################
-        
-        # since the pretraining model has class token
-        self.pos_embed = nn.Parameter(torch.zeros(1, num_patches + 1, embed_dim))
-
-        dpr = [x.item() for x in torch.linspace(0, drop_path_rate, depth)]  # stochastic depth decay rule
-
-        self.blocks = nn.ModuleList([
-            Block(
-                dim=embed_dim, num_heads=num_heads, mlp_ratio=mlp_ratio, qkv_bias=qkv_bias, qk_scale=qk_scale,
-                drop=drop_rate, attn_drop=attn_drop_rate, drop_path=dpr[i], norm_layer=norm_layer,
-                )
-            for i in range(depth)])
-
-        self.last_norm = norm_layer(embed_dim) if last_norm else nn.Identity()
-
-        if self.pos_embed is not None:
-            trunc_normal_(self.pos_embed, std=.02)
-
-        self._freeze_stages()
-
-    def _freeze_stages(self):
-        """Freeze parameters."""
-        if self.frozen_stages >= 0:
-            self.patch_embed.eval()
-            for param in self.patch_embed.parameters():
-                param.requires_grad = False
-
-        for i in range(1, self.frozen_stages + 1):
-            m = self.blocks[i]
-            m.eval()
-            for param in m.parameters():
-                param.requires_grad = False
-
-        if self.freeze_attn:
-            for i in range(0, self.depth):
-                m = self.blocks[i]
-                m.attn.eval()
-                m.norm1.eval()
-                for param in m.attn.parameters():
-                    param.requires_grad = False
-                for param in m.norm1.parameters():
-                    param.requires_grad = False
-
-        if self.freeze_ffn:
-            self.pos_embed.requires_grad = False
-            self.patch_embed.eval()
-            for param in self.patch_embed.parameters():
-                param.requires_grad = False
-            for i in range(0, self.depth):
-                m = self.blocks[i]
-                m.mlp.eval()
-                m.norm2.eval()
-                for param in m.mlp.parameters():
-                    param.requires_grad = False
-                for param in m.norm2.parameters():
-                    param.requires_grad = False
-
-    def init_weights(self):
-        """Initialize the weights in backbone.
-        Args:
-            pretrained (str, optional): Path to pre-trained weights.
-                Defaults to None.
-        """
-        def _init_weights(m):
-            if isinstance(m, nn.Linear):
-                trunc_normal_(m.weight, std=.02)
-                if isinstance(m, nn.Linear) and m.bias is not None:
-                    nn.init.constant_(m.bias, 0)
-            elif isinstance(m, nn.LayerNorm):
-                nn.init.constant_(m.bias, 0)
-                nn.init.constant_(m.weight, 1.0)
-
-        self.apply(_init_weights)
-
-    def get_num_layers(self):
-        return len(self.blocks)
-
-    @torch.jit.ignore
-    def no_weight_decay(self):
-        return {'pos_embed', 'cls_token'}
-
-    def forward_features(self, x):
-        B, C, H, W = x.shape
-        x, (Hp, Wp) = self.patch_embed(x)
-  
-        if self.pos_embed is not None:
-            # fit for multiple GPU training
-            # since the first element for pos embed (sin-cos manner) is zero, it will cause no difference
-            x = x + self.pos_embed[:, 1:] + self.pos_embed[:, :1]
-        # X [B, 192, 1280]
-        # x cat [ mean_pose, mean_shape, mean_cam] tokens 
-        pose_tokens  = self.pose_emb(self.init_hand_pose.reshape(1, self.cfg.MANO.NUM_HAND_JOINTS + 1, self.joint_rep_dim)).repeat(B, 1, 1)
-        shape_tokens = self.shape_emb(self.init_betas).unsqueeze(1).repeat(B, 1, 1)
-        cam_tokens   = self.cam_emb(self.init_cam).unsqueeze(1).repeat(B, 1, 1)
-        
-        x = torch.cat([pose_tokens, shape_tokens, cam_tokens, x], 1)
-        for blk in self.blocks:
-            if self.use_checkpoint:
-                x = checkpoint.checkpoint(blk, x)
-            else:
-                x = blk(x)
-
-        x = self.last_norm(x)
-        
-        
-        pose_feat  = x[:, :(self.cfg.MANO.NUM_HAND_JOINTS + 1)]
-        shape_feat = x[:, (self.cfg.MANO.NUM_HAND_JOINTS + 1):1+(self.cfg.MANO.NUM_HAND_JOINTS + 1)]
-        cam_feat   = x[:, 1+(self.cfg.MANO.NUM_HAND_JOINTS + 1):2+(self.cfg.MANO.NUM_HAND_JOINTS + 1)]
-        
-        #print(pose_feat.shape, shape_feat.shape, cam_feat.shape)
-        pred_hand_pose = self.decpose(pose_feat).reshape(B, -1) + self.init_hand_pose  #B , 96
-        pred_betas = self.decshape(shape_feat).reshape(B, -1) + self.init_betas        #B , 10
-        pred_cam = self.deccam(cam_feat).reshape(B, -1) + self.init_cam                #B , 3
-        
-        pred_mano_feats = {}
-        pred_mano_feats['hand_pose'] = pred_hand_pose
-        pred_mano_feats['betas']     = pred_betas
-        pred_mano_feats['cam']       = pred_cam
-
-        
-        joint_conversion_fn = {
-                '6d': rot6d_to_rotmat,
-                'aa': lambda x: aa_to_rotmat(x.view(-1, 3).contiguous())
-            }[self.joint_rep_type]
- 
-        pred_hand_pose = joint_conversion_fn(pred_hand_pose).view(B, self.cfg.MANO.NUM_HAND_JOINTS+1, 3, 3)
-        pred_mano_params = {'global_orient': pred_hand_pose[:, [0]],
-                            'hand_pose': pred_hand_pose[:, 1:],
-                            'betas': pred_betas}
-        
-        img_feat = x[:, 2+(self.cfg.MANO.NUM_HAND_JOINTS + 1):].reshape(B, Hp, Wp, -1).permute(0, 3, 1, 2)
-        return pred_mano_params, pred_cam, pred_mano_feats, img_feat
-
-    def forward(self, x):
-        x = self.forward_features(x)
-        return x
-
-    def train(self, mode=True):
-        """Convert the model into training mode."""
-        super().train(mode)
-        self._freeze_stages()

WiLoR/wilor/models/discriminator.py

@@ -1,98 +0,0 @@
-import torch
-import torch.nn as nn
-
-class Discriminator(nn.Module):
-
-    def __init__(self):
-        """
-        Pose + Shape discriminator proposed in HMR
-        """
-        super(Discriminator, self).__init__()
-
-        self.num_joints = 15
-        # poses_alone
-        self.D_conv1 = nn.Conv2d(9, 32, kernel_size=1)
-        nn.init.xavier_uniform_(self.D_conv1.weight)
-        nn.init.zeros_(self.D_conv1.bias)
-        self.relu = nn.ReLU(inplace=True)
-        self.D_conv2 = nn.Conv2d(32, 32, kernel_size=1)
-        nn.init.xavier_uniform_(self.D_conv2.weight)
-        nn.init.zeros_(self.D_conv2.bias)
-        pose_out = []
-        for i in range(self.num_joints):
-            pose_out_temp = nn.Linear(32, 1)
-            nn.init.xavier_uniform_(pose_out_temp.weight)
-            nn.init.zeros_(pose_out_temp.bias)
-            pose_out.append(pose_out_temp)
-        self.pose_out = nn.ModuleList(pose_out)
-
-        # betas
-        self.betas_fc1 = nn.Linear(10, 10)
-        nn.init.xavier_uniform_(self.betas_fc1.weight)
-        nn.init.zeros_(self.betas_fc1.bias)
-        self.betas_fc2 = nn.Linear(10, 5)
-        nn.init.xavier_uniform_(self.betas_fc2.weight)
-        nn.init.zeros_(self.betas_fc2.bias)
-        self.betas_out = nn.Linear(5, 1)
-        nn.init.xavier_uniform_(self.betas_out.weight)
-        nn.init.zeros_(self.betas_out.bias)
-
-        # poses_joint
-        self.D_alljoints_fc1 = nn.Linear(32*self.num_joints, 1024)
-        nn.init.xavier_uniform_(self.D_alljoints_fc1.weight)
-        nn.init.zeros_(self.D_alljoints_fc1.bias)
-        self.D_alljoints_fc2 = nn.Linear(1024, 1024)
-        nn.init.xavier_uniform_(self.D_alljoints_fc2.weight)
-        nn.init.zeros_(self.D_alljoints_fc2.bias)
-        self.D_alljoints_out = nn.Linear(1024, 1)
-        nn.init.xavier_uniform_(self.D_alljoints_out.weight)
-        nn.init.zeros_(self.D_alljoints_out.bias)
-
-
-    def forward(self, poses: torch.Tensor, betas: torch.Tensor) -> torch.Tensor:
-        """
-        Forward pass of the discriminator.
-        Args:
-            poses (torch.Tensor): Tensor of shape (B, 23, 3, 3) containing a batch of MANO hand poses (excluding the global orientation).
-            betas (torch.Tensor): Tensor of shape (B, 10) containign a batch of MANO beta coefficients.
-        Returns:
-            torch.Tensor: Discriminator output with shape (B, 25)
-        """
-        #bn = poses.shape[0]
-        # poses B x 207
-        #poses = poses.reshape(bn, -1)
-        # poses B x num_joints x 1 x 9
-        poses = poses.reshape(-1, self.num_joints, 1, 9)
-        bn = poses.shape[0]
-        # poses B x 9 x num_joints x 1
-        poses = poses.permute(0, 3, 1, 2).contiguous()
-
-        # poses_alone
-        poses = self.D_conv1(poses)
-        poses = self.relu(poses)
-        poses = self.D_conv2(poses)
-        poses = self.relu(poses)
-
-        poses_out = []
-        for i in range(self.num_joints):
-            poses_out_ = self.pose_out[i](poses[:, :, i, 0])
-            poses_out.append(poses_out_)
-        poses_out = torch.cat(poses_out, dim=1)
-
-        # betas
-        betas = self.betas_fc1(betas)
-        betas = self.relu(betas)
-        betas = self.betas_fc2(betas)
-        betas = self.relu(betas)
-        betas_out = self.betas_out(betas)
-
-        # poses_joint
-        poses = poses.reshape(bn,-1)
-        poses_all = self.D_alljoints_fc1(poses)
-        poses_all = self.relu(poses_all)
-        poses_all = self.D_alljoints_fc2(poses_all)
-        poses_all = self.relu(poses_all)
-        poses_all_out = self.D_alljoints_out(poses_all)
-
-        disc_out = torch.cat((poses_out, betas_out, poses_all_out), 1)
-        return disc_out

WiLoR/wilor/models/heads/__init__.py

@@ -1 +0,0 @@
-from .refinement_net import RefineNet

WiLoR/wilor/models/heads/refinement_net.py

@@ -1,204 +0,0 @@
-import torch
-import torch.nn as nn 
-import torch.nn.functional as F 
-import math 
-from ...utils.geometry import rot6d_to_rotmat, aa_to_rotmat
-from typing import Optional
-
-def make_linear_layers(feat_dims, relu_final=True, use_bn=False):
-    layers = []
-    for i in range(len(feat_dims)-1):
-        layers.append(nn.Linear(feat_dims[i], feat_dims[i+1]))
-
-        # Do not use ReLU for final estimation
-        if i < len(feat_dims)-2 or (i == len(feat_dims)-2 and relu_final):
-            if use_bn:
-                layers.append(nn.BatchNorm1d(feat_dims[i+1]))
-            layers.append(nn.ReLU(inplace=True))
-
-    return nn.Sequential(*layers)
-
-def make_conv_layers(feat_dims, kernel=3, stride=1, padding=1, bnrelu_final=True):
-    layers = []
-    for i in range(len(feat_dims)-1):
-        layers.append(
-            nn.Conv2d(
-                in_channels=feat_dims[i],
-                out_channels=feat_dims[i+1],
-                kernel_size=kernel,
-                stride=stride,
-                padding=padding
-                ))
-        # Do not use BN and ReLU for final estimation
-        if i < len(feat_dims)-2 or (i == len(feat_dims)-2 and bnrelu_final):
-            layers.append(nn.BatchNorm2d(feat_dims[i+1]))
-            layers.append(nn.ReLU(inplace=True))
-
-    return nn.Sequential(*layers)
-
-def make_deconv_layers(feat_dims, bnrelu_final=True):
-    layers = []
-    for i in range(len(feat_dims)-1):
-        layers.append(
-            nn.ConvTranspose2d(
-                in_channels=feat_dims[i],
-                out_channels=feat_dims[i+1],
-                kernel_size=4,
-                stride=2,
-                padding=1,
-                output_padding=0,
-                bias=False))
-
-        # Do not use BN and ReLU for final estimation
-        if i < len(feat_dims)-2 or (i == len(feat_dims)-2 and bnrelu_final):
-            layers.append(nn.BatchNorm2d(feat_dims[i+1]))
-            layers.append(nn.ReLU(inplace=True))
-
-    return nn.Sequential(*layers)
-
-def sample_joint_features(img_feat, joint_xy):
-    height, width = img_feat.shape[2:]
-    x = joint_xy[:, :, 0] / (width - 1) * 2 - 1
-    y = joint_xy[:, :, 1] / (height - 1) * 2 - 1
-    grid = torch.stack((x, y), 2)[:, :, None, :]
-    img_feat = F.grid_sample(img_feat, grid, align_corners=True)[:, :, :, 0]  # batch_size, channel_dim, joint_num
-    img_feat = img_feat.permute(0, 2, 1).contiguous()  # batch_size, joint_num, channel_dim
-    return img_feat
-
-def perspective_projection(points: torch.Tensor,
-                           translation: torch.Tensor,
-                           focal_length: torch.Tensor,
-                           camera_center: Optional[torch.Tensor] = None,
-                           rotation: Optional[torch.Tensor] = None) -> torch.Tensor:
-    """
-    Computes the perspective projection of a set of 3D points.
-    Args:
-        points (torch.Tensor): Tensor of shape (B, N, 3) containing the input 3D points.
-        translation (torch.Tensor): Tensor of shape (B, 3) containing the 3D camera translation.
-        focal_length (torch.Tensor): Tensor of shape (B, 2) containing the focal length in pixels.
-        camera_center (torch.Tensor): Tensor of shape (B, 2) containing the camera center in pixels.
-        rotation (torch.Tensor): Tensor of shape (B, 3, 3) containing the camera rotation.
-    Returns:
-        torch.Tensor: Tensor of shape (B, N, 2) containing the projection of the input points.
-    """
-    batch_size = points.shape[0]
-    if rotation is None:
-        rotation = torch.eye(3, device=points.device, dtype=points.dtype).unsqueeze(0).expand(batch_size, -1, -1)
-    if camera_center is None:
-        camera_center = torch.zeros(batch_size, 2, device=points.device, dtype=points.dtype)
-    # Populate intrinsic camera matrix K.
-    K = torch.zeros([batch_size, 3, 3], device=points.device, dtype=points.dtype)
-    K[:,0,0] = focal_length[:,0]
-    K[:,1,1] = focal_length[:,1]
-    K[:,2,2] = 1.
-    K[:,:-1, -1] = camera_center
-    # Transform points
-    points = torch.einsum('bij,bkj->bki', rotation, points)
-    points = points + translation.unsqueeze(1)
-
-    # Apply perspective distortion
-    projected_points = points / points[:,:,-1].unsqueeze(-1)
-
-    # Apply camera intrinsics
-    projected_points = torch.einsum('bij,bkj->bki', K, projected_points)
-
-    return projected_points[:, :, :-1]
-
-class DeConvNet(nn.Module):
-    def __init__(self, feat_dim=768, upscale=4):
-        super(DeConvNet, self).__init__()
-        self.first_conv = make_conv_layers([feat_dim, feat_dim//2], kernel=1, stride=1, padding=0, bnrelu_final=False)
-        self.deconv = nn.ModuleList([])
-        for i in range(int(math.log2(upscale))+1):
-            if i==0:
-                self.deconv.append(make_deconv_layers([feat_dim//2, feat_dim//4]))
-            elif i==1:
-                self.deconv.append(make_deconv_layers([feat_dim//2, feat_dim//4, feat_dim//8]))
-            elif i==2:
-                self.deconv.append(make_deconv_layers([feat_dim//2, feat_dim//4, feat_dim//8, feat_dim//8]))
-
-    def forward(self, img_feat):
-        
-        face_img_feats = []
-        img_feat = self.first_conv(img_feat)
-        face_img_feats.append(img_feat)
-        for i, deconv in enumerate(self.deconv):
-            scale = 2**i
-            img_feat_i = deconv(img_feat)
-            face_img_feat = img_feat_i
-            face_img_feats.append(face_img_feat)
-        return face_img_feats[::-1]   # high resolution -> low resolution
-
-class DeConvNet_v2(nn.Module):
-    def __init__(self, feat_dim=768):
-        super(DeConvNet_v2, self).__init__()
-        self.first_conv = make_conv_layers([feat_dim, feat_dim//2], kernel=1, stride=1, padding=0, bnrelu_final=False)
-        self.deconv = nn.Sequential(*[nn.ConvTranspose2d(in_channels=feat_dim//2, out_channels=feat_dim//4, kernel_size=4, stride=4, padding=0, output_padding=0, bias=False), 
-                       nn.BatchNorm2d(feat_dim//4), 
-                       nn.ReLU(inplace=True)])
-    
-    def forward(self, img_feat):
-        
-        face_img_feats = []
-        img_feat = self.first_conv(img_feat)
-        img_feat = self.deconv(img_feat) 
-        
-        return [img_feat]
-        
-class RefineNet(nn.Module):
-    def __init__(self, cfg, feat_dim=1280, upscale=3):
-        super(RefineNet, self).__init__()
-        #self.deconv     = DeConvNet_v2(feat_dim=feat_dim) 
-        #self.out_dim    = feat_dim//4
-        
-        self.deconv     = DeConvNet(feat_dim=feat_dim, upscale=upscale)
-        self.out_dim    = feat_dim//8  + feat_dim//4 + feat_dim//2 
-        self.dec_pose   = nn.Linear(self.out_dim, 96) 
-        self.dec_cam    = nn.Linear(self.out_dim, 3)
-        self.dec_shape  = nn.Linear(self.out_dim, 10)
-        
-        self.cfg        = cfg
-        self.joint_rep_type = cfg.MODEL.MANO_HEAD.get('JOINT_REP', '6d')
-        self.joint_rep_dim = {'6d': 6, 'aa': 3}[self.joint_rep_type]
-        
-    def forward(self, img_feat, verts_3d, pred_cam, pred_mano_feats, focal_length):
-        B = img_feat.shape[0]
-        
-        img_feats = self.deconv(img_feat)
-        
-        img_feat_sizes = [img_feat.shape[2] for img_feat in img_feats] 
-        
-        temp_cams  = [torch.stack([pred_cam[:, 1], pred_cam[:, 2],  
-                                  2*focal_length[:, 0]/(img_feat_size * pred_cam[:, 0] +1e-9)],dim=-1) for img_feat_size in img_feat_sizes] 
-
-        verts_2d   = [perspective_projection(verts_3d,
-                                translation=temp_cams[i],
-                                focal_length=focal_length / img_feat_sizes[i]) for i in range(len(img_feat_sizes))]
-        
-        vert_feats = [sample_joint_features(img_feats[i], verts_2d[i]).max(1).values  for i in range(len(img_feat_sizes))] 
-
-        vert_feats = torch.cat(vert_feats, dim=-1)
-
-        delta_pose  = self.dec_pose(vert_feats)
-        delta_betas = self.dec_shape(vert_feats)
-        delta_cam   = self.dec_cam(vert_feats)
-
-        
-        pred_hand_pose = pred_mano_feats['hand_pose'] + delta_pose
-        pred_betas     = pred_mano_feats['betas']     + delta_betas 
-        pred_cam       = pred_mano_feats['cam']       + delta_cam
-
-        joint_conversion_fn = {
-                '6d': rot6d_to_rotmat,
-                'aa': lambda x: aa_to_rotmat(x.view(-1, 3).contiguous())
-            }[self.joint_rep_type]
- 
-        pred_hand_pose = joint_conversion_fn(pred_hand_pose).view(B, self.cfg.MANO.NUM_HAND_JOINTS+1, 3, 3)
-        
-        pred_mano_params = {'global_orient': pred_hand_pose[:, [0]],
-                            'hand_pose': pred_hand_pose[:, 1:],
-                            'betas': pred_betas}
-        
-        return  pred_mano_params, pred_cam
-    
-    

WiLoR/wilor/models/losses.py

@@ -1,92 +0,0 @@
-import torch
-import torch.nn as nn
-
-class Keypoint2DLoss(nn.Module):
-
-    def __init__(self, loss_type: str = 'l1'):
-        """
-        2D keypoint loss module.
-        Args:
-            loss_type (str): Choose between l1 and l2 losses.
-        """
-        super(Keypoint2DLoss, self).__init__()
-        if loss_type == 'l1':
-            self.loss_fn = nn.L1Loss(reduction='none')
-        elif loss_type == 'l2':
-            self.loss_fn = nn.MSELoss(reduction='none')
-        else:
-            raise NotImplementedError('Unsupported loss function')
-
-    def forward(self, pred_keypoints_2d: torch.Tensor, gt_keypoints_2d: torch.Tensor) -> torch.Tensor:
-        """
-        Compute 2D reprojection loss on the keypoints.
-        Args:
-            pred_keypoints_2d (torch.Tensor): Tensor of shape [B, S, N, 2] containing projected 2D keypoints (B: batch_size, S: num_samples, N: num_keypoints)
-            gt_keypoints_2d (torch.Tensor): Tensor of shape [B, S, N, 3] containing the ground truth 2D keypoints and confidence.
-        Returns:
-            torch.Tensor: 2D keypoint loss.
-        """
-        conf = gt_keypoints_2d[:, :, -1].unsqueeze(-1).clone()
-        batch_size = conf.shape[0]
-        loss = (conf * self.loss_fn(pred_keypoints_2d, gt_keypoints_2d[:, :, :-1])).sum(dim=(1,2))
-        return loss.sum()
-
-
-class Keypoint3DLoss(nn.Module):
-
-    def __init__(self, loss_type: str = 'l1'):
-        """
-        3D keypoint loss module.
-        Args:
-            loss_type (str): Choose between l1 and l2 losses.
-        """
-        super(Keypoint3DLoss, self).__init__()
-        if loss_type == 'l1':
-            self.loss_fn = nn.L1Loss(reduction='none')
-        elif loss_type == 'l2':
-            self.loss_fn = nn.MSELoss(reduction='none')
-        else:
-            raise NotImplementedError('Unsupported loss function')
-
-    def forward(self, pred_keypoints_3d: torch.Tensor, gt_keypoints_3d: torch.Tensor, pelvis_id: int = 0):
-        """
-        Compute 3D keypoint loss.
-        Args:
-            pred_keypoints_3d (torch.Tensor): Tensor of shape [B, S, N, 3] containing the predicted 3D keypoints (B: batch_size, S: num_samples, N: num_keypoints)
-            gt_keypoints_3d (torch.Tensor): Tensor of shape [B, S, N, 4] containing the ground truth 3D keypoints and confidence.
-        Returns:
-            torch.Tensor: 3D keypoint loss.
-        """
-        batch_size = pred_keypoints_3d.shape[0]
-        gt_keypoints_3d = gt_keypoints_3d.clone()
-        pred_keypoints_3d = pred_keypoints_3d - pred_keypoints_3d[:, pelvis_id, :].unsqueeze(dim=1)
-        gt_keypoints_3d[:, :, :-1] = gt_keypoints_3d[:, :, :-1] - gt_keypoints_3d[:, pelvis_id, :-1].unsqueeze(dim=1)
-        conf = gt_keypoints_3d[:, :, -1].unsqueeze(-1).clone()
-        gt_keypoints_3d = gt_keypoints_3d[:, :, :-1]
-        loss = (conf * self.loss_fn(pred_keypoints_3d, gt_keypoints_3d)).sum(dim=(1,2))
-        return loss.sum()
-
-class ParameterLoss(nn.Module):
-
-    def __init__(self):
-        """
-        MANO parameter loss module.
-        """
-        super(ParameterLoss, self).__init__()
-        self.loss_fn = nn.MSELoss(reduction='none')
-
-    def forward(self, pred_param: torch.Tensor, gt_param: torch.Tensor, has_param: torch.Tensor):
-        """
-        Compute MANO parameter loss.
-        Args:
-            pred_param (torch.Tensor): Tensor of shape [B, S, ...] containing the predicted parameters (body pose / global orientation / betas)
-            gt_param (torch.Tensor): Tensor of shape [B, S, ...] containing the ground truth MANO parameters.
-        Returns:
-            torch.Tensor: L2 parameter loss loss.
-        """
-        batch_size = pred_param.shape[0]
-        num_dims = len(pred_param.shape)
-        mask_dimension = [batch_size] + [1] * (num_dims-1)
-        has_param = has_param.type(pred_param.type()).view(*mask_dimension)
-        loss_param = (has_param * self.loss_fn(pred_param, gt_param))
-        return loss_param.sum()

WiLoR/wilor/models/mano_wrapper.py

@@ -1,40 +0,0 @@
-import torch
-import numpy as np
-import pickle
-from typing import Optional
-import smplx
-from smplx.lbs import vertices2joints
-from smplx.utils import MANOOutput, to_tensor
-from smplx.vertex_ids import vertex_ids
-
-
-class MANO(smplx.MANOLayer):
-    def __init__(self, *args, joint_regressor_extra: Optional[str] = None, **kwargs):
-        """
-        Extension of the official MANO implementation to support more joints.
-        Args:
-            Same as MANOLayer.
-            joint_regressor_extra (str): Path to extra joint regressor.
-        """
-        super(MANO, self).__init__(*args, **kwargs)
-        mano_to_openpose = [0, 13, 14, 15, 16, 1, 2, 3, 17, 4, 5, 6, 18, 10, 11, 12, 19, 7, 8, 9, 20]
-
-        #2, 3, 5, 4, 1
-        if joint_regressor_extra is not None:
-            self.register_buffer('joint_regressor_extra', torch.tensor(pickle.load(open(joint_regressor_extra, 'rb'), encoding='latin1'), dtype=torch.float32))
-        self.register_buffer('extra_joints_idxs', to_tensor(list(vertex_ids['mano'].values()), dtype=torch.long))
-        self.register_buffer('joint_map', torch.tensor(mano_to_openpose, dtype=torch.long))
-
-    def forward(self, *args, **kwargs) -> MANOOutput:
-        """
-        Run forward pass. Same as MANO and also append an extra set of joints if joint_regressor_extra is specified.
-        """
-        mano_output = super(MANO, self).forward(*args, **kwargs)
-        extra_joints = torch.index_select(mano_output.vertices, 1, self.extra_joints_idxs)
-        joints = torch.cat([mano_output.joints, extra_joints], dim=1)
-        joints = joints[:, self.joint_map, :]
-        if hasattr(self, 'joint_regressor_extra'):
-            extra_joints = vertices2joints(self.joint_regressor_extra, mano_output.vertices)
-            joints = torch.cat([joints, extra_joints], dim=1)
-        mano_output.joints = joints
-        return mano_output

WiLoR/wilor/models/wilor.py

@@ -1,376 +0,0 @@
-import torch
-import pytorch_lightning as pl
-from typing import Any, Dict, Mapping, Tuple
-
-from yacs.config import CfgNode
-
-from ..utils import SkeletonRenderer, MeshRenderer
-from ..utils.geometry import aa_to_rotmat, perspective_projection
-from ..utils.pylogger import get_pylogger
-from .backbones import create_backbone
-from .heads import RefineNet
-from .discriminator import Discriminator
-from .losses import Keypoint3DLoss, Keypoint2DLoss, ParameterLoss
-from . import MANO
-
-log = get_pylogger(__name__)
-
-class WiLoR(pl.LightningModule):
-
-    def __init__(self, cfg: CfgNode, init_renderer: bool = True):
-        """
-        Setup WiLoR model
-        Args:
-            cfg (CfgNode): Config file as a yacs CfgNode
-        """
-        super().__init__()
-
-        # Save hyperparameters
-        self.save_hyperparameters(logger=False, ignore=['init_renderer'])
-
-        self.cfg = cfg
-        # Create backbone feature extractor
-        self.backbone = create_backbone(cfg)
-        if cfg.MODEL.BACKBONE.get('PRETRAINED_WEIGHTS', None):
-            log.info(f'Loading backbone weights from {cfg.MODEL.BACKBONE.PRETRAINED_WEIGHTS}')
-            self.backbone.load_state_dict(torch.load(cfg.MODEL.BACKBONE.PRETRAINED_WEIGHTS, map_location='cpu')['state_dict'], strict = False)
-            
-        # Create RefineNet head
-        self.refine_net = RefineNet(cfg, feat_dim=1280, upscale=3)
-        
-        # Create discriminator
-        if self.cfg.LOSS_WEIGHTS.ADVERSARIAL > 0:
-            self.discriminator = Discriminator()
-
-        # Define loss functions
-        self.keypoint_3d_loss = Keypoint3DLoss(loss_type='l1')
-        self.keypoint_2d_loss = Keypoint2DLoss(loss_type='l1')
-        self.mano_parameter_loss = ParameterLoss()
-
-        # Instantiate MANO model
-        mano_cfg = {k.lower(): v for k,v in dict(cfg.MANO).items()}
-        self.mano = MANO(**mano_cfg)
-
-        # Buffer that shows whetheer we need to initialize ActNorm layers
-        self.register_buffer('initialized', torch.tensor(False))
-        # Setup renderer for visualization
-        if init_renderer:
-            self.renderer = SkeletonRenderer(self.cfg)
-            self.mesh_renderer = MeshRenderer(self.cfg, faces=self.mano.faces)
-        else:
-            self.renderer = None
-            self.mesh_renderer = None
-            
-
-        # Disable automatic optimization since we use adversarial training
-        self.automatic_optimization = False
-
-    def on_after_backward(self):
-        for name, param in self.named_parameters():
-            if param.grad is None:
-                print(param.shape)
-                print(name)
-
-
-    def get_parameters(self):
-        #all_params = list(self.mano_head.parameters())
-        all_params = list(self.backbone.parameters())
-        return all_params
-
-    def configure_optimizers(self) -> Tuple[torch.optim.Optimizer, torch.optim.Optimizer]:
-        """
-        Setup model and distriminator Optimizers
-        Returns:
-            Tuple[torch.optim.Optimizer, torch.optim.Optimizer]: Model and discriminator optimizers
-        """
-        param_groups = [{'params': filter(lambda p: p.requires_grad, self.get_parameters()), 'lr': self.cfg.TRAIN.LR}]
-
-        optimizer = torch.optim.AdamW(params=param_groups,
-                                        # lr=self.cfg.TRAIN.LR,
-                                        weight_decay=self.cfg.TRAIN.WEIGHT_DECAY)
-        optimizer_disc = torch.optim.AdamW(params=self.discriminator.parameters(),
-                                            lr=self.cfg.TRAIN.LR,
-                                            weight_decay=self.cfg.TRAIN.WEIGHT_DECAY)
-
-        return optimizer, optimizer_disc
-
-    def forward_step(self, batch: Dict, train: bool = False) -> Dict:
-        """
-        Run a forward step of the network
-        Args:
-            batch (Dict): Dictionary containing batch data
-            train (bool): Flag indicating whether it is training or validation mode
-        Returns:
-            Dict: Dictionary containing the regression output
-        """
-        # Use RGB image as input
-        x = batch['img']
-        batch_size = x.shape[0]
-        # Compute conditioning features using the backbone
-        # if using ViT backbone, we need to use a different aspect ratio
-        temp_mano_params, pred_cam, pred_mano_feats, vit_out = self.backbone(x[:,:,:,32:-32]) # B, 1280, 16, 12  
-
-    
-        # Compute camera translation
-        device = temp_mano_params['hand_pose'].device
-        dtype = temp_mano_params['hand_pose'].dtype
-        focal_length = self.cfg.EXTRA.FOCAL_LENGTH * torch.ones(batch_size, 2, device=device, dtype=dtype)
-        
-        
-        ## Temp MANO 
-        temp_mano_params['global_orient'] = temp_mano_params['global_orient'].reshape(batch_size, -1, 3, 3)
-        temp_mano_params['hand_pose'] = temp_mano_params['hand_pose'].reshape(batch_size, -1, 3, 3)
-        temp_mano_params['betas'] = temp_mano_params['betas'].reshape(batch_size, -1)
-        temp_mano_output  = self.mano(**{k: v.float() for k,v in temp_mano_params.items()}, pose2rot=False)
-        #temp_keypoints_3d = temp_mano_output.joints
-        temp_vertices     = temp_mano_output.vertices
-
-        pred_mano_params, pred_cam = self.refine_net(vit_out, temp_vertices, pred_cam, pred_mano_feats, focal_length) 
-        # Store useful regression outputs to the output dict
-        
-       
-        output = {}
-        output['pred_cam'] = pred_cam
-        output['pred_mano_params'] = {k: v.clone() for k,v in pred_mano_params.items()}
-
-        pred_cam_t = torch.stack([pred_cam[:, 1],
-                                  pred_cam[:, 2],
-                                  2*focal_length[:, 0]/(self.cfg.MODEL.IMAGE_SIZE * pred_cam[:, 0] +1e-9)],dim=-1)
-        output['pred_cam_t'] = pred_cam_t
-        output['focal_length'] = focal_length
-
-        # Compute model vertices, joints and the projected joints
-        pred_mano_params['global_orient'] = pred_mano_params['global_orient'].reshape(batch_size, -1, 3, 3)
-        pred_mano_params['hand_pose'] = pred_mano_params['hand_pose'].reshape(batch_size, -1, 3, 3)
-        pred_mano_params['betas'] = pred_mano_params['betas'].reshape(batch_size, -1)
-        mano_output = self.mano(**{k: v.float() for k,v in pred_mano_params.items()}, pose2rot=False)
-        pred_keypoints_3d = mano_output.joints
-        pred_vertices = mano_output.vertices
-  
-        output['pred_keypoints_3d'] = pred_keypoints_3d.reshape(batch_size, -1, 3)
-        output['pred_vertices'] = pred_vertices.reshape(batch_size, -1, 3)
-        pred_cam_t = pred_cam_t.reshape(-1, 3)
-        focal_length = focal_length.reshape(-1, 2)
-        
-        pred_keypoints_2d = perspective_projection(pred_keypoints_3d,
-                                                   translation=pred_cam_t,
-                                                   focal_length=focal_length / self.cfg.MODEL.IMAGE_SIZE)
-        output['pred_keypoints_2d'] = pred_keypoints_2d.reshape(batch_size, -1, 2)
-        
-        return output
-
-    def compute_loss(self, batch: Dict, output: Dict, train: bool = True) -> torch.Tensor:
-        """
-        Compute losses given the input batch and the regression output
-        Args:
-            batch (Dict): Dictionary containing batch data
-            output (Dict): Dictionary containing the regression output
-            train (bool): Flag indicating whether it is training or validation mode
-        Returns:
-            torch.Tensor : Total loss for current batch
-        """
-
-        pred_mano_params = output['pred_mano_params']
-        pred_keypoints_2d = output['pred_keypoints_2d']
-        pred_keypoints_3d = output['pred_keypoints_3d']
-
-
-        batch_size = pred_mano_params['hand_pose'].shape[0]
-        device = pred_mano_params['hand_pose'].device
-        dtype = pred_mano_params['hand_pose'].dtype
-
-        # Get annotations
-        gt_keypoints_2d = batch['keypoints_2d']
-        gt_keypoints_3d = batch['keypoints_3d']
-        gt_mano_params = batch['mano_params']
-        has_mano_params = batch['has_mano_params']
-        is_axis_angle = batch['mano_params_is_axis_angle']
-
-        # Compute 3D keypoint loss
-        loss_keypoints_2d = self.keypoint_2d_loss(pred_keypoints_2d, gt_keypoints_2d)
-        loss_keypoints_3d = self.keypoint_3d_loss(pred_keypoints_3d, gt_keypoints_3d, pelvis_id=0)
-
-        # Compute loss on MANO parameters
-        loss_mano_params = {}
-        for k, pred in pred_mano_params.items():
-            gt = gt_mano_params[k].view(batch_size, -1)
-            if is_axis_angle[k].all():
-                gt = aa_to_rotmat(gt.reshape(-1, 3)).view(batch_size, -1, 3, 3)
-            has_gt = has_mano_params[k]
-            loss_mano_params[k] = self.mano_parameter_loss(pred.reshape(batch_size, -1), gt.reshape(batch_size, -1), has_gt)
-
-        loss = self.cfg.LOSS_WEIGHTS['KEYPOINTS_3D'] * loss_keypoints_3d+\
-               self.cfg.LOSS_WEIGHTS['KEYPOINTS_2D'] * loss_keypoints_2d+\
-               sum([loss_mano_params[k] * self.cfg.LOSS_WEIGHTS[k.upper()] for k in loss_mano_params])
-
-
-        losses = dict(loss=loss.detach(),
-                      loss_keypoints_2d=loss_keypoints_2d.detach(),
-                      loss_keypoints_3d=loss_keypoints_3d.detach())
-
-        for k, v in loss_mano_params.items():
-            losses['loss_' + k] = v.detach()
-
-        output['losses'] = losses
-
-        return loss
-
-    # Tensoroboard logging should run from first rank only
-    @pl.utilities.rank_zero.rank_zero_only
-    def tensorboard_logging(self, batch: Dict, output: Dict, step_count: int, train: bool = True, write_to_summary_writer: bool = True) -> None:
-        """
-        Log results to Tensorboard
-        Args:
-            batch (Dict): Dictionary containing batch data
-            output (Dict): Dictionary containing the regression output
-            step_count (int): Global training step count
-            train (bool): Flag indicating whether it is training or validation mode
-        """
-
-        mode = 'train' if train else 'val'
-        batch_size = batch['keypoints_2d'].shape[0]
-        images = batch['img']
-        images = images * torch.tensor([0.229, 0.224, 0.225], device=images.device).reshape(1,3,1,1)
-        images = images + torch.tensor([0.485, 0.456, 0.406], device=images.device).reshape(1,3,1,1)
-        #images = 255*images.permute(0, 2, 3, 1).cpu().numpy()
-
-        pred_keypoints_3d = output['pred_keypoints_3d'].detach().reshape(batch_size, -1, 3)
-        pred_vertices = output['pred_vertices'].detach().reshape(batch_size, -1, 3)
-        focal_length = output['focal_length'].detach().reshape(batch_size, 2)
-        gt_keypoints_3d = batch['keypoints_3d']
-        gt_keypoints_2d = batch['keypoints_2d']
-        
-        losses = output['losses']
-        pred_cam_t = output['pred_cam_t'].detach().reshape(batch_size, 3)
-        pred_keypoints_2d = output['pred_keypoints_2d'].detach().reshape(batch_size, -1, 2)
-        if write_to_summary_writer:
-            summary_writer = self.logger.experiment
-            for loss_name, val in losses.items():
-                summary_writer.add_scalar(mode +'/' + loss_name, val.detach().item(), step_count)
-        num_images = min(batch_size, self.cfg.EXTRA.NUM_LOG_IMAGES)
-
-        gt_keypoints_3d = batch['keypoints_3d']
-        pred_keypoints_3d = output['pred_keypoints_3d'].detach().reshape(batch_size, -1, 3)
-
-        # We render the skeletons instead of the full mesh because rendering a lot of meshes will make the training slow.
-        #predictions = self.renderer(pred_keypoints_3d[:num_images],
-        #                            gt_keypoints_3d[:num_images],
-        #                            2 * gt_keypoints_2d[:num_images],
-        #                            images=images[:num_images],
-        #                            camera_translation=pred_cam_t[:num_images])
-        predictions = self.mesh_renderer.visualize_tensorboard(pred_vertices[:num_images].cpu().numpy(),
-                                                               pred_cam_t[:num_images].cpu().numpy(),
-                                                               images[:num_images].cpu().numpy(),
-                                                               pred_keypoints_2d[:num_images].cpu().numpy(),
-                                                               gt_keypoints_2d[:num_images].cpu().numpy(),
-                                                               focal_length=focal_length[:num_images].cpu().numpy())
-        if write_to_summary_writer:
-            summary_writer.add_image('%s/predictions' % mode, predictions, step_count)
-
-        return predictions
-
-    def forward(self, batch: Dict) -> Dict:
-        """
-        Run a forward step of the network in val mode
-        Args:
-            batch (Dict): Dictionary containing batch data
-        Returns:
-            Dict: Dictionary containing the regression output
-        """
-        return self.forward_step(batch, train=False)
-
-    def training_step_discriminator(self, batch: Dict,
-                                    hand_pose: torch.Tensor,
-                                    betas: torch.Tensor,
-                                    optimizer: torch.optim.Optimizer) -> torch.Tensor:
-        """
-        Run a discriminator training step
-        Args:
-            batch (Dict): Dictionary containing mocap batch data
-            hand_pose (torch.Tensor): Regressed hand pose from current step
-            betas (torch.Tensor): Regressed betas from current step
-            optimizer (torch.optim.Optimizer): Discriminator optimizer
-        Returns:
-            torch.Tensor: Discriminator loss
-        """
-        batch_size = hand_pose.shape[0]
-        gt_hand_pose = batch['hand_pose']
-        gt_betas = batch['betas']
-        gt_rotmat = aa_to_rotmat(gt_hand_pose.view(-1,3)).view(batch_size, -1, 3, 3)
-        disc_fake_out = self.discriminator(hand_pose.detach(), betas.detach())
-        loss_fake = ((disc_fake_out - 0.0) ** 2).sum() / batch_size
-        disc_real_out = self.discriminator(gt_rotmat, gt_betas)
-        loss_real = ((disc_real_out - 1.0) ** 2).sum() / batch_size
-        loss_disc = loss_fake + loss_real
-        loss = self.cfg.LOSS_WEIGHTS.ADVERSARIAL * loss_disc
-        optimizer.zero_grad()
-        self.manual_backward(loss)
-        optimizer.step()
-        return loss_disc.detach()
-
-    def training_step(self, joint_batch: Dict, batch_idx: int) -> Dict:
-        """
-        Run a full training step
-        Args:
-            joint_batch (Dict): Dictionary containing image and mocap batch data
-            batch_idx (int): Unused.
-            batch_idx (torch.Tensor): Unused.
-        Returns:
-            Dict: Dictionary containing regression output.
-        """
-        batch = joint_batch['img']
-        mocap_batch = joint_batch['mocap']
-        optimizer = self.optimizers(use_pl_optimizer=True)
-        if self.cfg.LOSS_WEIGHTS.ADVERSARIAL > 0:
-            optimizer, optimizer_disc = optimizer
-
-        batch_size = batch['img'].shape[0]
-        output = self.forward_step(batch, train=True)
-        pred_mano_params = output['pred_mano_params']
-        if self.cfg.get('UPDATE_GT_SPIN', False):
-            self.update_batch_gt_spin(batch, output)
-        loss = self.compute_loss(batch, output, train=True)
-        if self.cfg.LOSS_WEIGHTS.ADVERSARIAL > 0:
-            disc_out = self.discriminator(pred_mano_params['hand_pose'].reshape(batch_size, -1), pred_mano_params['betas'].reshape(batch_size, -1))
-            loss_adv = ((disc_out - 1.0) ** 2).sum() / batch_size
-            loss = loss + self.cfg.LOSS_WEIGHTS.ADVERSARIAL * loss_adv
-
-        # Error if Nan
-        if torch.isnan(loss):
-            raise ValueError('Loss is NaN')
-
-        optimizer.zero_grad()
-        self.manual_backward(loss)
-        # Clip gradient
-        if self.cfg.TRAIN.get('GRAD_CLIP_VAL', 0) > 0:
-            gn = torch.nn.utils.clip_grad_norm_(self.get_parameters(), self.cfg.TRAIN.GRAD_CLIP_VAL, error_if_nonfinite=True)
-            self.log('train/grad_norm', gn, on_step=True, on_epoch=True, prog_bar=True, logger=True)
-        optimizer.step()
-        if self.cfg.LOSS_WEIGHTS.ADVERSARIAL > 0:
-            loss_disc = self.training_step_discriminator(mocap_batch, pred_mano_params['hand_pose'].reshape(batch_size, -1), pred_mano_params['betas'].reshape(batch_size, -1), optimizer_disc)
-            output['losses']['loss_gen'] = loss_adv
-            output['losses']['loss_disc'] = loss_disc
-
-        if self.global_step > 0 and self.global_step % self.cfg.GENERAL.LOG_STEPS == 0:
-            self.tensorboard_logging(batch, output, self.global_step, train=True)
-
-        self.log('train/loss', output['losses']['loss'], on_step=True, on_epoch=True, prog_bar=True, logger=False)
-
-        return output
-
-    def validation_step(self, batch: Dict, batch_idx: int, dataloader_idx=0) -> Dict:
-        """
-        Run a validation step and log to Tensorboard
-        Args:
-            batch (Dict): Dictionary containing batch data
-            batch_idx (int): Unused.
-        Returns:
-            Dict: Dictionary containing regression output.
-        """
-        # batch_size = batch['img'].shape[0]
-        output = self.forward_step(batch, train=False)
-        loss = self.compute_loss(batch, output, train=False)
-        output['loss'] = loss
-        self.tensorboard_logging(batch, output, self.global_step, train=False)
-
-        return output

WiLoR/wilor/utils/__init__.py

@@ -1,25 +0,0 @@
-import torch
-from typing import Any
-
-from .renderer import Renderer
-from .mesh_renderer import MeshRenderer
-from .skeleton_renderer import SkeletonRenderer
-from .pose_utils import eval_pose, Evaluator
-
-def recursive_to(x: Any, target: torch.device):
-    """
-    Recursively transfer a batch of data to the target device
-    Args:
-        x (Any): Batch of data.
-        target (torch.device): Target device.
-    Returns:
-        Batch of data where all tensors are transfered to the target device.
-    """
-    if isinstance(x, dict):
-        return {k: recursive_to(v, target) for k, v in x.items()}
-    elif isinstance(x, torch.Tensor):
-        return x.to(target)
-    elif isinstance(x, list):
-        return [recursive_to(i, target) for i in x]
-    else:
-        return x