Abstract: In this paper, we consider the challenging task of simultaneously locating and recovering multiple hands from single 2D image. Previous studies either focus on single hand reconstruction or solve this problem in a multi-stage way. Moreover, the conventional two-stage pipeline firstly detects hand areas, and then estimates 3D hand pose from each cropped patch. To reduce the computational redundancy in preprocessing and feature extraction, we propose a concise but efficient single-stage pipeline. Specifically, we design a multi-head auto-encoder structure for multi-hand reconstruction, where each head network shares the same feature map and outputs the hand center, pose and texture, respectively. Besides, we adopt a weakly-supervised scheme to alleviate the burden of expensive 3D real-world data annotations. To this end, we propose a series of losses optimized by a stage-wise training scheme, where a multi-hand dataset with 2D annotations is generated based on the publicly available single hand datasets. In order to further improve the accuracy of the weakly supervised model, we adopt several feature consistency constraints in both single and multiple hand settings. Specifically, the keypoints of each hand estimated from local features should be consistent with the re-projected points predicted from global features. Extensive experiments on public benchmarks including FreiHAND, HO3D, InterHand2.6M and RHD demonstrate that our method outperforms the state-of-the-art model-based methods in both weakly-supervised and fully-supervised manners. The code and models are available at {\url{this https URL}}.

SMHR: Single-stage Multiple Hand Reconstruction with weak supervision

Introduction

• This repo is official PyTorch implementation of End-to-end Weakly-supervised Multiple 3D Hand Mesh Reconstruction from Single Image.

Overview of the proposed framework

Install

• Environment

    conda create -n SMHR python=3.7
    conda activate SMHR

    # If you failed to install pytorch, you may try to modify your conda source: https://mirrors.tuna.tsinghua.edu.cn/help/anaconda/
    conda install -c pytorch pytorch==1.6.0 torchvision cudatoolkit=10.1
    
    # install pytorch3d from source if you are not using latest pytorch version
    conda install -c fvcore -c iopath -c conda-forge fvcore iopath
    conda install -c bottler nvidiacub
    conda install pytorch3d -c pytorch3d

    pip install -r requirements.txt    

Directory

Root

The ${ROOT} is described as below.

${ROOT}
|-- data
|-- lib
|-- outputs
|-- scripts
|-- assets

• data contains packaged dataset loaders and soft links to the datasets' image and annotation directories.
• lib contains main codes for SMHR, including dataset loader code, network model, training code and other utils files.
• scripts contains running scripts.
• outputs contains log, trained models, imgs, and pretrained models for model outputs.
• assets contains demo images.

Data

You need to follow directory structure of the data as below. (recommend to use soft links to access the datasets)

${ROOT}
|-- data
|   |-- FreiHAND
|   |   |-- evaluation
|   |   |-- training
|   |   |***.json
|   |-- HO3D
|   |   |-- evaluation
|   |   |-- train
|   |   |***.txt
|   |-- MultiHand
|   |***.pth 
|   |***.pkl
|   |joint_dataset.txt

• Download the FreiHAND dataset from the [website]
• Download the HO3D dataset from the [website]
• Download pre-trained models and dataset loaders here [cloud] (Extraction code: krg9) and put them in the data folder.

Demo on single/multiple hand images

Single Hand dataset(FreiHAND)

• Modify scripts/demo.sh to Single-FreiHAND setting. task = 'simplified',dataset='FreiHAND',Input_RES=224,CHECKPOINTS_PATH='${ROOT}/data/FreiHAND-2d-simplified.pth', comment out #L30 '--pick_hand'
• Modify demo.py to use images from FreiHAND dataset. #L80: base_dir = 'assets/Single/FreiHAND'
• run bash scripts/demo.sh
• You can see rendered outputs in outputs/FreiHAND/.

Single Hand dataset(HO3D)

• Modify scripts/demo.sh to Single-HO3D setting. task = 'simplified',dataset='HO3D',Input_RES=224,CHECKPOINTS_PATH='${ROOT}/data/HO3D-2d-simplified.pth', comment out #L30 '--pick_hand'
• Modify demo.py to use images from HO3D dataset. #L80: base_dir = 'assets/Single/HO3D'
• run bash scripts/demo.sh
• You can see rendered outputs in outputs/HO3D/.

Multiple Hand (Joint dataset)

• Modify scripts/demo.sh to Multi-hand setting. task = 'artificial',dataset='Joint',Input_RES=384,CHECKPOINTS_PATH='${ROOT}/data/FreiHAND-2d-artificial.pth', uncomment #L30 '--pick_hand'
• Modify demo.py to use multi-hand images. #L80: base_dir = 'assets/Multi'
• run bash scripts/demo.sh
• You can see rendered outputs in outputs/Multi/.

Training and Evaluation

Coming Soon...

Outputs of Single Hand images

Outputs of Multiple Hand images

Citation

If you find our work useful in your research, please consider citing:

@article{Ren2022EndtoendWM,
  title={End-to-end Weakly-supervised Multiple 3D Hand Mesh Reconstruction from Single Image},
  author={Jinwei Ren and Jianke Zhu and Jialiang Zhang},
  journal={ArXiv},
  year={2022},
  volume={abs/2204.08154}
}

Download Source Code

Paper Preview