Code for paper "REALY: Rethinking the Evaluation of 3D Face Reconstruction"

Code for paper "REALY: Rethinking the Evaluation of 3D Face Reconstruction"
Abstract: The evaluation of 3D face reconstruction results typically relies on a rigid shape alignment between the estimated 3D model and the ground-truth scan. We observe that aligning two shapes with different reference points can largely affect the evaluation results. This poses difficulties for precisely diagnosing and improving a 3D face reconstruction method. In this paper, we propose a novel evaluation approach with a new benchmark REALY, consists of 100 globally aligned face scans with accurate facial keypoints, high-quality region masks, and topology-consistent meshes. Our approach performs region-wise shape alignment and leads to more accurate, bidirectional correspondences during computing the shape errors. The fine-grained, region-wise evaluation results provide us detailed understandings about the performance of state-of-the-art 3D face reconstruction methods. For example, our experiments on single-image based reconstruction methods reveal that DECA performs the best on nose regions, while GANFit performs better on cheek regions. Besides, a new and high-quality 3DMM basis, HIFI3D++, is further derived using the same procedure as we construct REALY to align and retopologize several 3D face datasets. We will release REALY, HIFI3D++, and our new evaluation pipeline at this https URL.

REALY Benchmark

This is the official repository for 3D face reconstruction evaluation on the Region-aware benchmark based on the LYHM Benchmark (REALY). The REALY benchmark aims to introduce a region-aware evaluation pipeline to measure the fine-grained normalized mean square error (NMSE) of 3D face reconstruction methods from under-controlled image sets.

Evaluation Metric

Given the reconstructed mesh from the 2D image in REALY by a specific method, the REALY benchmark calculates the similarity of ground-truth scans on four regions (nose, mouth, forehead, cheek) with the predicted mesh. The detailed evaluation pipeline is available in the REALY paper.

REALY: Rethinking the Evaluation of 3D Face Reconstruction.
Zenghao Chai*, Haoxian Zhang*, Jing Ren, Di Kang, Zhengzhuo Xu, Xuefei Zhe, Chun Yuan, and Linchao Bao (* Equal contribution)
ECCV 2022
Project Page: https://www.realy3dface.com/
arXiv: https://arxiv.org/abs/2203.09729

Requirements

This evaluation implementation is tested under Windows, macOS, and Ubuntu environments. NO GPU is required.

Installation

Clone the repository and set up a conda environment with all dependencies as follows:

git clone https://github.com/czh-98/REALY
cd REALY
conda env create -f environment.yaml
conda activate REALY

  • NOTE: for Windows, you need to install scikit-sparse according to the guidline here.

Evaluation

1. Data Preparation

  • Obtain the access of Headspace dataset. Please sign the Agreement according to their guideline, after getting the permission, send the reply message to Zenghao Chai. Then, we will send the download url to you.

  • Download the benchmark data, and put the "REALY_HIFI3D_keypoints/" and "REALY_scan_region/" folder into "REALY/data/".

  • Use the images from REALY to reconstruct 3D meshes with your method(s). We provide the cropped and the original + depth map versions (512x512), respectively. You may use them according to your need.

  • [Important] Please save meshes as "*.obj", where "*" should have the same name as input images. NOTE: REALY is only suitable for meshes with the same topology. Please make sure the saved meshes share the same topology as your template mesh (e.g., if you use Trimesh to save meshes, please check whether you have set "process=False".)

2. Keypoints Preparation

  • [Important] To make a more accurate alignment, we extend 68 keypoints to 85 with additional keypoints in the facial cheek. Prepare the 85 barycentric keypoints file. The example of HIFI3D topology can be found at "REALY/data/HIFI3D.obj" and corresponding barycentric coordinate "REALY/data/HIFI3D.txt".

  • [Optional] NOTE: If you use one of the same template(s) as the methods we compared in the paper (e.g., BFM09, Deep3D, 3DDFA_v2, FLAME, HIFI3D, etc.), you can use the predefined keypoints in "REALY/data/" folder directly; or you do not know how to export the barycentric file, you may ignore this step, and send one template mesh (".obj" file) to Zenghao Chai, and then the barycentric file will be sent back to you.

  • Put your template mesh "*.obj" and corresponding barycentric coordinate "REALY/data/*.txt" into "/REALY/data/".

3. Evaluation

  • To evaluate the results on the frontal/multi-view image sets, run
python main.py --REALY_HIFI3D_keypoints ./data/REALY_HIFI3D_keypoints/ --REALY_scan_region ./data/REALY_scan_region --prediction <PREDICTION_PATH> --template_topology <TEMPLATE_NAME> --scale_path ./data/metrical_scale.txt --save <SAVE_PATH>
  • Wait for the evaluation results, the NMSE of each region will be saved at "<SAVE_PATH>/REALY_error.txt", and the global aligned, regional aligned SP*, deformation SH*, and error map will be saved at "<SAVE_PATH>/region_align_save/".

  • [Optional] If you want to present your method(s) on REALY, please send the reconstructed meshes and barycentric coordinate files to us, and we will re-evaluate and check the results. After that, we will update the project page accordingly.

HIFI3D++

If you want to use the 3DMM introduced in this paper, please refer to the instructions and demos.

Contact

If you have any question, please contact Zenghao Chai or Linchao Bao.

Citation

If you use the code or REALY evaluation pipeline or results in your research, please cite:

@inproceedings{REALY,
  title={REALY: Rethinking the Evaluation of 3D Face Reconstruction},
  author={Chai, Zenghao and Zhang, Haoxian and Ren, Jing and Kang, Di and Xu, Zhengzhuo and Zhe, Xuefei and Yuan, Chun and Bao, Linchao},
  booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
  year = {2022}
}

Download Source Code

Download ZIP

Paper Preview

Aug 19, 2022