Initialization and Alignment
for Adversarial Texture Optimization

ECCV 2022

Initialization and Alignment for Adversarial Texture Optimization. ECCV 2022.
Xiaoming Zhao, Zhizhen Zhao, and Alexander G. Schwing.

Project Page | Paper (coming soon)

Environment Setup
UofI Texture Scenes
Compile TexInit
Train
Evaluate
Citation

Environment Setup

conda env create -f environment.yml

We set the following environment variables for later usage:

cd /path/to/this/repo
CODE_ROOT=$PWD
export TEX_INIT_DIR=${CODE_ROOT}/tex_reconstruct/tex_init

export THIRDPARTY_DIR=${CODE_ROOT}/third_parties
mkdir -p ${THIRDPARTY_DIR}

And we use scene_04 as an example in this README:

export SCENE_ID=scene_04

PyTorch3D

We rely on PyTorch3D for rendering and rasterization. Please follow the official instruction to build it from source:

conda install -c conda-forge -c fvcore fvcore
conda install -c bottler nvidiacub

cd ${THIRDPARTY_DIR}
git clone https://github.com/facebookresearch/pytorch3d.git
cd pytorch3d && git checkout d07307a

pip install -e . --verbose

UofI Texture Scenes

We self-collect data with an iOS App developed based on ARKit (XCode 13.1). Please download and uncompress the dataset and place it under ${TEX_DIR}/dataset/raw. The structure should be:

.
+-- dataset
|  +-- raw
|  |  +-- scene_01
|  |  +-- scene_02
|  |  ...

Data Reader

The data is in binary format and consists of information for RGB, depth, camera matrices, and mesh. We provide a Python script to read data from it. Run the following command:

export PYTHONPATH=${CODE_ROOT}:$PYTHONPATH && \
python ${CODE_ROOT}/advtex_init_align/data/bin_data_reader.py \
--stream_dir ${CODE_ROOT}/dataset/raw/${SCENE_ID} \
--save_dir ${CODE_ROOT}/dataset/extraced/${SCENE_ID}

The extracted data will be saved with structure:

.
+-- dataset
|  +-- extracted
|  |  +-- scene_04
|  |  |  +-- mesh.ply  # file
|  |  |  +-- rgb       # folder
|  |  |  +-- depth     # folder
|  |  |  +-- mat       # folder

We provide a notebook to illustrate the format of the extracted data.

Compile TexInit

The code has been tested on Ubuntu 18.04 with GCC 7.5.0.

Install Dependencies

Install dependencies from Package Manager:

sudo apt-get install libmetis-dev libpng-dev libsuitesparse-dev libmpfr-dev libatlas-base-dev liblapack-dev libblas-dev

Manually install Eigen3.4:

cd ${THIRDPARTY_DIR}
wget https://gitlab.com/libeigen/eigen/-/archive/3.4/eigen-3.4.zip
unzip eigen-3.4.zip

Manuall install Boost 1.75.0. Please follow the official instruction to install Boost:

cd ${THIRDPARTY_DIR}
wget https://boostorg.jfrog.io/artifactory/main/release/1.75.0/source/boost_1_75_0.tar.bz2
tar --bzip2 -xf boost_1_75_0.tar.bz2

mkdir -p ${THIRDPARTY_DIR}/boost_1_75_0/build
cd ${THIRDPARTY_DIR}/boost_1_75_0/
./bootstrap.sh --prefix=${THIRDPARTY_DIR}/boost_1_75_0/build
./b2 install

Manually install CGAL 5.1.5. Please follow the official instruction:

cd ${THIRDPARTY_DIR}
wget https://github.com/CGAL/cgal/releases/download/v5.1.5/CGAL-5.1.5.zip
unzip CGAL-5.1.5.zip
cd ${THIRDPARTY_DIR}/CGAL-5.1.5 && mkdir install && mkdir build && cd build
cmake -DCGAL_HEADER_ONLY=OFF -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=../install ..
make
make install

Compile

Modify the Makefile based on your setup:

Set the variable THIRDPARTY_DIR to the value of ${THIRDPARTY_DIR};
Set the variable CUDA_ROOT to your CUDA's root directory, e.g., /usr/local/cuda-11.1.

Then use the following command to compile TexInit execution file:

cd ${TEX_INIT_DIR}
make tex_init DEBUG=0 -j 8

Train

We provide run.sh to illusrtate the overall pipeline. The following command will generate texture in ${CODE_ROOT}/experiments/${SCENE_ID}/optimized_texture_test_1_10, which can be directly used in 3D rendering engine, e.g., Blender and MeshLab.

bash ${CODE_ROOT}/run.sh \
  ${CODE_ROOT} \
  ${SCENE_ID} \
  run_train

Evaluate

The following command will compute and save quantitative results in ${CODE_ROOT}/experiments/${SCENE_ID}/test_1_10/eval_results.

bash ${CODE_ROOT}/run.sh \
  ${CODE_ROOT} \
  ${SCENE_ID} \
  run_eval

Citation

Xiaoming Zhao, Zhizhen Zhao, and Alexander G. Schwing. Initialization and Alignment for Adversarial Texture Optimization. ECCV 2022.

@inproceedings{zhao-tex2022,
  title = {Initialization and Alignment for Adversarial Texture Optimization},
  author = {Xiaoming Zhao and Zhizhen Zhao and Alexander G. Schwing},
  booktitle = {Proc. ECCV},
  year = {2022},
}