Abstract: Neural fields have emerged as a new data representation paradigm and have shown remarkable success in various signal representations. Since they preserve signals in their network parameters, the data transfer by sending and receiving the entire model parameters prevents this emerging technology from being used in many practical scenarios. We propose streamable neural fields, a single model that consists of executable sub-networks of various widths. The proposed architectural and training techniques enable a single network to be streamable over time and reconstruct different qualities and parts of signals. For example, a smaller sub-network produces smooth and low-frequency signals, while a larger sub-network can represent fine details. Experimental results have shown the effectiveness of our method in various domains, such as 2D images, videos, and 3D signed distance functions. Finally, we demonstrate that our proposed method improves training stability, by exploiting parameter sharing.

Streamable Neural Fields

Paper link

Junwoo Cho*, Seungtae Nam*, Daniel Rho, Jong Hwan Ko, Eunbyung Park†
* Equal contribution, alphabetically ordered.
† Corresponding author.

European Conference on Computer Vision (ECCV), 2022

0. Requirements

Setup a conda environment using commands below:

conda env create -f environment.yml
conda activate snf

1. Dataset

Download Kodak dataset from here.

Download UVG dataset from here.
When downloading UVG video, please use this version:

• Resolution: 1080p
  
• Bit depth: 8
  
• Format: AVC
  
• Container: MP4
  

Download 3D point cloud dataset from here.

'data/' directory must be in your working directory. The structure is as follows:

Data layout

data/
    kodak/
        kodim01.png
        ...
        kodim24.png
    shape/
        armadillo.xyz
        dragon.xyz
        happy_buddha.xyz
    uvg/
        Beauty.mp4
        ...
        YachtRide.mp4

2. Reproducing experiments

Run the commands below.

Image spectral growing

bash scripts/train_image_spectral.sh

Image spatial growing

bash scripts/train_image_spatial.sh

Video temporal growing

bash scripts/train_video_temporal.sh

SDF spectral growing

bash scripts/train_sdf_spectral.sh

3. Results

You can find both qualitative and quantitative results in \results directory.

Download Source Code

Paper Preview