Code for paper "Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wild"

Code for paper "Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wild"
Abstract: Recognizing scenes and objects in 3D from a single image is a longstanding goal of computer vision with applications in robotics and AR/VR. For 2D recognition, large datasets and scalable solutions have led to unprecedented advances. In 3D, existing benchmarks are small in size and approaches specialize in few object categories and specific domains, e.g. urban driving scenes. Motivated by the success of 2D recognition, we revisit the task of 3D object detection by introducing a large benchmark, called Omni3D. Omni3D re-purposes and combines existing datasets resulting in 234k images annotated with more than 3 million instances and 97 categories.3D detection at such scale is challenging due to variations in camera intrinsics and the rich diversity of scene and object types. We propose a model, called Cube R-CNN, designed to generalize across camera and scene types with a unified approach. We show that Cube R-CNN outperforms prior works on the larger Omni3D and existing benchmarks. Finally, we prove that Omni3D is a powerful dataset for 3D object recognition, show that it improves single-dataset performance and can accelerate learning on new smaller datasets via pre-training.

Omni3D & Cube R-CNN

Support Ukraine

Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wild

Garrick Brazil, Julian Straub, Nikhila Ravi, Justin Johnson, Georgia Gkioxari

[Project Page] [arXiv] [BibTeX]

Zero-shot (+ tracking) on Project Aria data Aria demo video

Predictions on COCO COCO demo

Installation Requirements

# setup new evironment
conda create -n cubercnn python=3.8
source activate cubercnn

# main dependencies
conda install -c fvcore -c iopath -c conda-forge -c pytorch3d-nightly -c pytorch fvcore iopath pytorch3d pytorch=1.8 torchvision cudatoolkit=10.1

# OpenCV, COCO, detectron2
pip install cython opencv-python
pip install 'git+'
python -m pip install detectron2 -f

# other dependencies
conda install -c conda-forge scipy seaborn

We used cuda/10.1 and cudnn/v7.6.5.32 for our experiments, but expect that slight variations in versions are also compatible.


Run Cube R-CNN on a folder of input images using our DLA34 model trained on the full Omni3D dataset. See our Model Zoo for more model variations.

# Download example COCO images
sh demo/

# Run an example demo
python demo/ \
--config cubercnn://omni3d/cubercnn_DLA34_FPN.yaml \
--input-folder "datasets/coco_examples" \
--threshold 0.25 --display \
MODEL.WEIGHTS cubercnn://omni3d/cubercnn_DLA34_FPN.pth \
OUTPUT_DIR output/demo 

See for more details.

Training on Omni3D

Coming soon!

Inference on Omni3D

Coming soon!


Cube R-CNN is released under CC-BY-NC 4.0


Please use the following BibTeX entry if you use Omni3D and/or Cube R-CNN in your research or refer to our results.

  author =       {Garrick Brazil and Julian Straub and Nikhila Ravi and Justin Johnson and Georgia Gkioxari},
  title =        {{Omni3D}: A Large Benchmark and Model for {3D} Object Detection in the Wild},
  journal =      {arXiv:2207.10660},
  year =         {2022}

Download Source Code

Download ZIP

Paper Preview

Jul 26, 2022