NEWS

Join the SIX Challenge Google group to stay up to date.

Introduction

A pose of a rigid object has six degrees of freedom and its full knowledge is required in many robotic and augmented reality applications. The goal of the challenge is to evaluate methods for 6D object pose estimation from RGB or RGB-D images and to establish the state of the art.

Challenge task

The methods are evaluated on the task of 6D localization of a single instance of a single object. All test images are used for the evaluation, even those with multiple instances of the object of interest.

This task, which is a special variant of the 6D localization task described in [1], allows to evaluate most of the state-of-the-art methods out of the box. The task is relevant for industry, e.g. for an assembly robot when it needs to find a bolt to complete an assembly step. Even if there are multiple bolts in its workspace, the robot needs to know the pose of a single bolt, arbitrarily chosen.

The difficulty of the "multiple instances, find one that you pick" is close to "find the instance in most favorable pose" (least occlusion, unambiguous view). Most methods are expected, but not required, to report the most favorable pose, treating the rest as clutter.

Datasets

The datasets selected for the challenge were converted to a standard format. All contain 3D object models and training and test RGB-D images. The training images show individual objects from different viewpoints and were either captured by a Kinect-like sensor or obtained by rendering of the 3D object models. The test images were captured in scenes with varying complexity, often with clutter and occlusion, and are fully annotated (the ground truth is provided for each instance of the modeled objects).

Hinterstoisser et al.
Hinterstoisser et al. [3] with extra ground truth from Brachmann et al. [4]
DOWNLOAD
T-LESS
T-LESS [2] - use Primesense images
DOWNLOAD
TUD Light
TUD Light
DOWNLOAD
Rutgers APC
Rutgers APC [7] - reduced version
DOWNLOAD
Tejani et al.
Tejani et al. [5] - reduced version
DOWNLOAD
Doumanoglou et al.
Doumanoglou et al. [6] - scenario 2
coming soon

How to participate

To have your method evaluated, run it on all of the datasets and submit the results in this format to hodantom@cmp.felk.cvut.cz. The submission deadline is 1st October 2017.

Challenge rules:

  1. For training, you can use the provided object models and training images (both real and rendered). You can also render extra training images using the object models.
  2. Not a single pixel of test images may be used in training, nor the ground truth poses provided for the test images.
  3. For each submission, keep the parameters of your method constant across all objects and datasets.
  4. If you want your results to be included in a publication about the challenge, a documentation of results is required. Without the documentation, your results will be listed on this website but not included in the publication.
  5. To become the winning method, authors need to provide also an implementation of the method (source code or a binary file) that will be validated. Besides the absolute winner, we will announce an open source winner selected from the methods whose source code is publicly available.

The 6D object pose estimates will be evaluated with two pose error functions: the Visible Surface Discrepancy [1] and the Average Distance [3].

We provide SIXD toolkit that includes python scripts for reading the standard dataset format, rendering, evaluation etc.

Organizers

Tomáš Hodaň1, Frank Michel2, Caner Sahin3, Tae-Kyun Kim3, Jiří Matas1, Carsten Rother2

1 Center for Machine Perception, Czech Technical University in Prague, Czech Republic
2 Computer Vision Lab, TU Dresden, Germany
3 Imperial Computer Vision & Learning Lab, Imperial College London, United Kingdom

References