Locally Rigid Models for 3D Scene Flow

Stefan Roth (TU Darmstadt, Germany)

3D scene flow estimation -- simultaneously recovering geometry and 3D motion from stereo video sequences -- remains a challenging task, despite much progress in both classical disparity and 2D optical flow estimation. To overcome the limitations of existing techniques, we introduce a novel model that represents the dynamic 3D scene by a collection of planar, rigidly moving, local segments. Scene flow estimation then amounts to jointly estimating the pixel-to-segment assignment, and the 3D position, normal vector, and rigid motion parameters of a plane for each segment. The proposed model combines an occlusion-sensitive data term with appropriate shape, motion, and segmentation regularizers. Inference is carried out using discrete fusion moves.

I will demonstrate the benefits of our model on different real-world image sets, including the challenging KITTI benchmark. In particular, the locally rigid scene representation enables 3D scene flow to outperform dedicated optical flow techniques at 2D motion estimation, thus for the first time realizing the theoretical advantage of having multiple views.

This is joint work with Christoph Vogel and Konrad Schindler.