The existing state-of-the art in structure-from-motion systems is the construction of sparse feature-based scene representations, e.g. from points and lines. The main drawback of such systems is the lack of surface information, which restricts their usefulness. Although it is possible to build surface modelling on top of the feature information, we have designed an algorithm that allows surface information to be built directly into the reconstruction algorithm, so that along with features, surface parameters may be computed. Constraints between surfaces and features, and between the surfaces themselves, may be incorporated.

Our work may be seen as extending existing photogrammetric bundle adjustment algorithms to compute surface parameters. We employ the recursive partitioning algorithm, well-known in the photogrammetry community, to obtain efficient iterative updates of the reconstruction. Incorporating the surface constraints modifies the sparse structure of the normal equations, making it important to order the feature, surface and motion parameter blocks appropriately to achieve the best performance. We employ the Variable State Dimension Filter (VSDF) to effect both batch and recursive updates within the same framework.

Another novel aspect of our work is the method we use to select the global coordinate frame in which to represent the scene and the camera motion. Existing schemes both in photogrammetry and computer vision have drawbacks, in that the results of a bundle adjustment algorithm depend on the initially chosen coordinate frame, or on the order of the images. Our scheme, which eliminates both these effects, involves first normalizing an initial reconstruction to achieve coordinate frame invariance, and then selecting gauge constraints on the parameter updates so that the normalization conditions applied are maintained to first order by the bundle adjustment iteration. Or results suggest that this technique achieves faster and more stable convergence than existing methods.