In the previous chapter it was seen how different views could be related to each other. In this chapter the relation between the views and the correspondences between the features will be used to retrieve the structure of the scene and the motion of the camera. This problem is called Structure and Motion. In this chapter, the structure and motion will only be determined up to an arbitrary projective transformation. The question how to restrict this ambiguity will be answered in the chapter on self-calibration.
At first two images are selected and an initial reconstruction frame is set-up. Then the pose of the camera for the other views is determined in this frame and each time the initial reconstruction is refined and extended. In this way the pose estimation of views that have no common features with the reference views also become possible. This approach was inspired by the work of Beardsley et al. [9]. Typically, a view is only matched with its predecessor in the sequence. In most cases this works fine, but in some cases (e.g. when the camera moves back and forth) it can be interesting to also relate a new view to a number of additional views. Once the structure and motion has been determined for the whole sequence, the results can be refined to achieve the best possible reconstruction.