Accurate registration of virtual objects into a real environment is an outspoken problem in Augmented Reality(AR). This problem needs to be solved regardless of the complexity of the virtual objects one wishes to enhance the real environment with. Both simple text annotations and complex virtual mimics of real-life objects need to be placed rigidly into the real environment. Augmented reality systems that lack this requirement will demonstrate serious `jittering' of virtual objects in the real environment and will therefore fail to give the user a real-life impression of the augmented outcome.
The registration problem has already been tackled by several researchers in the AR-domain. A general discussion of all coordinate frames that need to be registered with each other can be found in [204]. Some researchers use predefined geometric models of real objects in the environment to obtain vision-based object registration [79,171,209]. However, this delimits the application of such systems because geometric models of real objects in a general scene are not always readily available. Other techniques have been devised to make the calibration of the video camera obsolete by using affine object representations [94]. These techniques are simple and fast but fail to provide a real impression when projective skew is dominant in the video images. Therefore virtual objects can be viewed correctly only from large distances where the affine projection model is almost valid. So it seems that the most flexible registration solutions are those that don't depend on any a priori knowledge of the real environment and use the full perspective projection model. Our AR-System belongs to this class of flexible solutions.
To further enhance the real-life impression of an augmentation the occlusion and illumination problems need to be solved. The solutions to the occlusion problem are versatile. They differ in whether a 3D reconstruction of the real environment is needed or not [11,19]. Also the illumination problem has been handled in different ways. A first method uses an image of a reflective object at the place of insertion of the virtual object to get an idea of the incoming light at that point [33]. A second approach obtains the total reconstruction of a 3D radiance distribution by the same methods used to reconstruct a 3D scene [164]. Another approach consists of the approximation of the illumination distribution by a sphere of illumination directions at infinity [165].
As computer generated graphics of virtual objects are mostly created with non physically-based rendering methods, techniques that use image-based rendering can be applied to incorporate real objects into another real environment [173] to obtain realistic results.
However, the `jittering' of virtual objects in the real environment can degrade the final augmented result severely, even if problems of occlusion and illumination can be resolved exactly. We focussed on developing an AR-System that solves the registration problem as a prerequisite. It is based primarily on a 3D reconstruction scheme that extracts motion and structure from uncalibrated video images and uses the results to incorporate virtual objects into the real environment.