Complex Correlation Statistic for Sub-pixel Stereovision

The dense stereo algorithm usually involves two steps:

1. Compute a correlation table, i.e. evaluate a similarity statistic for each candidate pair.
2. Run a matching algorithm on the correlation table, i.e. establish matches from high similarity candidate pairs.

The importance of high quality correlation statistic is undisputable. Traditional usage of  windowed pixel intensity statistics (Sum of Squared Differences, Normalized Cross-Correlation, etc.) suffers from many artifacts. A big obstacle is an image discretization. Simple correlation statistics are corrupted because of image sampling. For illustration, here is a 1D example of  two sine waves:

Sampling in the right (image) signal is 0.5 pixel shifted from the left one. The Moravec's Normalized Cross Correlation [4]  has the value 0.8 only (not 1) and is even lower with increasing frequency, the Birchfield-Tomasi's statistic [1]  is sampling insensitive and has the value 1.0, our Complex Correlation Statistic (CCS) [2] is also sampling invariant and has the value 1.0, moreover it estimates the sub-pixel offset too.

The basic idea behind is to represent an image point neighbourhood as a response to a bank of Gabor filters. The images are convolved with the filter bank and the CCS is evaluated from the responses - in a closed form, without iterations. The CCS is a complex number

CCS = A e ^jd,

where the magnitude A is the similarity value and the phase d is the sub-pixel position of the correlation maximum. The magnitude A is invariant to image sampling and it is used as a similarity statistic for integer matching and the phase d provides a natural estimate of the sub-pixel disparity. The details can be found in our paper [2].

According to the experiments we carried out on both synthetic and real data we conclude the CCS is highly discriminable compared to standard window statistics and the sub-pixel accuracy is higher than the Maximum Likelihood sub-pixel window-fitting disparity correction method [5] we compared with. Results on the real data follow. More experiments with quantitative evaluation can be found in the paper [2].

left image	MNCC+dc	CCS

We used Confidently Stable Matching algorithm [6] for integer matching. In the leftmost column there is a left rectified image. Next two columns are the results as color-coded disparity maps (gray color stands for unassigned disparity). The middle column are the results for the matching with Moravec Normalized Cross Correlation [4] followed by the post processing sub-pixel disparity correction method [5] (MNCC+dc), the last column are the disparity maps from our method, i.e. integer matching on the CCS magnitude and the sub-pixel disparity estimate from the CCS phase. First row is a laboratory test scene consisting of thin stripes, other two are outdoor scenes.

The CCS disparity maps are significantly denser, which is due to higher discriminability of the statistic. The MNCC has problems where the sub-pixel disparity is about 0.5px - see gray contours between disparity levels. The CCS does not suffer with this problem at all, because its sub-pixel resolution increases the discriminability. The sub-pixel accuracy of the CCS is also higher. Notice, an inscription above the door (last row) which is sub-pixel in disparity and it is still visible with the CCS.

To conclude, the main advantages of using the CCS are:

	Discrete algorithm with sub-pixel precision.  (Closed-form formula for CCS, no iterative optimization.)
	Sampling invariance.
	High accuracy and discriminability.

References