Image search systems based on local descriptors typically achieve orientation
invariance by aligning the patches on their dominant orientations. Albeit
successful, this choice introduces too much invariance because it does not
guarantee that the patches are rotated consistently. We introduce an
aggregation
strategy of local descriptors that achieves this covariance property by jointly
encoding the angle in the aggregation stage in a continuous manner.

We show that the similarity between an image and the rotated counterpart of
another image forms a trigonometric polynomial with coefficients independent
from the rotation angle. In this fashion, we provide a very fast way to
evaluate
similarity for a large number of possible rotations.

We combine our strategy with an efficient monomial embedding to provide a
codebook-free method to aggregate local descriptors into a single vector
representation. It is also compatible and employed with several popular en-
coding methods, in particular bag-of-words, VLAD and the Fisher vector. Our
geometric-aware aggregation strategy is effective for image search, as shown by
experiments performed on standard benchmarks for image and particular object
retrieval, namely Holidays and Oxford buildings.