This site is devoted to mathematics and its applications. Created and run by Peter Saveliev.

Image-to-image search

From Mathematics Is A Science
Jump to navigationJump to search

... as opposed to text-to-text and text-to-image we are familiar with (See also Pixcavator image search).

In this article we will review some attempts to create a visual image search engine. The main interest is in is the image search technology that is independent of color (and of course independent of tags). In some areas, such as radiology, all images are gray scale.


CBIR = Content based image retrieval

CBIR is based on collecting "descriptors" (texture, color histogram, etc) from the image and then looking for images with similar descriptors.

The question is, This can work but what would guarantee that it will? More narrowly, what information about the image should be passed to the computer to ensure that the computer will succeed in finding good matches?

Option 1: we pass all information. Then, this could work. For example, the computer represents every 100x100 image a point in the 10,000-dimensional space and then runs clustering. First, this may be impractical and, second,.. does it really work? Will the one-object images form a cluster? Or maybe a hyperplane? One thing is clear, these images will be very close to the rest because the difference between one object and two may be just a single pixel.

Option 2: we pass some information. What if you pass information that cannot possibly help to classify the images the way we want? For example, you may pass just the value of the (1,1) pixel, or the number of 1’s, or 0’, or their proportion. Who will make sure that the relevant information (adjacency) isn’t left out? Computer doesn’t know what is relevant – it’s hasn’t learned “yet”. If it’s the human, then he would have to solve the problem first, if not algorithmically then at least mathematically.

Thus, CBIR shares some of the problems of machine learning.


  • Visual image search engines exist only as experimental prototypes (demos, toys, etc). Worse yet, many make broad claims with nothing to back them up. If you have the technology, how hard is it to create a little web based demo!
  • Most demos work with small collections of images, with no upload feature, which makes testing impossible.
  • When testing is possible, the results are questionable.
  • The only well developed approaches are based on the distribution of colors, texture, and image segmentation.
  • In the image search context, Fourier transform and wavelet transform have been mostly used for image compression.
  • The engines provide only “likeness” search, which is very subjective and obscures the fact the image analysis methods are inadequate.
  • “User feedback”, “learning”, and “semantic” features obscure the fact the image analysis methods are inadequate.

Compare to Object recognition.

See also Other software projects.