We are developing the social individualist meta-context for the future. From the very serious to the extremely frivolous... lets see what is on the mind of the Samizdata people.

Samizdata, derived from Samizdat /n. - a system of clandestine publication of banned literature in the USSR [Russ.,= self-publishing house]

Are there (or will there ever be) search engines for pictures?

Friedrich Blowhard’s latest and pleasingly whimsical posting is called Visual Google. What he was doing was typing in words, albeit words with visual connotations and consequences. Hello “clouds”. Hello “sky”.

Says Friedrich:

It may be an exaggeration to describe a Google search as “found art” but I generally like the results at least as well as a John Cage musical composition.

Indeed.

But now here’s what I thought Friedrich might have been writing about. For some time now I’ve been wondering how you search the net for a picture, when all you have to go on is a bit of picture yourself. Suppose you have a rather blurry or unsatisfactory image, or perhaps a fragment of an image, or maybe a quite good drawing of an image, and you want the Giant Computer in the Sky to tell you what it is, and to show you a far, far better version of it … can you now do that? Are there truly visual search engines out there? And how about a visual description (“cubist woman, with transparent handkerchief in front of her face, crying, lots of yellow, red and blue”) but not the official title? Can search engines now – or will search engines ever be able to – respond intelligently to a query like that?

And how about music? Can you now, or will there ever be a day when you can, go “you know that thing that goes Dah dah de dah dit kabang swoosh …” and get five suggestions for the original track listed for you and ready to roll?

9 comments to Are there (or will there ever be) search engines for pictures?

  • Dale Amon

    For music there is a mobile phone service. You can ring it, hold up your phone towards the sound source and I think you get the title back in an SMS.

    In fact, didn’t someone blog it here? No idea when could have been anywhere from year 1 AS (After Samizdata) to the present.

    As for pictures, there are laser holographic memories that can do some fairly serious picture lookup. Probably some research work on the lookup by description but I’ve not been following it much lately.

    I’ll bet our spook readers have access to it but they’d have to shred us all if they told us.

  • There is a ‘pictures’ option under Google!

  • Peter

    All you are asking for is a solution to the problem of Artificial Intelligence.

    To do properly all the things you describe, is essentially to do AI. A lot of people worked (and still work) on these problems but they are essentially still unsolved.

    What you need is some kind of internal “semantic” representation of objects, and then you need a “neighbour search”. So you give it your picture or tune, it turns it into an internal representation and then finds the objects that have a similar representation. The difficult part is to do this encoding in a way that conceptually similar (but superficially totally different) pictures, sounds, etc. end up being similar in their representation.

    I am sure you could do it on some level even today, but you may only capture some superficial structure (e.g. lots of red diagonal texture in the upper left corner). but don’t expect it to find pictures of cats if you show it a picture of a cat anytime soon.

  • mike

    They’re working on one for searching out similar technical drawings at work. Will it ever work properly? I dunno. But our mainframe room’s impressive (& there’s 2 of them!)

    Where’s work? Let’s just say that we do a lot of searching of technical drawings….

  • Andy Duncan

    Hi Brian,

    I think they’ve currently got neural networks with up to 10,000 neurons in them, for AI, which is about the same amount of brainpower as a bee.

    I’m afraid it may be some time before they get up to the level of an Oxford Street music shop assistant.

    [please insert own inoffensive joke]

    I did work on something called an Illustra database, a few years ago, which used some weird technology generated by Professor Michael Stonebreaker to store searchable image/video/sound information in object-relational DBMSs. Illustra got bought out by Informix, who got bought out by IBM (and their DB2 database product).

    Though I think this got nowhere. But you can read about it, here.

    You could do a google on Object Relational DBMS, to see where they ended up?

    So until either Object-Relational DBMSs arise from the grave, or AI gets bigger than a bee brain, we’re going to have to stick to the ‘Pictures’ option under Google! 🙂

    Rgds, andyD

  • TheWobbly Guy

    I particularly like the idea of a music identification engine.

  • Visual dictionaries have been available for a while this one for example. They attempt to address the “visual search” problem in a traditional book form, without electronics.

    It may be that a useful visual search/online-dictionary could be created without a lot of AI, simply by identifying enough visual vectors and coding all the entries relative to them. For instance, the predominance of particular shapes, colors, or textures, the basic subject of the picture (landscape, portrait, still-life) etc. Then, a wizard-style front end could help you to describe the thing to be searched for (or added to the dictionary) in terms of those vectors.

    I agree that AI would be needed to solve the general problem — automatically classifying an arbitrary graphic — but a process that involves human judgment could lead to a useful tool, sooner, while we await the solution to the AI problem.

    The architects of the web are already talking about the inclusion of semantic information with web objects, to allow for automated search by “meaning” and other non-literal qualities. I don’t imagine that an online “visual dictionary” is too far down the road, if it doesn’t exist already.

  • David Gillies

    Whenever anyone says that AI will ‘never’ solve problems like this, they’re thinking linearly, not exponentially, and will probably be proved wrong. If anyone says AI will solve problems like this in the near future, count your teaspoons after they’ve left: they’re trying to sell you something.

  • tbrosz

    The Google image search function mentioned earlier can be found here. It’s probably not as intelligent as what you are looking for, but it’s damn useful.