Simon’s Blog

Lots of people think search is done. Google has won the war. Yahoo! And Microsoft are but a poor second. However, Microsoft’s Robert Scoble is pretty vocal about the need for continued innovation in search. And just recently, I was reminded of a category of search that simply doesn’t work at all in the search engines; highlighting the fact that, in truth, we’re a million miles away from searching being done.

The other day, I was talking with an up-and-coming rock singer from Los Angeles, California. He’s a major talent, and people are starting to notice what he’s doing: his music gets played on radio stations in the US and the UK; eight of the tracks on his debut album have been picked up by TV and film companies; he just won the award for “Best Singer Songwriter” at the LA Weekly Music Awards; and he tours constantly in the US and Europe (he’s totally awesome live, by the way, so if you get a chance to see him live up close and personal in the venues he currently plays – you should grab it while you can!). The point is: there’s quite a few people out there want to visit his web-site.

But here’s the deal. He’s always being told – “Do you know that you’re completely impossible to Google?” And the truth is, with the kind of searches that 99% of people do, you just can’t find his web-site. Why? Because his name is a really common word: “AM” (pronounced as in “3am” not as in “I am blogging”).

This kind of search exposes a fundamental flaw in the keyword approaches to search that the major search engines use; where even things like page rank algorithms aren’t helpful. I’ve blogged about next generation search challenges before – and to quote myself from an article I wrote in 2001 on natural language understanding:

Keyword-based methods tend not to be helpful in cases in which a large amount of information needs to be searched. Searches are highly sensitive to small changes in the formulation of a query, the output is not always ranked in order of relevancy, results may be misleading because important results are missed, and large numbers of false positive results may be returned.

Now of course, it would be rather unreasonable to expect a search engine to know what you’re looking for if you just type – AM - into a search box. That could mean anything. But, it’s not unreasonable to want to type - Rock singer named AM - into a search box and have sensible search results returned. The problem is, for this kind of query, keyword approaches are just never going to work.

Now, what do I mean by “it’s not unreasonable” to want to do this? I mean simply this: as a user of a search engine, I would like to be able to do searches like that and get good results. I’m fully aware that 99% of developers will think – “But this is impossible. It’s much too hard a problem to solve”. Well, I don’t buy that it’s impossible. I just think it’s tough nut to crack. I don’t think we can say search is done until we can start to extract semantic meaning from both queries and web pages in a useful way.

Anyhow, maybe you’re better at using search engines than most people. If you have a spare 30 seconds, see if you can find AM’s web-site! (it’s here, if you want to cheat!).