Is It Finally the End for Real-time Search Engines?

During the past four months we've seen not one but two well known real-time search engines disappear. First there was OneRiot, which in October 2010 decided to focus on advertising. More recently, Collecta closed it's real-time search engine and API to focus on alternative real-time products. Digging further into real-time search offerings you will also discover that crowdeye has also decided to pull its real-time search engine. This now appears to leave Topsy, and of course Google as the main players focusing on building a real-time search destination. Does this trend signal the end for all real-time search engines or just that their focus has been wrong?

There are a couple of quotes that seem to indicate what the problem has been for some of these real-time search engines. On the front page of crowdeye there is now the paragraph:

While we have been successful pushing the state of the art forward with features like location-based search, relevance sorting of results and sentiment, we have not yet built a profitable business around CrowdEye.

The OneRiot blog post that announced the move to focus on advertising states:

Now, of course, since OneRiot has been around we’ve been known as leaders in the realtime search space. In reality, we’ve been in the market with two products that leverage the same underlying technology platform: consumer-facing realtime search and an innovative advertising product that monetizes both realtime search the wider realtime social web. Our advertising platform has taken off like a rocket – both in terms of network growth and the number of advertisers who are seeking to engage with the social influencers across that network.

The problem is monetization. Isn't it always!

As the real-time web exploded we saw a bunch of real-time search engines appear. That list has significantly decreased in size and of those that still exist at all quite a few have shifted focus.

So the question is what could they have done to monetize their real-time search engine product? The obvious solution is advertising and Google have already proven that this works and continues to do so. OneRiot have followed suit but have decided to drop the search destination offering and instead have chosen to partner with companies and offer value through the OneRiot API.

Using an API as a source of monetization isn't a new idea with both Google and Amazon proving that this can be a fantastic revenue generator. Google provide access to a host of functionality through their APIs with a reasonable proportion of them focused on advertising, monetization and revenue generation and Amazon offer the ability to generate revenue via their Product Advertising API (UK version). However, what is interesting is that there appears to be a movement away from the absolute requirement that in order to generate revenue your product or service must be a destination - a website. By providing access to quality data, generated through ground breaking technology and exposed via an accessible API a service can be very successful.

Are real-time search engines dead? No. However the trends discussed above do indicate that consumers don't seem to need, or want, as many real-time search destinations (websites). But that in no way means that we don't need real-time search engines - we do still need technology products and services that can consume the vast amounts of real-time data being generated, extract value and expose this value to others. Therefore the engines will continue to be developed it's just that the focus and the value exposed by these products may well shift away from being consumer focused and instead the target user will be the programmable web.

Photo via Blake Patterson

Be sure to read the next Search article: At a Half Billion Queries Per Month, Topsy Launches Twitter Trending Analytics


Comments (9)

@Justin - Thanks for your comment and I agree.

>> One of the biggest problem with real time searches on Google is that the results update too frequently to even read on some topics and you can’t really see where the links are pointing to.

This is a user experience problem that hasn't been perfectly solved yet. TweetDeck, using the Twitter User Streams which update in real-time, have controlled how frequent a column can update even if new data has appeared. TwitterFall (!%231F3547) have decided to let the tweets "fall" into view but show a queue indicator so you are aware how many tweets there are waiting. There might be a better way of displaying or just using all this information.

>> People want to read news articles or find information and you really can’t get that from reading a 140 character tweet. Real Time search is ideal for breaking news stories or such, but won’t ever take notch over website article content.

Twitter clearly isn't going to be able to replace quality news articles but it can help users find the best articles through analyses of Tweets. You can actually see this on (e.g. that it's possible to use retweets to indicate which tweets potentially have value. If a tweet contains a link to a news article then it potentially means that the linked article has value.

Whilst real-time data delivery is possible we still need to get the user experience right rather than giving users information overload. We also need make sure that users get the most relevant information to their search through text analysis, trend tracking, authority analysis or a number of other clever methods. I'm very confident that the latter is exactly what real-time search engines are working on right now.


Real Time, I like the technology, it has a future I'm sure, but where? Real Time Search, a couple of questions...

1) Is it not a contradiction? Search as we know it, is about searching a big index of existing resources, if those resources are real time, we can't build an index to search, and if we do build the index, they're not real time?

2) Monetization, advertising, apparently the only model to support search. Real time results don't increase the revenue of advertisers, but real time adverts from advertisers may increase revenues, is that the missing focus?


[...] interesting article over on Programmable Web the other day about whether not it is finally the end for real time search that I suggest you read.  In the article it talks about two major real time search sites that shut [...]

[...] Meanwhile, there’s been some coverage of Collecta’s decision to terminate support for their real-time news/activity API, for instance ProgrammableWeb’s Is It Finally the End for Real-time Search Engines? [...]

One of the biggest problem with real time searches on Google is that the results update too frequently to even read on some topics and you can't really see where the links are pointing to. People want to read news articles or find information and you really can't get that from reading a 140 character tweet. Real Time search is ideal for breaking news stories or such, but won't ever take notch over website article content.

Hi Phil – My name is Meredith Klee and I work with Topsy, the realtime search leader. Great article, I found your discussion and thoughts on the state of realtime search very interesting. I do want to clarify that OneRiot is not a partner of Topsy. On October 21, OneRiot announced it would be focusing 100 percent on the advertising network for the realtime social web and would be transitioning its search customers to Topsy. As for the future of search, we agree with you, realtime search is not dead. In fact, Topsy is thriving – we’ll serve half a billion queries this month (mostly through our API), and this figure will continue to increase as more consumers, publishers and marketers capitalize on the value of realtime search results.

@Meredith Thanks for the comment and for the correction. I've updated the post and simply removed the partner bit. I was going to put a strike-through in there but that might give the wrong impression.

Really interesting, and I'm sure a massive bonus, to see that Topsy have Ron Conway as an advisor. Is there anything public on Topsy's business model or is it purely funding driven at the moment?

[...] January we asked. is it finally the end for real-time search engines? Today’s announcement would seem to answer with a resounding [...]