We were recently asked an interesting question:

The short answer is no. If you're trying to decide which search product to buy, there are no benchmarks or reference data that allow you to compare them. One reason is that the vendors have no incentive to participate in such an exercise. Another is that apples-to-apples comparisons are difficult because each organization has its own unique requirements — even if they are in the same industry. Finally, search engine performance depends to some extent on how it is customized and how the content is structured. In other words, you may be better off testing and tweaking your current search engine instead of shopping for a new one.

That said, we did find some valuable tools and techniques that can be used to evaluate search engine options:

• a government-sponsored benchmarking test bed for new retrieval technologies;
• information technology analyst reports;
• a metric called MRR that can be applied to search engine results;
• an analytics service for Google searches, both public and intranet;
• a list of best practices for search systems.

Each of these is discussed below.

More ... (members only) How to become a member

Created on June 7, 2007 l Updated on July 24, 2007