In previous articles we've made the distinction between search engines and search systems — in other words, full text search programs versus integrated collections of discovery tools. The distinction is important because it impacts implementation, staffing, customization, and information policy. It also affects how you answer the question, "How well is search working and how do we improve its performance?" The key metric for a search system is knowledge worker productivity, while the most common yardstick for a search engine is "findability."

In this article we look beyond search engine logs at other ways to evaluate search systems. We use examples from SharePoint, not because it has the best search engine, but because it's a good example of a search system that's easy for non-programmers to customize.

Search system components
A search system is a collection of automated tools and techniques that people use to find information. A search engine is only one of them. Others include:

• Collaboration systems designed to help people locate and interact with experts, both within their immediate area of interest and in peripheral areas;

• Authority files that standardize names of people, products, and topics to reduce duplication and produce more accurate searching and reporting;

• Semantic tools, such as thesauri and glossaries that help users differentiate among multiple meanings for the same term or multiple terms with the same meaning and show cross references to related topics;

• Classification tools that make it easy for both professional indexers and end users to assign categories to documents and Web sites;

• Content creation tools that make it easy for authors to create accurate and descriptive metadata when a new document is created;

• A – Z topical indexes that expose commonly used words and phrases, show cross references among them, and list highly relevant documents as determined by a subject matter expert or knowledge base editor;

• Personal knowledge management tools that allow users to export search results, “tag” documents with categories, and search the contents of their own computers;

• Workflow tools that make it easy for users to perform searches from within a specific desktop application.

Some of these components, such as search engines, A - Z indexes, authority files, and thesauri, lend themselves to quantitative analysis because their underlying data structures generate log files. Log data can be supplemented with online surveys linked to search results. Others, such as collaboration systems and workflow tools, are more suitable to qualitative approaches such as usability testing, team studies, and "collaborative profiling." Each of these methods is discussed below.

More ... (members only) How to become a member