A framework for evaluation was developed covering a wide range of criteria involved in the use and performance of four categories of search engines. Performance indicators of nine search engines using five health scenarios were selected, with Precision and Relative Recall used as relevance indicators. Popularity, the impact of sponsorship on the Search Engine Results Page (SERP), as well as differences in features of each search engine were addressed. The first ten results from each search in each search engine were evaluated in terms of relevance to informational need, usefulness of the information, usability of the website and quality of the website.
A health search is dependent on the technology used but also user motivation, skills and characteristics including interaction with the technology. Each search is a unique experience because the user and user needs vary for each search. After an initial search, the search terms may have to be modified.
Search scenarios were based on situations reflecting user need over a range of health searches. Unusual search terms entered into the health portals and verticals did not return relevant and useful results. General and meta search engines performed better using these scenarios. Overall, the general search engines offered the best starting point for search with Goggle the better of the two general and Jux2 the better of the two meta search engines. Uncluttered visual pages of the style of Google, Search.Yahoo were rated more highly than the cluttered landing pages of the health portals.
Traditional search engines are based on text relevance and link analysis. New Web 2.0 technologies are enabling many different methods of information retrieval over and above the key words into a search engine query field. Web 2.0 search involves creating, sharing and collaborating of information using text, video and sound. It has created a participating rather than passive environment. Vehicles such as custom-built search engines, blog sites, wikis, podcasts, social bookmarking, RSS site feeds and alerts allow users to pull selected information automatically into emails, customised page readers or aggregators. A custom search portal on any topic with can easily be set up and maintained by communities with shared interests. EureksterSwiki, Google, Yahoo and other sites offer this service at no cost. Mobile search engines are increasing in number in line with increasingly available Internet-enabled mobile devices.
Personalisation, which involves creating a user-profile, and customisation, which allows the user to select pre-defined sets, are increasinly offer as part of the search. The personalised search in general search engines is on a larger scale compared with the limited features offered by the verticals, Kosmix and Healia. Google search is now based on search history, recent searches and localisation. The localisation feature within Google is based on Internet Provider location even if users are not logged in. Features are themes, colour choices, home page, language options and page reader options for RSS. Personalisation features are continually being added. Signup and log in is a prerequisite for use of these features. The google.com search engine now offers details of the way the search has been personalised for each search as well as dropdown search suggest. It also includes a default definition of the search term on the SERP. Google.co.nz does not incorporate this feature to date. My Yahoo!Pipes  service from Yahoo! enables users to combine and filter web content and direct it to a feed reader. A “pipe” or feed can then be shared. Using this service requires a greater degree of skill than a simple key word entry into a query field of a search engine but the motivated health consumer has more choice with the new technologies increasingly available. Yahoo!360 is a personalised Yahoo! service offering personalised page, blog, mailbox as well as search. Yahoo also provides an “alpha beta” search engine ,which enables customisation of both information source and search profile. This search engine could come into the category of a Custom Search Engine, which, along with Google Coop , Rollyo  and others, allows users to create a search engine using only trusted sources.
With so many new services coming on-stream, it is possible for a motivated computer-user to make use of these services. It is questionable whether an average health searcher would be motivated enough to spend the time and make the effort, based on the current model of a health searcher. The use of these new technologies by the average health searcher is still in the future.
Semantic search has not been considered in this search engine comparison but word meaning and concept matching rather than key word matching are an evolving direction of search. Medline.cognition.com uses word-meaning technology to search Medline but this is a recent developement. Google, Yahoo! and Live search engines use blended (universal) search, which returns news, images, video and books as well as web pages.
The three dominant general search engines enable users to create their own personal searchable web and tags and to save content they find relevant. Users can share content, search content that other searchers have found useful, import contacts and save favourite pages by using a button on the toolbar of the browser.
Results pages can be structured according to preferences by extracting structured data from feeds and search across subscribed content can be part of the search. Signing into Google opens up an enhanced search experience. Personally selected pages can be saved pages into Google Notebook, with saved history, alerts sent to email, the ability to create custom search engines create individual pages with information feeds as more options. Allowing saved pages to remain private or shared by only contacts or everyone enables individual preferences and control over content. Microsoft, offers mylive.com, another subscription service within which users can create and share pages, use email services and experience personalised features. The three major general search engines have extended their user experiences, expecting from a commitment to a whole service, which becomes a personalised experience.
Folksonomies are collections of tags or labels that are also searchable. This study did not include search engines from this category but used Del.icio.us tags as one factor in assessing popularity. Searching for content using social bookmarks is another new way of searching for content. Social bookmarking sites allow users to save searches, label or tag them and identify other taggers with similar interests. Individuals can be added as contacts within a network, and setup options enable alerts when recent content is added. A tag search in Del.icio.us for “cancer” returned 66,675 results with the National Cancer Institute (US) having the highest number of tags (523). “Ulcerative colitis” returned 342 results with the Wikipedia article highest with 18 tags. As more people vote up, label or tag sites, there is potential for the best voted sites to be useful in health search. Privacy issues surrounding health are an issue in social search and searchers may be reluctant to allow others to view their tags or saved sites.
As search engines improve, some features that were previously considered challenges are being resolved. With the very good results from trustworthy sources now often appearing on the first page, a large volume of returned information is less of a concern. While this has an upside, new good quality sites have difficulty in achieving a high ranking and depend on social bookmarking to achieve recognition. Previous lack of organisation is also resolving with the general search engines all going some way to providing better organised results in health searches. Information provided for recognised medical conditions is categorised with options that include, treatment, test/diagnosis, symptoms and cause/risk factors identified at the top of the SERP in the general search engines.
One of the health scenarios covered in this study was seeking a self-help tool for depression. None of the nine search engine results returned satisfying results. Mindgym is one such tool. A search for the tag “mindgym” in Del.icio.us showed 50 results, which indicates it has been valuable to these voters. There is potential for a health user to find other sites saved by a user who has also used the tag “mindgym”. There is also the potential to add this tagger to create a personal network of people and keep an eye on new tags added. This method of searching for content or reliable health sites is a step away from the keyword entry into a general search engine, and requires motivation and new skill. At present there are more tags for design, software, programming websites so far but there is potiential for health tags to be added.
Question and answer systems - yet another method of information retrieval for consumers -allow users to type a question into the query field and receive user-voted answers. Users are invited to log in and vote answers up or down. Answers.com, wikianswers.com and answers.yahoo.com are examples. Answers.yahoo.com has a health section with options available for different countries.
Evaluation of the first ten sites for each scenario in each search engine included relevance, usefulness, usability of sites, and quality of health information evaluation (FA4CT). The range of queries included unusual search terms and common health terms, with the vertical and health portal search engines ranking lowest when using uncommon terms. Vertical search has excited some health sectors with the promise of targeting a section of the millions of pages on the web. However, when the large general search engines can return relevant pages in a short time it is difficult to see an advantage. This study has gone some way to establishing the advantages of the best general search engines over vertical health search.
Google emerged as possibly the best search engine across all comparisons and search.yahoo next. Both were general search engines that offered a wide range of additional features. The meta search engines, Jux2 and Dogpile, ranked relatively high with precision and relative recall, but did not have language or other commonly added features. All search engines except Revolution Health were fast but the efficiency of the general search engines was superior. Sponsored results were more intrusive in the health portals except for Dogpile. Dogpile interspaced the sponsored links with organic results in a manner that did not clearly differentiate the sponsored and organic links. Searchers could be unaware of links leading to sponsored sites.
Performance indicators within a group of search engines were measured to ascertain which search engines are more likely to lead health searchers more simply and directly to organised, relevant health information that complies with the major indicators of "quality" health information. Overall, the search engines within the general and meta search categories performed better than the vertical health search category and the search within a health portal category.
Search engine comparison can be made on many fronts. Characteristics of the technology, the user, their interaction and the information requirements are some of the broad categories that need consideration.
This study has focused on traditional evaluation measures plus popularity, usability and search engine features. It has considered that most users do not search past the first page of results. Unless sites are on the first page of results, they may not be available to the majority of searchers using a search engine, despite their quality, excellent or otherwise.
Google search engine is continually upgrading services, and despite newer search engines mounting challenges, the popularity of Google is increasing. Health consumers who simply want a standard key word matching search, they can do no better at this time than start at the Google search engine, but learning to use it better is an option. Seventy percent of searchers use it already, including possibly 56% of health professionals searching for health information . Health professionals may also lack the skills required to use PubMed  and find better results starting from a general search engine. Advanced search features including Google Scholar search, in general search engines allow searches that are more specific. Web 2.0, semantic search and rapid evolution of technologies around information retrieval have provided a multitude of other ways for consumers to acquire health information if they have the search skills required.
The Internet continues to morph. Current skill levels for consumer search are limited, pushing search engine companies to carefully consider usability, ease of use and consumer satisfaction.
1. (2008) Yahoo!Pipes. Accessed from: http://pipes.yahoo.com/pipes/ Access date: August 2008 Archived at: http://www.webcitation.org/5aPyLCIAB
2. YahooAlpha. Accessed from: http://au.alpha.yahoo.com/ Access date: August 2008 Archived at: http://www.webcitation.org/5aQ23cXh1
3. Google Coop Search Engine, Accessed from: http://www.google.com/coop/cse/ Access date: August 2008 Archived at: http://www.webcitation.org/5aQ3BISew
4. Rollyo Custom Search Engine. Accessed from: http://www.webcitation.org/5aQ3GgjeC Access date: August 2008 Archived at: http://www.webcitation.org/5aQ3GgjeC
5. Steinbrook, R. (2006) Searching for the right search - reaching the medical literature. N Engl J Med 354, 4 Accessed from: http://content.nejm.org.ezproxy.otago.ac.nz/cgi/reprint/354/1/4.pdf Access date: June 2008