INTRODUCTION

Travel is a multi-billion dollar industry that has adapted to the needs of the public over its many years of existence. The increasing mobility of the average family, coupled with the globalization of business, has allowed the industry to continuously grow. With the advent of new technologies, such as computerized reservations systems and the internet, travelers are now able to access an unprecedented amount of information about the industry and the businesses operating within it. A major starting point and vehicle by which this information can be accessed is the internet search engine. Unfortunately, this important vehicle is woefully inadequate in providing information about what consumers actually prefer in the industry. Studies of the search engines and how they perform in relation to travel search criteria confirm this fact.

BACKGROUND

Testimonials to the limitations and ineffectiveness of internet search engines abound. They are supported by studies that have been conducted on a regular basis over several years. The Wall Street Journal, Reuters, ABC News, Time, PC World, and the Chronicle of Higher Education, among others, have all publicized these findings.

One such study, which is conducted on a regular basis, asks users of a number of search engines to rate their success rate in using the engines. The last results reported in Fall 1999 indicated that only 58% of the respondents found what they were looking for "most of the time," 19.7% found what they were looking for "some of the time," and 18.7% found what they were looking for "every time." Of particular interest are the statistics reflecting search failure rates over the past two years. The number of people reporting that they found what they were looking for only "some of the time" or "never" has steadily increased. The "never" category alone has increased from an almost insignificant amount to nearly 5% of the respondents. (1)

Another study illustrated an all too common problem with search engines: the low number of relevant and useable links returned from the search. In 1999, CNET.com tested several search engines and reported at best only half of the links returned were useable or relevant to the search. Most of the engines rated much poorer. For example, Lycos returned only 2.5 useable links out of ten sites returned. An additional fact noted in the study was the frequent listing of out-of-date and duplicate pages. (2)

The conclusions of these studies support the statement that, in searching for information on general topics, search engines are incapable of returning concise and directly relevant results due to the increasing vastness of the world wide web, inefficient methodology in searching the web, and constantly changing information. "The current state of search engines can be compared to a phone book which is updated irregularly, is biased toward listing more popular information, and has most of the pages ripped out." (3) This state of affairs leads to a significant amount of irrelevant and incomplete data being returned on even the most simple of searches conducted.

GENESIS
While preparing for the launch of Top9.com, a new venture by Omega World Travel, researchers discovered that the results provided by major search engines to simple searches on travel topics did not seem to correlate to the most popular websites visited by consumers. Moreover, it was discovered that although several studies had been conducted on the effectiveness of search engines to general inquiries, no studies had been conducted thus far on the effectiveness of the search engines to inquiries on travel topics.

In an effort to evaluate the performance of the internet search engines with respect to how they serve the travel industry in disseminating information, and how they serve the public in travel-related searches, researchers at Omega World Travel conducted an analysis of the major search engines and their performance on travel topics over several days in April 2000. The results of the study confirm that the Internet's major search engines, in addition to doing a poor job on general searches, do a very poor job of providing users the best information on travel-related topics.
 

METHODOLOGY AND RESULTS

The study tested 11 popular search engines on their performance in providing information about common travel topics between April 5 and 11, 2000, using statistics provided by PC Data, a consumer research firm. PC Data gathers website traffic data from a pool of 120,000 users and processes it using objective, scientific methodology. Thus, the sites used for comparison purposes here represent those that are indeed of the greatest interest to the general public, based on their popularity.

Projected popularity rankings are based on unique monthly visitors. Site visitors are counted only once in the course of a month, no matter how many times they visit a site. Data used are from the month of February, 2000, and are most valid among the top 2,500 rankings.

The 11 search engines selected for the study were Yahoo, MSN, Lycos, Netscape, Excite, AltaVista, Iwon, Snap, Direct Hit, Google, and Ask Jeeves. The topics selected for the study were Airlines, Airline Tickets, Cruise Lines, Hotels, and Car Rentals. No Boolean connectors or additional search characters were used in the search string.

The study evaluated the overall performance of the search engines by awarding them points for sites listed. Ten points were awarded if one of the top nine most popular sites reported by PC Data appeared in the search engine's top 10 website listings. Five points were awarded if the site appeared in the next 10 listings, or if a separate link was offered to the site. Listings numbered 20 and above were not counted. The highest possible base score in each category was 90.

Overall, even the highest ranking search engines fell well short of the maximum possible score, reflecting only about half of what consumers prefer. The lowest-scoring engine provided search results that reflected only 12% of consumer preferences.

MSN and Snap tied for first, with a score of 49. Netscape and Excite ranked lowest, with scores of 16 and 11, respectively. Overall scores are below. Explanatory notes follow the list.

Notes:

1. Five categories were searched: Airlines, Airline Tickets, Cruise Lines, Hotels, and Car Rental.
2. Average points above were derived by summing the search engine scores and dividing by 5.
3. Highest possible average score = 90, plus possible "extra credit" in some categories.

The results of the study confirm that the major search engines fail to return a significant amount of relevant information when presented with simple search topics, in this case in the travel categories tested. Perhaps more importantly, the study also reveals that in many cases the search engines failed to return the most popular websites, as determined by the number of unique visitors to those sites, in each travel category.

The search engines offered reasonable performance at the "micro" level, or in searching out obscure and narrowly focused sites. However, they did a poor job on the "macro" level, or in returning generally relevant sites, when searching for selected travel industry-related keywords.

For example, when "cruise lines" was entered into the Lycos search engine, only 3 of the top 9 cruise line sites visited appeared in the top 20 websites listed by the engine. Of these, one was merely a non-search-related link to a cruise line site. The actual website didn’t appear as a search result. Excite.com failed to return even one of the top 9 sites visited by users. Most of the sites returned by the search engines were cruise booking agents, small regional cruise lines, or small agencies.

Similar results were observed in the "airlines" category, where Netscape returned only one of the top 9 airline sites visited, and Yahoo, considered one of the most successful search engines, returned only two. Interestingly, when the more specific string "airline tickets" was used, results were poor for all search engines, with only MSN scoring in the respectable range.

The hotel category, which included hotels and hotel booking sites, returned the lowest average scores of all the travel categories. Ask Jeeves, Direct Hit, and Excite did the worst, failing to return more than one of the top nine sites visited in the category. "Car Rentals" produced the highest score for any of the categories with Snap's score of 80. However, other major engines, namely Lycos, Excite, and Netscape, scored poorly.

Several tables have been compiled to demonstrate the report findings, and are available online.

It is evident that the searches produced inconsistent and irrelevant information. Search engines also provided inconsistent results compared to each other. In addition, the same URLs were often listed multiple times within the top 10 or 20 sites returned from a search. The following links provide examples of the kind of URLs returned in some categories.

CONCLUSIONS

It has already been stated that the major search engines do a poor job of returning relevant results to simple travel searches. Data resulting from this and other studies confirm this fact. What is more important, and has not been reported previously, is that the search results also lack much of a correlation to the most popular websites visited, and what consumers actually prefer.

What factors contribute to such poor results? Prior studies have stated that the internet is growing so quickly that search engines cannot keep pace. Links are constantly changing, and the number of pages has ballooned. In the span of two years, from 1997 to 1999, the World Wide Web doubled in size, from 320 million pages to 800 million. "Over the same period, the top ranking search engine's coverage of those pages dropped from approximately 34% to 16%."(4)

With the increase in the size of the web comes an increase in the time it takes to effectively search it. The obvious conclusion is that search engines cannot possibly search the entire web every time they are used. This would seem to point to the conclusion that only the most popular websites, based on how frequently they are used, would be returned in most searches. However, as this study indicates, this is not the case.

Obviously, as the internet grows and more businesses use the web to provide information about their services, the methods and resources used by search engines must be improved. Unfortunately, many search engines may not be willing to make substantial investments in time and money to do so.

Complicating matters is the pressure for many engines to offer more than just a web searching service. For instance, Yahoo, one of the most popular "search" sites, has grown into an online site that offers many of the same features as the major internet service providers. Although this may prove convenient for some users, such additional services can drain resources and arguably alter priorities from providing an efficient searching agent to obtaining maximum advertising revenue.

There has also been some speculation that some search engines may be selling preferred placement in their indexes. This has been vigorously denied by the major search services, but it is a practice that has been cited as widely taking place on more specific sites, such as BigYellow.com and CitySurf.com, sites which list businesses and services locally and nationally.(5)

Like other businesses, travel companies seek to disseminate information about their services in the best possible manner. Websites of businesses in the travel industry have become one of the most significant sectors on the internet in recent years. Unfortunately, access to much of their information is stymied by search engines that produce an abundant amount of irrelevant material.

Web search engines need to reconsider the methods they use to search for data, for the benefit of their users, and the industries that rely on them.

The development of new searching vehicles, such as TOP9.com, which ranks the web's top sites into nearly 300 categories based on user popularity, will go a long way to assisting the industry and the public in internet searching. This is true in terms of quality of search results and ease of use. On travel topics alone, TOP9.com outperformed all of the search engines studied in seeking the most relevant information.

The travel industry should investigate all available methods of promoting their websites to ensure that their online presence is efficient and accessible. Search engine performance will only decline as the web expands, and should not be depended upon as a reliable source of client referrals.

NOTES:

(1) Danny Sullivan, "NDP Search and Portal Site Study," SearchEngineWatch.com, April 4, 2000 (http://www.searchenginewatch.com/reports/npd.html). BACK

(2) Gregg Keizer, "Search Engine Shoot-Out: CNET Compares the Top 5 Engines," CNET.com, April 7, 1999 (http://home.cnet.com/category/topic/0,10000,0-3817-7-276910,00.html). BACK

(3) Steve Lawrence and Lee Giles, "Accessibility and Distribution of Information on the Web," Nature, Vol. 400, pp. 107-109, 1999. BACK

(4) Maryann Jones Thompson, "Search Engines Can’t Keep Up With the Web," The Standard.net, July 7, 1999, (http://www.thestandard.net/article/display/0,1151,5423,00.html). BACK

(5) Brad Grimes, "Can You Trust Your Search Engine?", PC World, August, 1997. BACK

Go To: Home | Press Release | Index of Tables |
Omega World Travel | PCData | Top 9.com