From Aliweb to Yahoo!: A Brief History of Search Engines

Primitive search-engine technology.

I

n the beginning the Earth was formless and void, and darkness was upon the face of the deep. Al Gore said, “Let there be the internet” and there was the internet. But just as Isaac begat Jacob, and Jacob begat Judas and his brethren, the internet was fruitful and multiplied. And soon the people cried out and demanded a way to retrieve information from amidst the chat rooms and file transfer protocol (ftp) sites and Star Trek message boards.

Before there was an internet, if you wanted to find out something, you had to go to a library and thumb through index cards full of hand-typed, cross-referenced, Dewey-decimaled information that led you to a book on a shelf that you hoped had not been checked out. And these hundreds of thousands of cards only represented the contents of one library in one city in one tiny part of the world.

Although it has now become the planet’s de facto information library, the internet was not created as a repository of information. Nor was search engine technology created to retrieve information. The first tool for searching the web was called Archie. This program, created in 1990, simply downloaded directory listings from public ftp sites. 

Proof that man developed search engines before the advent of good graphic design.

The first true search engine was W3Catalog, which went online in 1993, followed closely by Aliweb. The first web-crawling robot, World Wide Web Wanderer, was created that same year to measure the size of the web. The first search engine to use a web-crawler to index the web was JumpStation. JumpStation was the first to combine crawling, indexing, and searching, which are essential components in all modern search engines.

Many search engines appeared in the mid-‘90s, including WebCrawler, Magellan, Infoseek, Northern Light, AltaVista, Excite, and Yahoo!, which was the first search engine to catch on with the public. Many more search engines came and went during the dot-com bubble of the late '90s, when Google’s search engine first began to gain traction.

It may seem difficult to imagine, but there was a time—not very long ago—when “google” was not yet a verb. The first known instance of someone referring to web searching as “googling” was in 1998, when Google co-founder Larry Page wrote “Have fun and keep googling!” on the Google Friends message board.                     

Now of course, Google is the gold standard of search engines, capturing 82-percent of the search engine market share. Surprisingly, many of the of the earliest search engines are still out there, although many are now powered by Yahoo! or Google. WebCrawler, Go, Lycos, Excite, Dogpile, and Teoma are all still online. Others, like Northern Light, have removed their search function and now offer other web-based services. And newcomers like Bing, Blekko, and the recipe-only search engine Yummly, struggle to win over a fraction of Google’s audience.

But on the internet, things can change overnight (remember MySpace?), so it’s anyone’s guess whether we’ll still be “googling” 10 years from now.