How Search Engines Work
Once a web site is finished and 'live', it can be seen by anyone who visits its URL (web address). Of course, you will probably be relying on visitors finding your site via listings on search engines such as Google. But before this can happen, the search engines have to know that the brand new site actually exists—which is where search engine submission comes in.
Search Engine Submission
Search engine submission traditionally involves visiting sites such as Google and filling in a form that tells them about the new site—the web address, the name of the site, and sometimes a short description. Nowadays, many search engine experts argue that submission to free search engines is no longer necessary, as the technology is much better at discovering new sites unaided.
Once the site has been submitted or found, the search engine will add it to the queue of new sites which it will examine in turn — sometimes this takes 48 hours, sometimes several weeks depending on the number of new sites it has to visit. This process is completely outside of our—or anyone else's—control, and is run by the search engine's own software.
Increasingly, search engines and directories such as Yahoo offer the option to pay for faster inclusion in their index. It is important to realise that this does not always guarantee a good listing, just that they will examine your site sooner. Other search engines now demand a fee before they will accept a site submission.
There are also companies that offer to submit your site to multiple search engines in one go. We do not recommend this, as the major engines prefer people to submit by hand, and sometimes even penalise bulk submissions. These companies often promise to submit your site to thousands of search engines, but as the vast majority of visitors tend to use only the half-a-dozen or so major search engines and directories, this is largely pointless (the main engines often re-sell their services under different branding to Internet Service Providers). Additionally, some of the very small search engines are just fronts set up to gather email addresses for spam.
Search Engine Spiders
Once a search engine is aware of a new site, it visits and examines the site using a program called a spider (a.k.a. a crawler or robot). The spider visits a web page, analyses its content, and then follows links to other pages within the site. This is known as being "spidered" or "crawled." The spider returns to the site on a regular basis to look for changes.
Everything the spider finds is sent back to the second part of the search engine, the index. The index is like a catalogue containing a copy of every web page that the spider finds. If a web page changes, then this catalogue is updated with the new information.
Sometimes it can take a while for new pages or changes that the spider finds to be added to the index; a web page may have been "spidered" but not yet "indexed." Until it is indexed—added to the index—it is not available to people searching using the search engine.