Spider Guts

Share |

What's inside the spiders? To get a good ranking in search engines, a good understanding of the fundamentals of SEO and how search robots crawl web pages is essential. The author includes valuable information such as a list of core elements considered by a typical search engine when calculating page relevance.

In the quest for that elusive nirvana of search engine friendliness, we frequently find ourselves searching for "instant fix" ways to improve a page's ranking without considering the big picture; that is, without looking at the problem of optimizing a web page as a whole and instead looking at several separate optimization steps as part of routine markup development or copy writing. While SEO experts do not tend to fit this mold, the average web developer certainly does. How many web pages have you "optimized" by simply adding keyword and description meta tags, and stopped right there? I imagine a hand count at this point would supply a fairly substantial number. An even better question at this point might be, "How many of you have tried to provide SEO for a web page without having even a basic understanding of search robot logic or what it expects to see in your pages?" Once again, I suspect we would have a healthy hand count.

The steps to optimize a page are well known to the SEO community, and many articles by authors far more knowledgeable than myself on the subject are available to web developers. So with all this knowledge out there, why are there so many developers who lack a big picture understanding of the subject? One word: fundamentals. It is crucial to know how the technology behind the scenes works, but like any other skill, the bulk of people attempting to learn that skill do not start at the bottom. They start somewhere that makes sense in trying to solve a particular problem and then they build up from that point.

If a developer held a greater understanding of the fundamentals of SEO and how search robots went about crawling web pages, they would in turn have a greater understanding of how to populate those alt attributes and meta tags. The objective of this article is to provide a general overview of how search robots (also called spiders) go about crawling and indexing web pages.