This is a technical article about the different parts and pieces of a search engine web spider. If I lose you in the technical babble that is this description, I apologize ahead of time. I will try to keep the explanations simple by using analogies where necessary. If you are not familiar with what a search engine or web spider is, here a general definition. A spider is an automated computer program that prowls the Internet looking for publicly accessible resources and websites that can be added to it’s database. Now, lets dive into the dissection of our biological friend; the spider.

A spider sees and feels its way around migrating from one place to another in order to find the most optimal place for finding and catching the most food in its web. It can only eat so much food before it gets too full or too tired. It will take a break to digest the food and will then wait until later to do it all over again. If the food dries up, the spider will move on to another more food enriched environment. Search engine spiders move in a similar manner moving from link to link across the mesh net that is the web of the Internet. The spider will “eat up” as many website links as it can find, it will then process “the food” or in this case the content until there are no more links left to follow or until its belly gets full for the day. After it digests all the information and consumes as much resources as it can find, it will either come to the same site tomorrow for more or it may move on to another website and start the whole process all over. It may also occasionally come back to the previous sites where it found food to see if there is any new links or new content.

There are many reasons to build a spider but there are generally only a few types of spiders. You will have search engine spiders which are used to build databases of websites such as for Google.com, Yahoo.com and Bing.com. You will have Corporate Web crawlers which are used to crawl and index files, websites, and other relevant data that is privately owned and stored within the companies private networks. There are specialized crawlers which are more often used to collect specific types of data and/or gain a statistical dataset for reports. And then there are the dreaded email harvesters and other malicious data gathering agents. The harvesters attempt to collect information from websites to solicit to spammers such as email lists or even more specific information from virus programs like malware and other computer viruses.

Lets breakdown the parts of the spider bot. At a minimum, it will have a scanning system, a crawling system, an array of pattern recognition tools, an indexing system, and an analytic reporting system. Depending on how advanced you make the engine will determine the complexity of each of these systems but at a minimum we can describe a basic bot. The scanning systems only job is take an initial path or a list of user submitted paths such as the URL of a website, scan it, and then find more paths or more URLs that it can then scan later. The array of pattern recognition tools will consist of two types. The first type is the path pattern or patterns, the bot should find. In this case we are looking for URLs. The second type of pattern tools is a much larger set which is a set of rules to find content within each path or file it is scanning. Bots such as the Googlebot have very very very large array of this type of pattern recognition tools. The information that it finds is then stored for later analysis. The indexing system will then takeover by scanning through the patterns that were collecting and then generating large datasets that basically describe what it is that the bot discovered. This system will then index the data much in the same way you would sort information into a phone book. Once the indexing has been completed, the data is available for the analysis and/or reporting system. In out example the data would be ready to be display in the Google search engine. It will also be available for gathering reports about what was discovered by the bot by defining the type data you would like to see such as you would do when analyzing data in an Excel spreadsheet.

There is no specific way to develop a spider or a bot. They come in all kinds of different programming languages but almost all of them will follow the basic system layout that I have described here. You could build a system yourself, pay someone to build you a system, or pay someone to use their system, or simply just pay someone to give you reports from their system that meet your guidelines.