The SEOniverse

What is a bot?

A bot – also known as an internet bot, web robot or web crawler – is a software application which automatically crawls the internet running scripts. Bots perform tasks that are structured and repetitive at a far higher rate than would be possible for a human.

Bots for SEO

The bots that are dispatched by the major search engines crawl the internet feeding back information about  web pages for the search engines to hold in their index. The most common bots for SEOs to be concerned with are Google bot and Bingbot.

Bots can validate hyperlinks and HTML code so they can send signals about poor quality pages and domains to the search engines. If your pages are found to be poor quality, the bots are less likely to come back and visit your domain making it less likely that all the relevant pages on your site will be indexed by the search engines and findable by users.

Crawl budget

The number of pages on the internet is incredibly large and, even for a bot, it's impossible for every page to be crawled and indexed. If your domain is seen as being low quality, or you do not offer enough positive signals to the bot about your pages, you may receive a small crawl budget, meaning only a selection of your pages will be crawled and indexed. Having good quality pages and an intuitive site structure can increase your crawl budget.

Keeping track of the bots

You can keep an eye on how often your site is crawled in Google Webmaster Tools' crawl rate report

If you're worried about Google not indexing all your pages, you can check this in the index status report. If the number of indexed pages drops, it's a clear sign that something is wrong with the SEO of your domain and you should address it as quickly as possible.

In the index status report above, you can see that a new website typically takes a while to be crawled by bots and indexed.

Bing also has crawl stats and index status reports on their own Webmaster Tools service. To get the clearest picture of how your webpages are being handled by the bots use both Bing and Google Webmaster Tools together.

Still concerned about how the search engine bots are crawling your site? Check out this article from Search Engine Journal on how to increase your crawl rate.

Back to the SEO glossary →