The SEOniverse

What is Bingbot?

Bingbot is one of Microsoft's five web-crawling robots that collect documents from the web to build the Bing search engine index. It performs the same function as Google's Googlebot, following index requests from Bing Webmaster Tools and links from web pages to build an accurate picture of the web.

What are Microsoft's other crawlers?

Bingbot is Microsoft's standard crawler and it manages the vast majority of crawling every day with a number of desktop and mobile variants. The other four are:

MSNBot

MSNBot was the standard crawler before Bingbot was unleashed in 2010 and it still admirably handles a fair portion of crawls.

MSNBot-Media

MSNBot media handles image and video crawls.

AdldxBot

The snappily named AdldxBot is the main crawler for Bing Ads. It follows all ads to verify their authenticity. Like Bingbot, AdldxBot has desktop and mobile variants.

BingPreview

BingPreview is the crawler used to generate previews of webpages.

Verifying Bingbot

All search crawlers are identified by a user agent string, for an example of what these should look like check out Bing's examples of user agent strings for each of their bots on their Webmaster Help page. However, these strings are easy to spoof so they may not be identifiable by matching the strings manually. Bing (understandably) does not share the IP addresses from which it crawls the web, but if you're in doubt you can use  the verification tool to see if you're being crawled by a Bingbot or someone masquerading as a Bingbot.

Controlling the crawlers

There are various methods for controlling how Microsoft's Bingbot crawls and indexes your website:

robots.txt

Like Googlebot, you can tell  the Bingbot how you would like it to crawl your website with a robots.txt file.

rel="nofollow"

"No follow" attributes (rel="nofollow" or content="nofollow") can also be added to any links that you don't want the bots to pass link juice through. Be aware that the pages linked to will still be crawled and possibly indexed too.

Crawl control

You can also use the crawl control feature within Bing Webmaster Tools to select a crawl pattern for your website. Setting the Bingbot to crawl when you have less visitors on your site will save bandwidth and allow the bots to perform a more in-depth, faster crawl at quieter times of the day – in theory anyway. If you're struggling to get Bing to index your site, make sure you submit an XML sitemap in Webmaster Tools and think about reviewing your internal linking strategy.

Back to the SEO glossary →