Categories: Online Marketing

Bing Makes its Search Crawler Bingbot More Efficient by @MattGSouthern

Bing’s search crawler, bingbot, has been improved to maximize crawl efficiency.

Bingbot, as one might assume, is similar to Googlebot. Its job is to discover new and updated content and add it to Bing’s index.

One of the concerns Bing has been hearing from webmasters regarding bingbot is that it doesn’t crawl frequently enough.

As a result of not crawling frequently enough, some site owners believe content in Bing’s index isn’t as fresh as it could be.

On the other hand, there are site owners who believe bingbot crawls too often, which causes constraints on website resources.

Bing says this is an engineering problem that hasn’t been fully resolved yet.

Making Bingbot More Efficient

The key issue is managing the frequency that bingbot needs to crawl a site to ensure new and updated content is included in the search index.

Bingbot also has to serve the requests of site owners. Some webmasters request to have their sites crawled daily, although the majority of webmasters would prefer to only their site crawled when new URLs have been added or content has been updated.

“The challenge we face, is how to model the bingbot algorithms based on both what a webmaster wants for their specific site, the frequency in which content is added or updated, and how to do this at scale.“

Measuring and Maximizing Crawl Efficiency

Bing measures the intelligence of bingbot using a metric called crawl efficiency, which is how often Bing crawls and discovers new and fresh content per page crawled.

Ideally, bingbot would crawl a URL only when the content has first been added to the web, or when a URL has been updated with fresh and useful content.

Crawl efficiency is lowered when bingbot crawls duplicated and/or unchanged content.

What This Means for Site Owners

Bing says it has improved the crawl efficiency of bingbot over the past few months.

For site owners, that means their new and updated content should appear in Bing’s index in a timely manner.

In addition, website resources should not be constrained due to Bingbot crawling duplicate content, or content that hasn’t been changed since the last crawl.

Subscribe to SEJ

Get our weekly newsletter from SEJ’s Founder Loren Baker about the latest news in the industry!

Share

Recent Posts

5 Best Ecommerce Tools Every Merchant Needs to Know!

The world is going online. With the new infrastructural developments taking place globally, more and more businesses are opting to… Read More

November 21, 2018 8:48 pm

Google Ads Brings Merchant Promotions to Shopping Actions by @MattGSouthern

Google Ads is bringing its Merchant Promotions program to Shopping Actions for retailers. This integration allows online retailers to add… Read More

November 21, 2018 4:48 pm

Popular AMP for WordPress Plugin Patches Critical Security Flaw by @MattGSouthern

The top AMP plugin for WordPress, AMP for WP, has released a released a patch for a critical security vulnerability.… Read More

November 21, 2018 12:39 pm

How SEO & PPC Keyword Research Can Work Together by @TaylorDanRW

A well-orchestrated PPC campaign can benefit a good SEO campaign, as Sergey Grybniak explores in How to Combine SEO &… Read More

November 21, 2018 8:38 am

3 Social Media Tools to Jumpstart Your Influencer Marketing

Social Media Marketing Industry Report In our 10th annual social media study (44 pages, 70+ charts) of 5700+ marketers, you'll… Read More

November 21, 2018 6:28 am

Digital Extortion: Don’t Let Your Data Be Held Hostage by @tonynwright

Recently, our firm took over the digital marketing efforts for a small company. Their previous digital marketing efforts were led… Read More

November 20, 2018 8:18 pm