Categories: Online Marketing

How Often Google Crawls and Indexes by @martinibuster

In a webmaster hangout, a publisher asked how fast Google removed pages from the index if they added a noindex nofollow to it. The publisher stated they had added noindex but the page remained in Google’s index. Google’s John Mueller responded with an answer that described how often some pages are indexed.

John Mueller revealed that URLs are crawled at different rates. That’s somewhat well understood. What was of more interest was that he said some URLs can be crawled as little as once every six months.

The publisher stated:

“We’re seeing stuff that’s from a long time ago, where we’ve changed the noindex nofollow but we’re still seeing it in the index. And this is several months after we’ve changed this.”

John Mueller answered:

“I think the hard part here is that we don’t crawl URLs with the same frequency all the time. So some URLs we will crawl daily. Some URLs maybe weekly. Other URLs every couple of months, maybe even every once half year or so.

So this is something that we try to find the right balance for, so that we don’t overload your server.

And if you made significant changes on your website across the board then probably a lot of those changes are picked up fairly quickly but there will be some leftover ones.

So in particular if you do things like site queries then there’s a chance that you’ll see those URLs that get crawled like once every half year. They’ll still be there after a couple of months.

And that’s kind of… the normal time for us to kind of reprocess/re-crawl things. So it’s not necessarily a sign that something is technically completely broken.

But it does mean that if you think that these URLs should really not be indexed at all, then maybe you can kind of back that up and say well here’s a sitemap file with the last modification date so that Google goes off and tries to double-check these a little bit faster than otherwise.”

Use The Site Map to Trigger Updated Crawling

John Mueller suggested updating the site map and letting Googlebot discover the last modified date and using that as a hint for it to go out and crawl the old web pages.

Google URL Inspection Tool

Something John Mueller didn’t mention is using Google’s URL Inspection tool. According to Google’s Webmaster Help page on re-indexing,  a submission can take up to a week or two.

The URL Inspection tool is useful if you have a few individual URLs that need re-crawling. If you have a large amount of web pages, Google recommends submitting a site map instead.

More information on how to ask Google to re-crawl URLs here
https://support.google.com/webmasters/answer/6065812

Watch the Webmaster Hangout here
https://youtu.be/gC7aVygbMIk?t=2594

Images by Shutterstock, Modified by Author

Subscribe to SEJ

Get our weekly newsletter from SEJ’s Founder Loren Baker about the latest news in the industry!

Sagie

Share
Published by
Sagie

Recent Posts

5 of the Best Ecommerce Tools Every Merchant Needs by @shane_barker

Ecommerce sales in 2018 have, so far, exceeded $500 billion in the U.S. alone. With the increasing use of smartphones… Read More

November 21, 2018 8:48 pm

Google Ads Brings Merchant Promotions to Shopping Actions by @MattGSouthern

Google Ads is bringing its Merchant Promotions program to Shopping Actions for retailers. This integration allows online retailers to add… Read More

November 21, 2018 4:48 pm

Popular AMP for WordPress Plugin Patches Critical Security Flaw by @MattGSouthern

The top AMP plugin for WordPress, AMP for WP, has released a released a patch for a critical security vulnerability.… Read More

November 21, 2018 12:39 pm

How SEO & PPC Keyword Research Can Work Together by @TaylorDanRW

A well-orchestrated PPC campaign can benefit a good SEO campaign, as Sergey Grybniak explores in How to Combine SEO &… Read More

November 21, 2018 8:38 am

3 Social Media Tools to Jumpstart Your Influencer Marketing

Social Media Marketing Industry Report In our 10th annual social media study (44 pages, 70+ charts) of 5700+ marketers, you'll… Read More

November 21, 2018 6:28 am

Digital Extortion: Don’t Let Your Data Be Held Hostage by @tonynwright

Recently, our firm took over the digital marketing efforts for a small company. Their previous digital marketing efforts were led… Read More

November 20, 2018 8:18 pm