Google dives into crawl budgets and answers some of the most common SEO-related questions in the latest episode of SEO Mythology.
Join Merkle’s Alexis Sanders and Google’s Martin Splitt to answer 12+ questions SEOs often ask about crawl budgets.
Here’s a quick summary of each question and answer, along with the corresponding time stamps in the video.
What is a crawl budget? (1:15)
A crawl budget is a balance between crawling as much content on your site as fast as possible without overwhelming your server.
Your budget is the number of simultaneous requests that Googlebot can make without overloading your server.
Related: How Search Engines Crawl and Index: What You Need to Know
What is crawl demand? (1:47)
This is how often Google desire Crawl the site based on subject matter.
For example, breaking news sites may have higher crawl demand than recipe sites.
How does Googlebot determine crawl rate and crawl requests? (2:44)
Google determines how often to crawl a page based on how often the content changes. If the frequency of changes is low, your site will not be crawled as often as other sites.
ETags, HTTP Headers, Last Modified Date, and More (3:43)
Google uses ETags, HTTP headers, last modified dates, etc. to determine how often to crawl your content.
ETag is a cache header that contains a fingerprint of content for detecting changes over time.
What size site should I care about crawl budget? (4:35)
This is primarily a concern for large sites with millions of pages. If your site has less than 1 million pages on his, don’t worry about your crawl budget.
Server setup and crawl budget (5:00)
Crawl budget is frequently cited as a site owner issue when the underlying issue is usually server setup or content quality.
Crawl Frequency and Content Quality (6:18)
It does not indicate that content that is crawled more frequently is of higher quality. Also, low crawl frequency does not mean low quality content.
Related: 7 tips to optimize your SEO crawl budget
If Google is testing your server, what would you expect to see in your log files? (7:45)
Crawl activity may increase, then decrease, and then increase again. In other words, a wave pattern.
Tips on how to accurately crawl your site during site migration (8:18)
Martin Splitt recommends incrementally updating your sitemap and keeping track of what changed and when. Additionally, make sure both servers run as smoothly as possible.
Different levels of crawl budget and site infrastructure (9:40)
The impact of crawl budget on various levels of site infrastructure depends on the site itself.
This is usually nothing for site owners to worry about, Splitt said.
Does Crawl Budget Affect Rendering? (10:37)
Yes, crawl budget affects rendering. Googlebot gets additional resources from your site’s crawl budget when rendering content.
Resource and Crawl Budget Caching (11:46)
Google caches resources as aggressively as possible so that resources don’t have to be recrawled every time.
Specific industries such as crawl budgets and publishing (13:34)
Ecommerce sites and large publishers should be most concerned about crawl budget.
What is generally recommended to assist Googlebot in crawling a site? (15:03)
Splitt recommends blocking crawling of resources that absolutely don’t need to be crawled. This allows Googlebot to crawl more efficiently.
What are the common pitfalls people fall into when it comes to crawl budgets? (16:52)
A common problem site owners encounter is blocking resources in their robots.txt that Googlebot actually needs.
For example, some sites block crawling of CSS files. This file is required by Googlebot to render the content as the visitor sees it in her web browser.
Can I tell Googlebot to crawl my site more? (17:40)
No you can’t do this.Site owners can limit Find out how often Googlebot crawls, but there’s no way to make Googlebot crawl more often.
See the full video below.
