{"id":15254,"date":"2022-08-29T09:29:29","date_gmt":"2022-08-29T09:29:29","guid":{"rendered":"https:\/\/www.searchenginejournal.com\/indexing-and-crawl-budget\/462742\/"},"modified":"2022-08-29T09:29:29","modified_gmt":"2022-08-29T09:29:29","slug":"google-shares-insights-into-indexing-crawl-budget-via-sejournal-martinibuster","status":"publish","type":"post","link":"https:\/\/marketingnewsbox.com\/?p=15254","title":{"rendered":"Google Shares Insights into Indexing & Crawl Budget via @sejournal, @martinibuster"},"content":{"rendered":"
<\/div>\n

Google recently published a podcast discussing what\u2019s known as a crawl budget and what influences Google to index content.<\/p>\n

Both Gary Illyes and Martin Splitt shared insights into indexing the web, as understood from Google\u2019s perspective.<\/p>\n

Origin of the Crawl Budget Concept<\/h2>\n

Gary Illyes said that the concept of a crawl budget was something created outside of Google by the search community.<\/p>\n

He explained there wasn\u2019t any one thing internally within Google that corresponded with the idea of a crawl budget.<\/p>\n

When people talked about a crawl budget, what was happening inside Google involved multiple metrics, not this one thing called a crawl budget.<\/p>\n

So inside Google they talked about what could represent a crawl budget and came up with a way of talking about it.<\/p>\n

He said:<\/strong><\/p>\n

\n

\u201c\u2026for the longest time we were saying that we don\u2019t have the concept of crawl budget. And it was true.<\/p>\n

We didn\u2019t have something that could mean crawl budget on its own- the same way we don\u2019t have a number for EAT, for example.<\/p>\n

And then, because people were talking about it, we tried to come up with something\u2026 at least, somehow defined.<\/p>\n

And then we worked with two or three or four teams\u2013 I don\u2019t remember\u2013 where we tried to come up with at least a few internal metrics that could map together into something that people externally define as crawl budget.\u201d<\/p>\n<\/blockquote>\n

What Crawl Budget Means Within Google<\/h2>\n

According to Gary, part of the calculation for a crawl budget is based on practical considerations like how many URLs does the server allow Googlebot to crawl without overloading the server.<\/p>\n

Gary Illyes and Martin Splitt:<\/strong><\/p>\n

\n

\u201cGary Illyes: \u2026we defined it as the number of URLs Googlebot can and is willing or is instructed to crawl.\u201d<\/p>\n

Martin Splitt: For a given site.<\/p>\n

Gary Illyes: For a given site, yes.<\/p>\n

And for us, that\u2019s roughly what crawl budget means because if you think about it, we don\u2019t want to harm websites because Googlebot has enough Chrome capacity to bring down sites\u2026\u201d<\/p>\n<\/blockquote>\n

Balancing Different Considerations<\/h2>\n

Another interesting point that was made was how, in relation to crawling, there are different considerations involved. There are limits to what can be stored so, according to Google, that means utilizing Google\u2019s resources \u201cwhere it matters.<\/em>\u201d<\/p>\n

\n

\u201cMartin Splitt: Apparently, obviously, everyone wants everything to be indexed as quickly as possible, be it the new website that just came online or be it websites that have plenty of pages, and they want to frequently change those, and they\u2019re worried about things not being crawled as quickly.<\/p>\n

I usually describe it as a challenge with the balance between not overwhelming the website and also spending our resources where it matters.\u201d<\/p>\n<\/blockquote>\n

John Mueller recently tweeted that Google doesn\u2019t index everything and mentioned that not everything is useful.<\/p>\n

Mueller\u2019s tweet<\/a>:<\/strong><\/p>\n

\u201c\u2026it\u2019s important to keep in mind that Google just doesn\u2019t index every page on the web, even if it\u2019s submitted directly. If there\u2019s no error, it might get selected for indexing over time \u2014 or Google might just focus on some other pages on your site.\u201d<\/p>\n

He followed up with another tweet<\/a>:<\/strong><\/p>\n

\n

\u201cWell, lots of SEOs & sites (perhaps not you\/yours!) produce terrible content that\u2019s not worth indexing. Just because it exists doesn\u2019t mean it\u2019s useful to users.\u201d<\/p>\n<\/blockquote>\n