What Is Crawling In SEO<\/h2>\n
In the context of SEO, crawling is the process in which search engine bots (also known as web crawlers or spiders) systematically discover content on a website.<\/p>\n
This may be text, images, videos, or other file types that are accessible to bots. Regardless of the format, content is exclusively found through links.<\/p>\n
How Web Crawling Works<\/h2>\n
A web crawler works by discovering URLs and downloading the page content.<\/p>\n
During this process, they may pass the content over to the search engine index and will extract links to other web pages.<\/p>\n
These found links will fall into different categorizations:<\/p>\n
- \n
- New URLs<\/b> that are unknown to the search engine.<\/li>\n
- Known URLs that give no guidance on crawling<\/b> will be periodically revisited to determine whether any changes have been made to the page\u2019s content, and thus the search engine index needs updating.<\/li>\n
- Known URLs that have been updated<\/b> and give clear guidance. They should be recrawled and reindexed, such as via an XML sitemap last mod date time stamp.<\/li>\n
- Known URLs that have not been updated<\/b> and give clear guidance. They should not be recrawled or reindexed, such as a HTTP 304 Not Modified response header.<\/li>\n
- Inaccessible URLs<\/b> that can not or should not be followed, for example, those behind a log-in form or links blocked by a \u201cnofollow\u201d robots tag.<\/li>\n
- Disallowed URLs<\/b> that search engine bots will not crawl, for example, those blocked by the robots.txt file.<\/li>\n<\/ul>\n
All allowed URLs will be added to a list of pages to be visited in the future, known as the crawl queue<\/b>.<\/p>\n
However, they will be given different levels of priority.<\/p>\n
This is dependent not only upon the link categorization but a host of other factors that determine the relative importance of each page in the eyes of each search engine.<\/p>\n
Most popular search engines<\/a> have their own bots that use specific algorithms to determine what they crawl and when. This means not all crawl the same.<\/p>\n
Googlebot behaves differently from Bingbot, DuckDuckBot, Yandex Bot, or Yahoo Slurp.<\/p>\n
Why It\u2019s Important That Your Site Can Be Crawled<\/h2>\n
If a page on a site is not crawled, it will not be ranked in the search results, as it is highly unlikely to be indexed.<\/p>\n
But the reasons why crawling is critical go much deeper.<\/p>\n
Speedy crawling is essential for time-limited content.<\/p>\n
Often, if it is not crawled and given visibility quickly, it becomes irrelevant to users.<\/p>\n
For example, audiences will not be engaged by last week\u2019s breaking news, an event that has passed, or a product that is now sold out.<\/p>\n
But even if you don\u2019t work in an industry where time to market is critical, speedy crawling is always beneficial.<\/p>\n
When you refresh an article or release a significant on-page SEO change, the faster Googlebot crawls it, the faster you\u2019ll benefit from the optimization \u2013 or see your mistake and be able to revert.<\/p>\n
You can\u2019t fail fast if Googlebot is crawling slowly.<\/p>\n
Think of crawling as the cornerstone of SEO; your organic visibility is entirely dependent upon it being done well on your website.<\/p>\n
Measuring Crawling: Crawl Budget Vs. Crawl Efficacy<\/h2>\n
Contrary to popular opinion, Google does not aim to crawl and index<\/a> all content of all websites across the internet.<\/p>\n
Crawling of a page is not guaranteed. In fact, most sites have a substantial portion of pages that have never been crawled by Googlebot.<\/p>\n
If you see the exclusion \u201cDiscovered \u2013 currently not indexed\u201d in the Google Search Console page indexing report, this issue is impacting you.<\/p>\n
But if you do not see this exclusion, it doesn\u2019t necessarily mean you have no crawling issues.<\/p>\n
There is a common misconception about what metrics are meaningful when measuring crawling.<\/p>\n
Crawl budget fallacy<\/h3>\n
SEO pros often look to crawl budget<\/b><\/a>, which refers to the number of URLs that Googlebot can and wants to crawl within a specific time frame for a particular website.<\/p>\n
This concept pushes for maximization of crawling. This is further reinforced by Google Search Console\u2019s crawl status report showing the total number of crawl requests.<\/p>\n
![\"Crawl](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2023\/06\/website-crawling-the-what-why-how-to-optimize-via-sejournal-jes_scholz.png\")
Screenshot from Google Search Console, May 2023<\/span><\/div>\nBut the idea that more crawling is inherently better is completely misguided. The total number of crawls is nothing but a vanity metric.<\/p>\n
Enticing 10 times the number of crawls per day doesn\u2019t necessarily correlate against faster (re)indexing of content you care about. All it correlates with is putting more load on your servers, costing you more money.<\/p>\n
The focus should never be on increasing the total amount of crawling, but rather on quality crawling that results in SEO value.<\/p>\n
Crawl Efficacy Value<\/h3>\n
Quality crawling means reducing the time between publishing or making significant updates to an SEO-relevant page and the next visit by Googlebot. This delay is the crawl efficacy<\/strong>.<\/p>\n
To determine the crawl efficacy, the recommended approach is to extract the created or updated datetime value from the database and compare it to the timestamp of the next Googlebot crawl of the URL in the server log files.<\/p>\n
If this is not possible, you could consider calculating it using the lastmod date in the XML sitemaps and periodically query the relevant URLs with the Search Console URL Inspection API until it returns a last crawl status.<\/p>\n
By quantifying the time delay between publishing and crawling, you can measure the real impact of crawl optimizations with a metric that matters.<\/p>\n
As crawl efficacy decreases, the faster new or updated SEO-relevant content will be shown to your audience on Google surfaces.<\/p>\n
If your site\u2019s crawl efficacy score shows Googlebot is taking too long to visit content that matters, what can you do to optimize crawling?<\/p>\n
Search Engine Support For Crawling<\/h2>\n
There has been a lot of talk in the last few years about how search engines and their partners are focused on improving crawling.<\/p>\n
After all, it\u2019s in their best interests. More efficient crawling not only gives them access to better content to power their results, but it also helps the world\u2019s ecosystem by reducing greenhouse gases.<\/p>\n
Most of the talk has been around two APIs that are aimed at optimizing crawling.<\/p>\n
The idea is rather than search engine spiders deciding what to crawl, websites can push relevant URLs directly to the search engines via the API to trigger a crawl.<\/p>\n
In theory, this not only allows you to get your latest content indexed faster, but also offers an avenue to effectively remove old URLs, which is something that is currently not well-supported by search engines.<\/p>\n
Non-Google Support From IndexNow<\/h3>\n
The first API is IndexNow<\/strong>.<\/a> This is supported by Bing, Yandex, and Seznam, but importantly not Google. It is also integrated into many SEO tools, CRMs & CDNs, potentially reducing the development effort needed to leverage IndexNow.<\/p>\n
This may seem like a quick win for SEO, but be cautious.<\/p>\n
Does a significant portion of your target audience use the search engines supported by IndexNow? If not, triggering crawls from their bots may be of limited value.<\/p>\n
But more importantly, assess what integrating on IndexNow does to server weight vs. crawl efficacy score improvement for those search engines. It may be that the costs are not worth the benefit.<\/p>\n
Google Support From The Indexing API<\/h3>\n
The second one is the Google Indexing API<\/b>. Google has repeatedly stated that the API can only be used to crawl pages with either jobposting or broadcast event markup. And many have tested this and proved this statement to be false.<\/p>\n
By submitting non-compliant URLs to the Google Indexing API you will see a significant increase in crawling. But this is the perfect case for why \u201ccrawl budget optimization\u201d and basing decisions on the amount of crawling is misconceived.<\/p>\n
Because for non-compliant URLs, submission has no impact on indexing. And when you stop to think about it, this makes perfect sense.<\/p>\n
You\u2019re only submitting a URL. Google will crawl the page quickly to see if it has the specified structured data.<\/p>\n
If so, then it will expedite indexing. If not, it won\u2019t. Google will ignore it.<\/p>\n
So, calling the API for non-compliant pages does nothing except add unnecessary load on your server and wastes development resources for no gain.<\/p>\n
Google Support Within Google Search Console<\/h3>\n
The other way in which Google supports crawling is manual submission<\/b> in Google Search Console<\/a>.<\/p>\n
Most URLs that are submitted in this manner will be crawled and have their indexing status changed within an hour. But there is a quota limit of 10 URLs within 24 hours, so the obvious issue with this tactic is scale.<\/p>\n
However, this doesn\u2019t mean disregarding it.<\/p>\n
You can automate the submission of URLs<\/a> you see as a priority via scripting that mimics user actions to speed up crawling and indexing for those select few.<\/p>\n
Lastly, for anyone who hopes clicking the \u2018Validate fix\u2019 button on \u2018discovered currently not indexed\u2019 exclusions will trigger crawling, in my testing to date, this has done nothing to expedite crawling.<\/p>\n
So if search engines will not significantly help us, how can we help ourselves?<\/p>\n
How To Achieve Efficient Site Crawling<\/h2>\n
There are five tactics that can make a difference to crawl efficacy.<\/p>\n
1. Ensure A Fast, Healthy Server Response<\/h3>\n
![\"Host](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2023\/06\/website-crawling-the-what-why-how-to-optimize-via-sejournal-jes_scholz-1.png\")
Screenshot from Google Search Console, May 2023<\/span><\/div>\nA highly performant server is critical. It must be able to handle the amount of crawling Googlebot wants to do without any negative impact on server response time or erroring out.<\/p>\n
Check your site host status is green in Google Search Console, that 5xx errors are below 1%, and server response times trend below 300 milliseconds.<\/p>\n
2. Remove Valueless Content<\/h3>\n
When a significant portion of a website\u2019s content is low quality, outdated, or duplicated, it diverts crawlers from visiting new or recently updated content as well as contributes to index bloat<\/a>.<\/p>\n
The fastest way to start cleaning up is to check the Google Search Console pages report for the exclusion \u2018Crawled \u2013 currently not indexed.\u2019<\/p>\n
In the provided sample, look for folder patterns or other issue signals. For those you find, fix it by merging similar content with a 301 redirect or deleting content with a 404 as appropriate.<\/p>\n
3. Instruct Googlebot What Not To Crawl<\/h3>\n
While rel=canonical links and noindex tags are effective at keeping the Google index of your website clean, they cost you in crawling.<\/p>\n
While sometimes this is necessary, consider if such pages need to be crawled in the first place. If not, stop Google at the crawling stage with a robot.txt disallow.<\/p>\n
Find instances where blocking the crawler may be better than giving indexing instructions by looking in the Google Search Console coverage report for exclusions from canonicals or noindex tags.<\/p>\n
Also, review the sample of \u2018Indexed, not submitted in sitemap\u2019 and \u2018Discovered \u2013 currently not indexed\u2019 URLs in Google Search Console. Find and block non-SEO relevant routes such as:<\/p>\n
- \n
- Parameter pages<\/a>, such as ?sort=oldest.<\/li>\n
- Functional pages, such as \u201cshopping cart.\u201d<\/li>\n
- Infinite spaces, such as those created by calendar pages.<\/li>\n
- Unimportant images, scripts, or style files.<\/li>\n
- API URLs.<\/li>\n<\/ul>\n
You should also consider how your pagination strategy<\/a> is impacting crawling.<\/p>\n
4. Instruct Googlebot On What To Crawl And When<\/h3>\n
- Known URLs that give no guidance on crawling<\/b> will be periodically revisited to determine whether any changes have been made to the page\u2019s content, and thus the search engine index needs updating.<\/li>\n