Indexing is the process of downloading information from your website, categorizing it, and storing it in a database. This database \u2013 the Google index \u2013 is the source of all information you can find via Google Search.<\/p>\n
Pages that aren\u2019t included in the index cannot appear in search results, no matter how well they match a given query.<\/p>\n
Let\u2019s assume you\u2019ve recently added a new page to your blog. In your new post, you discuss a trending topic, hoping it will provide you with a lot of new traffic.<\/p>\n
But before you can see how the page is doing on Google Search, you have to wait for it to be indexed.<\/p>\n
So, how long exactly does this process take? And when should you start worrying that the lack of indexing may signal technical problems on your site?<\/p>\n
Let\u2019s investigate!<\/p>\n
How Long Does Indexing Take? Experts\u2019 Best Guesses<\/h2>\n
The Google index contains hundreds of billions of web pages<\/a> and takes up over 100 million gigabytes of memory.<\/p>\n Additionally, Google doesn\u2019t limit how many pages on a website can be indexed<\/a>. While some pages may have priority in the indexing queue, pages generally don\u2019t have to compete for indexing.<\/p>\n There should still be room for one more small page in this colossal database, right? There\u2019s no need to worry about your blog entry? Unfortunately, you might have to.<\/p>\n Google admits that not every page processed by its crawlers will be indexed<\/a>.<\/p>\n In January 2021, Google Search Advocate, John Mueller, elaborated on the topic, disclosing that it\u2019s pretty normal that Google does not index all the pages of a large website<\/a>.<\/p>\n He explained that the challenge for Google is trying to balance wanting to index as much content as possible with estimating if it will be useful for search engine users.<\/p>\n Therefore, in many cases, not indexing a given piece of content is Google\u2019s strategic choice.<\/p>\n Google doesn\u2019t want its index to include pages of low quality, duplicate content<\/a>, or pages unlikely to be looked for by users. The best way to keep spam out of search results is not to index it.<\/p>\n But as long as you keep your blog posts valuable and useful, they\u2019re still getting indexed, right?<\/p>\n The answer is complicated.<\/p>\n Tomek Rudzki, an indexing expert at Onely \u2013 a company I work for \u2013 calculated that, on average, 16%<\/a> of valuable and indexable pages on popular websites never get indexed.<\/p>\n As you may have already guessed from the title of this article, there is no definitive answer to this indexing question.<\/p>\n You won\u2019t be able to set yourself a calendar reminder on the day your blog post is due to be indexed.<\/p>\n But many people have asked the same question before, urging Googlers and experienced SEO pros to provide some hints.<\/p>\n John Mueller says it can take anywhere from several hours to several weeks<\/a> for a page to be indexed. He suspects that most good content is picked up and indexed within about a week.<\/p>\n Research conducted by Rudzki showed that, on average, 83%<\/a> of pages are indexed within the first week of publication.<\/p>\n Some pages have to wait up to eight weeks to get indexed. Of course, this only applies to pages that do get indexed eventually.<\/p>\n For a new page on your blog to be discovered and indexed, Googlebot has to recrawl the blog.<\/p>\n How often Googlebot recrawls your website certainly impacts how quickly your new page will get indexed, and that depends on the nature of the content and the frequency with which it gets updated.<\/p>\n News websites that publish new content extremely often need to be recrawled frequently. We can say they\u2019re sites with high crawl demand.<\/p>\n An example of a low crawl demand site would be a site about the history of blacksmithing, as its content is unlikely to be updated very frequently.<\/p>\n Google automatically determines whether the site has a low or high crawl demand<\/a>. During initial crawling, it checks what the website is about and when it was last updated.<\/p>\n The decision to crawl the site more or less often has nothing to do with the quality of the content \u2013 the decisive factor is the estimated frequency of updates.<\/p>\n The second important factor is the crawl rate<\/a>. It\u2019s the number of requests Googlebot can make without overwhelming your server.<\/p>\n If you host your blog on a low-bandwidth server and Googlebot notices that the server is slowing down, it\u2019ll adjust and reduce the crawl rate.<\/p>\n On the other hand, if the site responds quickly, the limit goes up, and Googlebot can crawl more URLs.<\/p>\n Since indexing takes time, one can also wonder \u2013 how exactly is that time spent?<\/p>\n How is the information from your website categorized and included in the Google index?<\/p>\n Let\u2019s discuss the events that must happen before the indexing.<\/p>\n Let\u2019s go back to the example in which you posted a new blog entry. Googlebot needs to discover this page\u2019s URL in the first step of the indexing pipeline.<\/p>\n It can happen by:<\/p>\n The fact that the page has been discovered means that Google knows about its existence and URL.<\/p>\n Crawling is the process of visiting the URL and fetching the page\u2019s contents.<\/p>\n While crawling, Googlebot collects information about a given page\u2019s main topic, what files this page contains, what keywords appear on it, etc.<\/p>\n After finding links on a page, the crawler follows them to the next page, and the cycle continues.<\/p>\n It\u2019s important to remember that Googlebot follows the rules set up by robots.txt so that it won\u2019t crawl pages blocked by the directives you provide in that file.<\/p>\n The rendering needs to happen for Googlebot to understand both the JavaScript content and images, audio, and video files.<\/p>\n These types of files always were a bigger struggle for Google than HTML.<\/p>\n Google\u2019s developer advocate, Martin Splitt, compared rendering to cooking a dish<\/a>.<\/p>\n In this metaphor, the initial HTML file of a website with links to other contents is a recipe. You can press F12 on your keyboard to view it in your browser.<\/p>\n All the website\u2019s resources, such as CSS, JavaScript files, images, and videos, are the ingredients necessary to give the website its final look.<\/p>\n When the website achieves this state, you\u2019re dealing with the rendered HTML, more often called Document Object Model.<\/p>\n Martin also said that executing JavaScript is the very first rendering stage because JavaScript works like a recipe within a recipe.<\/p>\n In the not-too-distant past, Googlebot used to index the initial HTML version of a page and leave JavaScript rendering for late due to the cost and time-consuming nature of the process.<\/p>\n The SEO industry referred to that phenomenon as \u201cthe two waves of indexing.\u201d<\/p>\n However, now it seems that the two waves are no longer necessary.<\/p>\n Mueller and Splitt admitted that, nowadays, nearly every new website goes through the rendering stage by default<\/a>.<\/p>\n One of Google\u2019s goals is getting crawling, rendering, and indexing to happen closer together.<\/p>\n You can\u2019t force Google to index your new page.<\/p>\n How quickly this happens is also beyond your control. However, you can optimize your pages so that discovering and crawling run as smoothly as possible.<\/p>\n Here\u2019s what you need to do:<\/p>\n There are two important rules to follow to keep your pages indexable:<\/p>\n Robots.txt<\/a> is a file containing instructions for robots visiting your site.<\/p>\n You can use it to specify which crawlers are not allowed to visit certain pages or folders. All you have to do is use the disallow directive.<\/p>\n For example, if you don\u2019t want robots to visit pages and files in the folder titled \u201cexample,\u201d your robots.txt file should contain the following directives:<\/p>\n Sometimes, it\u2019s possible to block Googlebot from indexing valuable pages by mistake.<\/p>\n If you are concerned that your page is not indexed due to technical problems, you should definitely take a look at your robots.txt.<\/p>\n Googlebot is polite and won\u2019t pass any page it was told not to to the indexing pipeline. A way to express such a command is to put a noindex directive in:<\/p>\n Make sure that this directive doesn\u2019t appear on pages that should be indexed.<\/p>\n As we discussed, Google wants to avoid indexing duplicate content. If it finds two pages that look like copies of each other, it will likely only index one of them.<\/p>\n The canonical tag<\/a> was created to avoid misunderstandings and immediately direct Googlebot to the URL that the website owner considers the original version of the page.<\/p>\n Remember that the source code of a page you want to be present in the Google index shouldn\u2019t point to another page as canonical.<\/p>\nIs There A Guarantee That Your Page Will Be Indexed?<\/h2>\n
Crawl Demand And Crawl Budget<\/h2>\n
What Needs To Happen Before Your Page Is Indexed?<\/h2>\n
Content Discovery<\/h3>\n
\n
Crawling<\/h3>\n
Rendering<\/h3>\n
![\"rendering](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/09\/how-long-before-google-indexes-my-new-page-and-why-it-depends-via-sejournal-aniasiano.png\")
Screenshot from Google Search Console, September 2022<\/span><\/div>\nCan You Get Your Page Indexed Faster?<\/h2>\n
Make Sure Your Page Is Indexable<\/h3>\n
\n
User-agent: * Disallow: \/example\/<\/pre>\n
Submit A Sitemap<\/h3>\n
![\"Inspect](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/09\/how-long-before-google-indexes-my-new-page-and-why-it-depends-via-sejournal-aniasiano-1.png\")