Indexing: How Search Engines Work Today<\/h2>\n
Let\u2019s look at the nuts and bolts of how search engines operate.<\/p>\n
This article focuses on indexing. So, let\u2019s dive in.<\/p>\n
Web Indexing<\/h3>\n
Indexing is where the ranking process begins after a website has been crawled.<\/p>\n
Indexing essentially refers to adding a webpage\u2019s content to Google to be considered for rankings.<\/p>\n
When you create a new page on your site, there are several ways it can be indexed.<\/p>\n
The simplest method of getting a page indexed is to do absolutely nothing.<\/p>\n
Google has crawlers following links, and thus, provided your site is in the index already, and the new content is linked to from within your site, Google will eventually discover it and add it to its index. More on this later.<\/p>\n
How To Get A Page Indexed Faster<\/h2>\n
But what if you want Googlebot to get to your page faster?<\/p>\n
This can be important if you have timely content or if you\u2019ve made an important change to a page you need Google to know about.<\/p>\n
I use faster methods when I\u2019ve optimized a critical page or adjusted the title and description to improve click-throughs.<\/p>\n
I want to know specifically when they were picked up and displayed in the SERPs to know where the improvement measurement starts.<\/p>\n
In these instances, there are a few additional methods you can use.<\/p>\n
1. XML Sitemaps<\/h3>\n
XML sitemaps<\/a> are the oldest and a generally reliable way to call a search engine\u2019s attention to content.<\/p>\n An XML sitemap gives search engines a list of all the pages on your site, as well as additional details about it, such as when it was last modified.<\/p>\n A sitemap can be submitted to Bing<\/a> via Bing Webmaster Tools<\/a>, and it can also be submitted to Google via Search Console<\/a>.<\/p>\n Definitely recommended!<\/p>\n But when you need a page indexed immediately, it\u2019s not particularly reliable.<\/p>\n In Search Console, you can \u201cRequest Indexing.\u201d<\/p>\n You begin by clicking on the top search field, which reads by default, \u201cInspect and URL in domain.com.\u201d<\/p>\n Enter the URL you want to be indexed, then hit Enter.<\/p>\n If the page is already known to Google, you will be presented with a bunch of information. We won\u2019t get into that here, but I recommend logging in and seeing what\u2019s there if you haven\u2019t already.<\/p>\n For our purposes here, the important button appears whether the page has been indexed or not \u2013 meaning that it\u2019s good for content discovery or just requesting Google to understand a recent change.<\/p>\n You\u2019ll find the button shown below.<\/p>\n Within a few seconds to a few minutes, you can search the new content or URL in Google and find the change or new content picked up.<\/p>\n Bing has an open protocol based on a push method of alerting search engines of new or updated content.<\/p>\n This new search engine indexing protocol is called, IndexNow<\/a>.<\/p>\n It\u2019s called a push protocol because the idea is to alert search engines using IndexNow about new or updated content, which will cause them to come and index it.<\/p>\n An example of a pull protocol is the old XML sitemap way that depends on a search engine crawler to decide to visit and index it (or to be fetched by Search Console).<\/p>\n The benefit of IndexNow is that it wastes less web hosting and data center resources, which is not only more environmentally friendly but it saves on bandwidth resources.<\/p>\n The biggest benefit, however, is faster content indexing.<\/p>\n IndexNow is currently used only by Bing and Yandex.<\/p>\n Implementing IndexNow is easy:<\/strong><\/p>\n In addition to participating in IndexNow, consider a Bing Webmaster Tools account.<\/p>\n If you don\u2019t have a Bing Webmaster Tools account, I can\u2019t recommend it enough.<\/p>\n The information provided is substantial and will help you better assess problem areas and improve your rankings on Bing, Google, and anywhere else \u2013 and probably provide a better user experience.<\/p>\n But to get your content indexed, you simply need to click: Configure My Site > Submit URLs.<\/em><\/p>\n Enter the URL(s) you want to be indexed and click \u201cSubmit.\u201d<\/p>\n So, that\u2019s most of what you need to know about indexing and how search engines do it (with an eye toward where things are going).<\/p>\n More details at the Bing Webmaster Tools URL Submission Tool help page<\/a>.<\/p>\n There is also a Bing Webmaster Tools Indexing API<\/a> that can also speed up the time that content appears in Bing\u2019s search results within hours.<\/p>\n More information about the Bing Indexing API is here<\/a>.<\/p>\n We can\u2019t talk about indexing without talking about the crawl budget<\/a>.<\/p>\n Basically, crawl budget is a term used to describe the number of resources that Google will expend crawling a website.<\/p>\n The budget assigned is based on a combination of factors, the two central ones being:<\/p>\n If you run a major news site with constantly updating content that search engine users will want to be aware of, your site will get crawled frequently (dare I say \u2013 constantly).<\/p>\n If you run a small barbershop, have a couple of dozen links, and rightfully are not deemed important in this context (you may be an important barber in the area, but you\u2019re not important when it comes to the crawl budget), then the budget will be low.<\/p>\n You can read more about crawl budgets and how they\u2019re determined in Google\u2019s explanation<\/a>.<\/p>\n Indexing by Google begins with crawling, which has two kinds:<\/a><\/p>\n Optimizing websites for search engines begins with good content and ends with sending it off to get indexed.<\/p>\n Whether you do that with an XML sitemap, Google Search Console URL Submission Tool, Bing Webmaster Tools, or IndexNow, getting that content indexed is when your webpage begins its journey to the top of the search results (if everything works out!).<\/p>\n That\u2019s why it\u2019s important to understand how search indexing works.<\/p>\n Read more about the key factors that influence search engine result pages in How Search Engines Work<\/a>.<\/strong><\/p>\n Featured Image: Overearth\/Shutterstock<\/em><\/p>\n2. Request Indexing With Google Search Console<\/h3>\n
![\"Request](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2023\/01\/website-indexing-for-search-engines-how-does-it-work-via-sejournal-beanstalkim.png\")
Screenshot from Google Search Console, December 2022<\/span><\/div>\n3. Participate In Bing\u2019s IndexNow<\/h3>\n
4. Bing Webmaster Tools<\/h3>\n
![\"Bing](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2023\/01\/website-indexing-for-search-engines-how-does-it-work-via-sejournal-beanstalkim-1.png\")
Screenshot from Bing Webmaster Tools, January 2023<\/span><\/div>\nCrawl Budget<\/h2>\n
\n
Google Has Two Kinds Of Crawling<\/h2>\n
\n
Discover How Search Engines Work<\/h2>\n
\n