Google\u2019s search index is \u201cwell over 100,000,000 gigabytes<\/a> in size.\u201d<\/p>\n Googlebot and variants (smartphones, news, images, etc.) have certain constraints for the frequency of JavaScript rendering or the size of the resources.<\/p>\n Google uses crawling constraints<\/a> to protect its own crawling resources and systems.<\/p>\n For instance, if a news website refreshes the recommended articles every 15 seconds, Googlebot might start to skip the frequently refreshed sections \u2013 since they won\u2019t be relevant or valid after 15 seconds.<\/p>\n Years ago, Google announced that it does not crawl or use resources bigger than 15 MB.<\/p>\n On June 28, 2022, Google republished this blog post<\/a> by stating that it does not use the excess part of the resources after 15 MB for crawling.<\/p>\n To emphasize that it rarely happens, Google stated that the \u201cmedian size of an HTML file is 500 times smaller\u201d than 15 MB.<\/p>\n Above, HTTPArchive.org<\/a> shows the median desktop and mobile HTML file size. Thus, most websites do not have the problem of the 15 MB constraint for crawling.<\/p>\n But, the web is a big and chaotic place.<\/p>\n Understanding the nature of the 15 MB crawling limit and ways to analyze it is important for SEOs.<\/p>\n An image, video, or bug can cause crawling problems, and this lesser-known SEO information can help projects protect their organic search value.<\/p>\n No.<\/p>\n 15 MB Googlebot crawling limit is for all indexable and crawlable documents, including Google Earth, Hancom Hanword (.hwp), OpenOffice text (.odt), and Rich Text Format (.rtf), or other Googlebot-supported file types.<\/p>\n No, every resource is evaluated separately by the 15 MB crawling limit.<\/p>\n If the HTML document is 14.99 MB, and the featured image of the HTML document is 14.99 MB again, they both will be crawled and used by Googlebot.<\/p>\n The HTML document\u2019s size is not summed with the resources that are linked via HTML tags.<\/p>\n Yes, inlined CSS, JS, or the Data URI are counted and used in the HTML document size.<\/p>\n Thus, if the document exceeds 15 MB due to inlined resources and commands, it will affect the specific HTML document\u2019s crawlability.<\/p>\n No, Google crawling systems do not stop crawling the resources that are bigger than the 15 MB limit.<\/p>\n They continue to fetch the file and use only the smaller part than the 15 MB.<\/p>\n For an image bigger than 15 MB, Googlebot can chunk the image until the 15 MB with the help of \u201ccontent range.\u201d<\/p>\n The Content-Range is a response header that helps Googlebot or other crawlers and requesters perform partial requests.<\/p>\n You can use Google Chrome Developer Tools<\/a> to audit the resource size manually.<\/p>\n Follow the steps below on Google Chrome.<\/p>\n Below, you can see an example of a searchenginejournal.com homepage HTML document, which is bigger than 77 KB.<\/p>\n Use Python to audit the HTML document size automatically and in bulk. Advertools and Pandas are two useful Python Libraries to automate and scale SEO tasks<\/a>.<\/p>\n Follow the instructions below.<\/p>\n The code block above extracts the sitemap URLs and crawls them.<\/p>\n The last line of the code is only for creating a data frame with a descending order based on the sizes.<\/p>\n You can see the sizes of HTML documents as above.<\/p>\n The biggest HTML document in this example is around 700 KB, which is a category page.<\/p>\n So, this website is safe for 15 MB constraints. But, we can check beyond this.<\/p>\n Puppeteer is used to check the size of CSS and JS Resources.<\/p>\n Puppeteer<\/a> is a NodeJS package to control Google Chrome with headless mode for browser automation and website tests.<\/p>\n Most SEO pros use Lighthouse or Page Speed Insights API<\/a> for their performance tests. But, with the help of Puppeteer, every technical aspect and simulation can be analyzed.<\/p>\n Follow the code block below.<\/p>\n If you do not know JavaScript or didn\u2019t finish any kind of Puppeteer tutorial, it might be a little harder for you to understand these code blocks. But, it is actually simple.<\/p>\n It basically opens a URL, takes all the resources, and gives their \u201ctransferSize\u201d, \u201cencodedSize\u201d, and \u201cdecodedSize.\u201d<\/p>\n In this example, \u201cdecodedSize\u201d is the size that we need to focus on. Below, you can see the result in the form of an XLS file.<\/p>\n If you want to automate these processes for every URL again, you will need to use a for loop in the \u201cawait.page.goto()\u201d command.<\/p>\n According to your preferences, you can put every web page into a different worksheet or attach it to the same worksheet by appending it.<\/p>\n The 15 MB of Googlebot crawling constraint is a rare possibility that will block your technical SEO processes for now, but HTTPArchive.org shows that the median video, image, and JavaScript<\/a> sizes have increased in the last few years.<\/p>\n The median image size on the desktop has exceeded 1 MB.<\/p>\n The video bytes exceed 5 MB in total.<\/p>\n In other words, from time to time, these resources \u2013 or some parts of these resources \u2013 might be skipped by Googlebot<\/a>.<\/p>\n Thus, you should be able to control them automatically, with bulk methods to make time and not skip.<\/p>\n More resources:<\/strong><\/p>\n Featured Image: BestForBest\/Shutterstock<\/em><\/p>\n![\"timeline](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/09\/find-resources-bigger-than-15-mb-for-better-googlebot-crawling-via-sejournal-koraygubur.png\")
Screenshot from the author, August 2022<\/em><\/span><\/div>\n![\"Find](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/09\/find-resources-bigger-than-15-mb-for-better-googlebot-crawling-via-sejournal-koraygubur.jpg\")
<\/p>\n<\/p>\nIs 15 MB Googlebot Crawling Limit Only For HTML Documents?<\/h2>\n
Are Image And Video Sizes Summed With HTML Document?<\/h2>\n
Does Inlined CSS, JS, Or Data URI Bloat HTML Document Size?<\/h2>\n
Does Google Stop Crawling The Resource If It Is Bigger Than 15 MB?<\/h2>\n
How To Audit The Resource Size Manually?<\/h2>\n
\n
![\"search](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/09\/find-resources-bigger-than-15-mb-for-better-googlebot-crawling-via-sejournal-koraygubur-1.png\")
Screenshot by author, August 2022<\/em><\/span><\/div>\nHow To Audit The Resource Size Automatically And Bulk?<\/h2>\n
\n
import advertools as adv import pandas as pd df = adv.sitemap_to_df(\"https:\/\/www.holisticseo.digital\/sitemap.xml\") adv.crawl(df[\"loc\"], output_file=\"output.jl\", custom_settings={\"LOG_FILE\":\"output_1.log\"}) df = pd.read_json(\"output.jl\", lines=True) df[[\"url\", \"size\"]].sort_values(by=\"size\", ascending=False)<\/code><\/pre>\n![\"holisticseo.com](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/09\/find-resources-bigger-than-15-mb-for-better-googlebot-crawling-via-sejournal-koraygubur-2.png\")
Image created by author, August 2022<\/em><\/span><\/div>\nHow To Check The Sizes of CSS And JS Resources?<\/h2>\n
const puppeteer = require('puppeteer'); const XLSX = require(\"xlsx\"); const path = require(\"path\"); (async () => { const browser = await puppeteer.launch({ headless: false }); const page = await browser.newPage(); await page.goto('https:\/\/www.holisticseo.digital'); console.log('Page loaded'); const perfEntries = JSON.parse( await page.evaluate(() => JSON.stringify(performance.getEntries())) ); console.log(perfEntries); const workSheetColumnName = [ \"name\", \"transferSize\", \"encodedSize\", \"decodedSize\" ] const urlObject = new URL(\"https:\/\/www.holisticseo.digital\") const hostName = urlObject.hostname const domainName = hostName.replace(\"\\www.|.com\", \"\"); console.log(hostName) console.log(domainName) const workSheetName = \"Users\"; const filePath = `.\/${domainName}`; const userList = perfEntries; const exportPerfToExcel = (userList) => { const data = perfEntries.map(url => { return [url.name, url.transferSize, url.encodedBodySize, url. decodedBodySize]; }) const workBook = XLSX.utils.book_new(); const workSheetData = [ workSheetColumnName, ...data ] const workSheet = XLSX.utils.aoa_to_sheet(workSheetData); XLSX.utils.book_append_sheet(workBook, workSheet, workSheetName); XLSX.writeFile(workBook, path.resolve(filePath)); return true; } exportPerfToExcel(userList) \/\/browser.close(); })();<\/code><\/pre>\n![\"Resource](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/09\/find-resources-bigger-than-15-mb-for-better-googlebot-crawling-via-sejournal-koraygubur-3.png\")
Byte sizes of the resources from the website.<\/em><\/span><\/div>\nConclusion<\/h2>\n
![\"Timeseries](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/09\/find-resources-bigger-than-15-mb-for-better-googlebot-crawling-via-sejournal-koraygubur-4.png\")
Screenshot by author, August 2022<\/em><\/span><\/div>\n![\"Timeseries](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/09\/find-resources-bigger-than-15-mb-for-better-googlebot-crawling-via-sejournal-koraygubur-5.png\")
Screenshot by author, August 2022<\/em><\/span><\/div>\n
\n