The app linked above created the following output automatically using nothing more than two crawl exports!<\/p>\n
- \n
- \n
![\"A](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/05\/how-to-automate-ecommerce-category-page-creation-with-python-via-sejournal-leefootseo.png\")
Screenshot from Microsoft Excel, May 2022<\/em><\/span><\/div>\n<\/dt>\n<\/dl>\n<\/div>\nNotice how the suggested categories are automatically tied back to the existing parent category?<\/p>\n
\n- \n
- \n
![\"A](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/05\/how-to-automate-ecommerce-category-page-creation-with-python-via-sejournal-leefootseo-1.png\")
Screenshot from Microsoft Excel, May 2022<\/em><\/span><\/div>\n<\/dt>\n<\/dl>\n<\/div>\nThe app even shows how many products are available to populate the category.<\/p>\n
\n- \n
- \n
![\"the](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/05\/how-to-automate-ecommerce-category-page-creation-with-python-via-sejournal-leefootseo-2.png\")
Screenshot from Microsoft Excel, May 2022<\/em><\/span><\/div>\n<\/dt>\n<\/dl>\n<\/div>\nBenefits And Uses<\/h3>\n
- \n
- Improve relevancy to high-demand<\/strong>, competitive queries by creating new landing pages.<\/li>\n
- Increase the chance of relevant site links<\/strong> displaying underneath the parent category.<\/li>\n
- Reduce CPCs to the landing page<\/strong> through increased relevancy.<\/li>\n
- Potential to inform merchandising decisions.<\/strong> (If there is high search demand vs. low product count \u2013 there is a potential to widen the range.0\n
\n- \n
- \n
![\"A](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/05\/how-to-automate-ecommerce-category-page-creation-with-python-via-sejournal-leefootseo-3.png\")
Mock-up Screenshot from Google Chrome, May 2022<\/em><\/span><\/div>\n<\/dt>\n<\/dl>\n<\/div>\n<\/li>\n<\/ul>\nCreating the suggested subcategories for the parent sofa category would align the site to an additional 3,500 searches per month with relatively little effort.<\/p>\n
Features<\/h3>\n
- \n
- Create subcategory suggestions automatically.<\/li>\n
- Tie subcategories back to the parent category (cuts out a lot of guesswork!).<\/li>\n
- Match to a minimum of X products before recommending a category.<\/li>\n
- Check similarity to an existing category (X % fuzzy match) before recommending a new category.<\/li>\n
- Set minimum search volume\/CPC cut-off for category suggestions.<\/li>\n
- Supports search volume and CPC data from multiple countries.<\/li>\n<\/ul>\n
Getting Started\/Prepping The Files<\/h2>\n
To use this app you need two things.<\/p>\n
At a high level, the goal is to crawl the target website with two custom extractions.<\/p>\n
The internal_html.csv report is exported, along with an inlinks.csv export.<\/p>\n
These exports are then uploaded to the Streamlit app, where the opportunities are processed.<\/p>\n
Crawl And Extraction Setup<\/h3>\n
When crawling the site, you\u2019ll need to set two extractions in Screaming Frog \u2013 one to uniquely identify product pages and another to uniquely identify category pages.<\/p>\n
The Streamlit app understands the difference between the two types of pages when making recommendations for new pages.<\/p>\n
The trick is to find a unique element for each page type.<\/p>\n
(For a product page, this is usually the price or the returns policy, and for a category page, it\u2019s usually a filter sort element.)<\/p>\n
Extracting The Unique Page Elements<\/h3>\n
Screaming Frog allows for custom extractions<\/a> of content or code from a web page when crawled.<\/p>\n
This section may be daunting if you are unfamiliar with custom extractions, but it\u2019s essential for getting the correct data into the Streamlit app.<\/p>\n
The goal is to end up with something looking like the below image.<\/p>\n
(A unique extraction for product and category pages with no overlap.)<\/p>\n
\n- \n
- \n
![\"A](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/05\/how-to-automate-ecommerce-category-page-creation-with-python-via-sejournal-leefootseo-4.png\")
Screenshot from Screaming Frog SEO Spider, May 2022<\/em><\/span><\/div>\n<\/dt>\n<\/dl>\n<\/div>\nThe steps below walk you through manually extracting the price element for a product page.<\/p>\n
Then, repeat for a category page afterward.<\/p>\n
If you\u2019re stuck or would like to read more about the web scraper tool in Screaming Frog, the official documentation<\/a> is worth your time.<\/p>\n
Manually Extracting Page Elements<\/strong><\/p>\n
Let\u2019s start by extracting a unique element only found on a product page (usually the price).<\/p>\n
Highlight the price element on the page with the mouse, right-click and choose Inspect<\/em>.<\/p>\n
\n- \n
- \n
![\"A](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/05\/how-to-automate-ecommerce-category-page-creation-with-python-via-sejournal-leefootseo-5.png\")
Screenshot from Google Chrome, May 2022<\/em><\/span><\/div>\n<\/dt>\n<\/dl>\n<\/div>\nThis will open up the elements window with the correct HTML line already selected.<\/p>\n
Right-click the pre-selected line and choose Copy<\/em> > Copy selector<\/em>. That\u2019s it!<\/p>\n
\n- \n
- \n
![\"A](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/05\/how-to-automate-ecommerce-category-page-creation-with-python-via-sejournal-leefootseo-6.png\")
Screenshot from Google Chrome, May 2022<\/em><\/span><\/div>\n<\/dt>\n<\/dl>\n<\/div>\nOpen Screaming Frog and paste the copied selector into the custom extraction section. (Configuration<\/em> > Custom<\/em> > Extraction<\/em>).<\/p>\n
\n- \n
- \n
![\"A](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/05\/how-to-automate-ecommerce-category-page-creation-with-python-via-sejournal-leefootseo-7.png\")
Screenshot from Screaming Frog SEO Spider, May 2022<\/em><\/span><\/div>\n<\/dt>\n<\/dl>\n<\/div>\nName the extractor as \u201cproduct,\u201d select the CSSPath drop down and choose Extract Text<\/em>.<\/p>\n
Repeat the process to extract a unique element from a category page. It should look like this once completed for both product and category pages.<\/p>\n
\n- \n
- \n
![\"A](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/05\/how-to-automate-ecommerce-category-page-creation-with-python-via-sejournal-leefootseo-8.png\")
Screenshot from Screaming Frog SEO Spider, May 2022<\/em><\/span><\/div>\n<\/dt>\n<\/dl>\n<\/div>\nFinally, start the crawl.<\/p>\n
The crawl should look like this when viewing the Custom Extraction tab.<\/p>\n
\n- \n
- \nScreenshot from Screaming Frog SEO Spider, May 2022<\/em><\/span><\/div>\n<\/dt>\n<\/dl>\n<\/div>\n
Notice how the extractions are unique to each page type? Perfect.<\/p>\n
The script uses the extractor to identify the page type.<\/p>\n
Internally the app will convert the extractor to tags.<\/p>\n
(I mention this to stress that the extractors can be anything as long as they uniquely identify both page types.)<\/p>\n
\n- \n
- \n
![\"A](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/05\/how-to-automate-ecommerce-category-page-creation-with-python-via-sejournal-leefootseo-9.png\")
Screenshot from Microsoft Excel, May 2022<\/em><\/span><\/div>\n<\/dt>\n<\/dl>\n<\/div>\nExporting The Files<\/h3>\n
Once the crawl has been completed, the last step is to export two types of CSV files.<\/p>\n
- \n
- internal_html.csv.<\/li>\n
- inlinks to product pages.<\/li>\n<\/ul>\n
Go to the Custom Extraction<\/em> tab in Screaming Frog and highlight all URLs that have an extraction for products.<\/p>\n
(You will need to sort the column to group it.)<\/p>\n
\n- \n
- \n
![\"A](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/05\/how-to-automate-ecommerce-category-page-creation-with-python-via-sejournal-leefootseo-10.png\")
Screenshot from Screaming Frog SEO Spider, May 2022<\/em><\/span><\/div>\n<\/dt>\n<\/dl>\n<\/div>\nLastly, right-click the product URLs, select Export<\/em>, and then Inlinks.<\/em><\/p>\n
\n- \n
- \n
![\"A](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/05\/how-to-automate-ecommerce-category-page-creation-with-python-via-sejournal-leefootseo-11.png\")
Screenshot from Screaming Frog SEO Spider, May 2022<\/em><\/span><\/div>\n<\/dt>\n<\/dl>\n<\/div>\nYou should now have a file called inlinks.csv.<\/p>\n
Finally, we just need to export the internal_html.csv file.<\/p>\n
Click the Internal<\/em> tab, select HTML from the dropdown menu below and click on the adjacent Export button.<\/p>\n
Finally, choose the option to save the file as a .csv<\/p>\n
\n- \n
- \n
![\"A](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/05\/how-to-automate-ecommerce-category-page-creation-with-python-via-sejournal-leefootseo-12.png\")
Screenshot from Screaming Frog SEO Spider, May 2022<\/span><\/div>\n<\/dt>\n<\/dl>\n<\/div>\nCongratulations! You are now ready to use the Streamlit<\/a> app!<\/p>\n
Using The Streamlit App<\/h2>\n
Using the Streamlit app is relatively simple.<\/p>\n
The various options are set to reasonable defaults, but feel free to adjust the cut-offs to better suit your needs.<\/p>\n
I would highly recommend using a Keywords Everywhere API key<\/a> (although it is not strictly necessary as this can be looked up manually later with an existing tool if preferred.<\/p>\n
(The script pre-qualifies opportunity by checking for search volume. If the key is missing, the final output will contain more irrelevant words.)<\/p>\n
If you want to use a key, this is the section on the left to pay attention to.<\/p>\n
\n- \n
- \n
![\"A](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/05\/how-to-automate-ecommerce-category-page-creation-with-python-via-sejournal-leefootseo-13.png\")
Screenshot from Streamlit.io, May 2022<\/em><\/span><\/div>\n<\/dt>\n<\/dl>\n<\/div>\nOnce you have entered the API key and adjusted the cut-offs to your links, upload the inlinks.csv crawl.<\/p>\n
\n- \n
- \n
![\"A](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/05\/how-to-automate-ecommerce-category-page-creation-with-python-via-sejournal-leefootseo-14.png\")
Screenshot from Streamlit.io, May 2022<\/em><\/span><\/div>\n<\/dt>\n<\/dl>\n<\/div>\nOnce complete, a new prompt will appear adjacent to it, prompting you to upload the internal_html.csv crawl file.<\/p>\n
\n- \n
- \n
![\"A](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/05\/how-to-automate-ecommerce-category-page-creation-with-python-via-sejournal-leefootseo-15.png\")
Screenshot from Streamlit.io, May 2022<\/em><\/span><\/div>\n<\/dt>\n<\/dl>\n<\/div>\nFinally, a new box will appear asking you to select the product and column names from the uploaded crawl file to be mapped correctly.<\/p>\n
\n- \n
- \n
![\"A](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/05\/how-to-automate-ecommerce-category-page-creation-with-python-via-sejournal-leefootseo-16.png\")
Screenshot from Streamlit.io, May 2022<\/em><\/span><\/div>\n<\/dt>\n<\/dl>\n<\/div>\nClick submit and the script will run. Once complete, you will see the following screen and can download a handy .csv export.<\/p>\n
\n- \n
- \n
![\"A](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/05\/how-to-automate-ecommerce-category-page-creation-with-python-via-sejournal-leefootseo-17.png\")
Screenshot from Streamlit.io, May 2022<\/em><\/span><\/div>\n<\/dt>\n<\/dl>\n<\/div>\nHow The Script Works<\/h2>\n
Before we dive into the script\u2019s output, it will help to explain what\u2019s going on under the hood at a high level.<\/p>\n
At a glance:<\/p>\n
- \n
- Generate thousands of keywords<\/strong> by generating n-grams from product page H1 headings.<\/li>\n
- Qualify keywords by checking whether the word is <\/strong>in an exact or fuzzy match in a product heading.<\/li>\n
- Further qualify keywords by checking for search volume<\/strong> using the Keywords Everywhere API (optional but recommended).<\/li>\n
- Check whether an existing category already exists<\/strong> using a fuzzy match (can find words out of order, different tenses, etc.).<\/li>\n
- Uses the inlinks report to assign suggestions to a parent category automatically<\/strong>.<\/li>\n<\/ul>\n
N-gram Generation<\/h2>\n
The script creates hundreds of thousands of n-grams from the product page H1s, most of which are completely nonsensical.<\/p>\n
In my example for this article, n-grams generated 48,307 words \u2013 so this will need to be filtered!<\/p>\n
\n- \n
- \n
![\"An](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/05\/how-to-automate-ecommerce-category-page-creation-with-python-via-sejournal-leefootseo-18.png\")
Screenshot from Microsoft Excel, May 2022<\/em><\/span><\/div>\n<\/dt>\n<\/dl>\n<\/div>\nThe first step in the filtering process is to check whether the keywords generated via n-grams are found at least X times within the product name column.<\/p>\n
(This can be in an exact or fuzzy match.)<\/p>\n
Anything not found is immediately discarded, which usually removes around 90% of the generated keywords.<\/p>\n
The second filtering stage is to check whether the remaining keywords have search demand.<\/p>\n
Any keywords without search demand are then discarded too.<\/p>\n
(This is why I recommend using the Keywords Everywhere API when running the script, which results in a more refined output.)<\/p>\n
It\u2019s worth noting you can do this manually afterward by searching Semrush\/Ahrefs etc., discarding any keywords without search volume, and running a VLOOKUP in Microsoft Excel.<\/p>\n
Cheaper if you have an existing subscription.<\/p>\n
Recommendations Tied To Specific Landing Pages<\/h3>\n
Once the keyword list has been filtered the script uses the inlinks report to tie the suggested subcategory back to the landing page.<\/p>\n
Earlier versions did not do this, but I realized that leveraging the inlinks.csv report meant it was possible.<\/p>\n
It really helps understand the context of the suggestion at a glance during QA.<\/p>\n
This is the reason the script requires two exports to work correctly.<\/p>\n
Limitations<\/h2>\n
- \n
- Not checking search volumes will result in more results for QA.<\/strong> (Even if you don\u2019t use the Keywords Everywhere API, I recommend shortlisting by filtering out 0 search volume afterward.)<\/li>\n
- Some irrelevant keywords will have search volume and appear in the final report<\/strong>, even if keyword volume has been checked.<\/li>\n
- Words will typically appear in the singular sense for the final output<\/strong> (because products are singular and categories are pluralized if they sell more than a single product). It\u2019s easy enough to add an \u201cs\u201d to the end of the suggestion though.<\/li>\n<\/ul>\n
User Configurable Variables<\/h2>\n
I\u2019ve selected what I consider to be sensible default options.<\/p>\n
But here is a run down if you\u2019d like to tweak and experiment.<\/p>\n
- \n
- Minimum products to match to (exact match)<\/strong> \u2013 The minimum number of products that must exist before suggesting the new category in an exact match.<\/li>\n
- Minimum products to match to (fuzzy match)<\/strong> \u2013 The minimum number of products that must exist before suggesting the new category in a fuzzy match, (words can be found in any order).<\/li>\n
- Minimum similarity to an existing category<\/strong> \u2013 This checks whether a category already exists in a fuzzy match before making the recommendation. The closer to 100 = stricter matching.<\/li>\n
- Minimum CPC in $<\/strong> \u2013 The minimum dollar amount of the suggested category keyword. (Requires the Keywords Everywhere API.)<\/li>\n
- Minimum search volume<\/strong> \u2013 The minimum search volume of the suggested category keyword. (Requires Keywords Everywhere API.)<\/li>\n
- Keywords Everywhere API key<\/strong> \u2013 Optional, but recommended. Used to pull in CPC\/search volume data. (Useful for shortlisting categories.)<\/li>\n
- Set the country to pull search data from<\/strong> \u2013 Country-specific search data is available. (Default is the USA.)<\/li>\n
- Set the currency for CPC data<\/strong> \u2013 Country-specific CPC data is available. (Default USD.)<\/li>\n
- Keep the longest word suggestion<\/strong> \u2013 With similar word suggestions, this option will keep the longest match.<\/li>\n
- Enable fuzzy product matching<\/strong> \u2013 This will search for product names in a fuzzy match. (Words can be found out of order, recommended \u2013 but slow and CPU intensive.)<\/li>\n<\/ul>\n
Conclusion<\/h2>\n
With a small amount of preparation, it is possible to tap into a large amount of organic opportunity while improving the user experience.<\/p>\n
Although this script was created with an ecommerce focus, according to feedback, it works well for other site types such as job listing sites.<\/p>\n
So even if your site isn\u2019t an ecommerce site, it\u2019s still worth a try.<\/p>\n
Python<\/a> enthusiast?<\/p>\n
I released the source code for a non-Streamlit version here<\/a>.<\/p>\n
More resources:<\/strong><\/p>\n
\nFeatured Image: patpitchaya\/Shutterstock<\/em><\/p>\n
- Minimum products to match to (fuzzy match)<\/strong> \u2013 The minimum number of products that must exist before suggesting the new category in a fuzzy match, (words can be found in any order).<\/li>\n
- Some irrelevant keywords will have search volume and appear in the final report<\/strong>, even if keyword volume has been checked.<\/li>\n
- Not checking search volumes will result in more results for QA.<\/strong> (Even if you don\u2019t use the Keywords Everywhere API, I recommend shortlisting by filtering out 0 search volume afterward.)<\/li>\n
- Qualify keywords by checking whether the word is <\/strong>in an exact or fuzzy match in a product heading.<\/li>\n
- Generate thousands of keywords<\/strong> by generating n-grams from product page H1 headings.<\/li>\n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- Increase the chance of relevant site links<\/strong> displaying underneath the parent category.<\/li>\n
- Improve relevancy to high-demand<\/strong>, competitive queries by creating new landing pages.<\/li>\n
- \n
- \n