![\"Compare](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/09\/compare-vocabulary-differences-between-ranking-web-pages-on-serp-with-python-via-sejournal-koraygubur.jpg\")
<\/p>\nVocabulary size and difference are semantic and linguistic concepts for mathematical and qualitative linguistics.<\/p>\n
For example, Heaps\u2019 law claims that the length of the article and vocabulary size are correlative. Still, after a certain threshold, the same words continue to appear without improving vocabulary size.<\/p>\n
The Word2Vec uses Continuous Bag of Words (CBOW) and Skip-gram to understand the locally contextually relevant words and their distance to each other. At the same time, GloVe tries to use matrix factorization with context windows.<\/p>\n
Zipf\u2019s law is a complementary theory to Heaps\u2019 law. It states that the most frequent and second most frequent words have a regular percentage difference between them.<\/p>\n
There are other distributional semantics and linguistic theories in statistical natural language processing.<\/p>\n
But \u201cvocabulary comparison\u201d is a fundamental methodology for search engines to understand \u201ctopicality differences,\u201d \u201cthe main topic of the document,\u201d or overall \u201cexpertise of the document.\u201d<\/p>\n
Paul Haahr of Google stated that it compares the \u201cquery vocabulary\u201d to the \u201cdocument vocabulary.\u201d<\/p>\n
David C. Taylor and his designs for context domains involve certain word vectors in vector search to see which document and which document subsection are more about what, so a search engine can rank and rerank documents based on search query modifications.<\/p>\n
Comparing vocabulary differences between ranking web pages on the search engine results page (SERP) helps SEO pros see what contexts, concurrent words, and word proximity they are skipping compared to their competitors.<\/p>\n
It is helpful to see context differences in the documents.<\/p>\n
In this guide, the Python programming language is used to search on Google and take SERP items (snippets) to crawl their content, tokenize and compare their vocabulary to each other.<\/p>\n
<\/p>\n
<\/p>\n
To compare the vocabularies of ranking web documents (with Python), the used libraries and packages of Python programming language are listed below.<\/p>\n
The steps for comparing the vocabulary size and content between ranking web pages are listed below.<\/p>\n
Import the necessary Python libraries and packages by using the \u201cfrom\u201d and \u201cimport\u201d commands and methods.<\/p>\n
from googlesearch import search from urllib.parse import urlparse import requests from fake_useragent import UserAgent import advertools as adv import pandas as pd from nltk.tokenize import word_tokenize import nltk from collections import Counter from nltk.corpus import stopwords import string nltk.download()<\/code><\/pre>\nUse the \u201cnltk.download\u201d only if you\u2019re using NLTK for the first time. Download all the corpora, models, and packages. It will open a window as below.<\/p>\n
![\"NLTK](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/09\/compare-vocabulary-differences-between-ranking-web-pages-on-serp-with-python-via-sejournal-koraygubur.png\")
Screenshot from author, August 2022<\/em><\/span><\/div>\nRefresh the window from time to time; if everything is green, close the window so that the code running on your code editor stops and completes.<\/p>\n
If you do not have some modules above, use the \u201cpip install\u201d method for downloading them to your local machine. If you have a closed-environment project, use a virtual environment in Python.<\/p>\n
2. Perform A Google Search To Retrieve The Result URLs On The Search Engine Result Pages<\/h3>\n
To perform a Google search to retrieve the result URLs on the SERP items, use a for loop in the \u201csearch\u201d object, which comes from \u201cGooglesearch\u201d package.<\/p>\n
serp_item_url = [] for i in search(\"search engine optimization\", num=10, start=1, stop=10, pause=1, lang=\"en\", country=\"us\"): serp_item_url.append(i) print(i)<\/code><\/pre>\nThe explanation of the code block above is:<\/p>\n
\n- Create an empty list object, such as \u201cserp_item_url.\u201d<\/li>\n
- Start a for loop within the \u201csearch\u201d object that states a query, language, number of results, first and last result, and country restriction.<\/li>\n
- Append all the results to the \u201cserp_item_url\u201d object, which entails a Python list.<\/li>\n
- Print all the URLs that you have retrieved from Google SERP.<\/li>\n<\/ul>\n
You can see the result below.<\/p>\n
The ranking URLs for the query \u201csearch engine optimization\u201d is given above.<\/p>\n
The next step is parsing these URLs for further cleaning.<\/p>\n
Because if the results involve \u201cvideo content,\u201d it won\u2019t be possible to perform a healthy text analysis if they do not have a long video description or too many comments, which is a different content type<\/a>.<\/p>\n3. Clean The Video Content URLs From The Result Web Pages<\/h3>\n
To clean the video content URLs, use the code block below.<\/p>\n
parsed_urls = [] for i in range(len(serp_item_url)): parsed_url = urlparse(serp_item_url[i]) i += 1 full_url = parsed_url.scheme + ':\/\/' + parsed_url.netloc + parsed_url.path if ('youtube' not in full_url and 'vimeo' not in full_url and 'dailymotion' not in full_url and \"dtube\" not in full_url and \"sproutvideo\" not in full_url and \"wistia\" not in full_url): parsed_urls.append(full_url)<\/code><\/pre>\nThe video search engines such as YouTube, Vimeo, Dailymotion, Sproutvideo, Dtube, and Wistia are cleaned from the resulting URLs if they appear in the results.<\/p>\n
You can use the same cleaning methodology for the websites that you think will dilute the efficiency of your analysis or break the results with their own content type.<\/p>\n
For example, Pinterest or other visual-heavy websites might not be necessary to check the \u201cvocabulary size\u201d differences between competing documents.<\/p>\n
Explanation of code block above:<\/p>\n
\n- Create an object such as \u201cparsed_urls.\u201d<\/li>\n
- Create a for loop in the range of length of the retrieved result URL count.<\/li>\n
- Parse the URLs with \u201curlparse\u201d from \u201cURLlib.\u201d<\/li>\n
- Iterate by increasing the count of \u201ci.\u201d<\/li>\n
- Retrieve the full URL by uniting the \u201cscheme\u201d, \u201cnetloc\u201d, and \u201cpath.\u201d<\/li>\n
- Perform a search with conditions in the \u201cif\u201d statement with \u201cand\u201d conditions for the domains to be cleaned.<\/li>\n
- Take them into a list with \u201cdict.fromkeys\u201d method.<\/li>\n
- Print the URLs to be examined.<\/li>\n<\/ul>\n
You can see the result below.<\/p>\n
![\"Video](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/09\/compare-vocabulary-differences-between-ranking-web-pages-on-serp-with-python-via-sejournal-koraygubur-1.png\")
Screenshot from author, August 2022<\/em><\/span><\/div>\n4. Crawl The Cleaned Examine URLs For Retrieving Their Content<\/h3>\n
Crawl the cleaned examine URLs for retrieving their content with advertools.<\/p>\n
You can also use requests with a for loop and list append methodology, but advertools is faster for crawling and creating the data frame with the resulting output.<\/p>\n
With requests, you manually retrieve and unite all the \u201cp\u201d and \u201cheading\u201d elements.<\/p>\n
adv.crawl(examine_urls, output_file=\"examine_urls.jl\", follow_links=False, custom_settings={\"USER_AGENT\": UserAgent().random, \"LOG_FILE\": \"examine_urls.log\", \"CRAWL_DELAY\": 2}) crawled_df = pd.read_json(\"examine_urls.jl\", lines=True) crawled_df<\/code><\/pre>\nExplanation of code block above:<\/p>\n
\n- Use \u201cadv.crawl\u201d for crawling the \u201cexamine_urls\u201d object.<\/li>\n
- Create a path for output files with \u201cjl\u201d extension, which is smaller than others.<\/li>\n
- Use \u201cfollow_links=false\u201d to stop crawling only for listed URLs.<\/li>\n
- Use custom settings to state a \u201crandom user agent\u201d and a crawl log file if some URLs do not answer the crawl requests. Use a crawl delay configuration to prevent 429 status code possibility.<\/li>\n
- Use pandas \u201cread_json\u201d with the \u201clines=True\u201d parameter to read the results.<\/li>\n
- Call the \u201ccrawled_df\u201d as below.<\/li>\n<\/ul>\n
You can see the result below.<\/p>\n
![\"Compare](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/09\/compare-vocabulary-differences-between-ranking-web-pages-on-serp-with-python-via-sejournal-koraygubur-2.png\")
Screenshot from author, August 2022<\/em><\/span><\/div>\nYou can see our result URLs and all their on-page SEO elements, including response headers, response sizes, and structured data information.<\/p>\n
5. Tokenize The Content Of The Web Pages For Text Processing In NLP Methodologies<\/h3>\n
Tokenization of the content of the web pages requires choosing the \u201cbody_text\u201d column of advertools crawl output and using the \u201cword_tokenize\u201d from NLTK.<\/p>\n
crawled_df[\"body_text\"][0]<\/code><\/pre>\nThe code line above calls the entire content of one of the result pages as below.<\/p>\n
![\"Compare](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/09\/compare-vocabulary-differences-between-ranking-web-pages-on-serp-with-python-via-sejournal-koraygubur-3.png\")
Screenshot from author, August 2022<\/em><\/span><\/div>\nTo tokenize these sentences, use the code block below.<\/p>\n
tokenized_words = word_tokenize(crawled_df[\"body_text\"][0]) len(tokenized_words)<\/code><\/pre>\nWe tokenized the content of the first document and checked how many words it had.<\/p>\n
![\"Compare](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/09\/compare-vocabulary-differences-between-ranking-web-pages-on-serp-with-python-via-sejournal-koraygubur-4.png\")
Screenshot from author, August 2022<\/em><\/span><\/div>\nThe first document we tokenized for the query \u201csearch engine optimization\u201d has 11211 words. And boilerplate content is included in this number.<\/p>\n
6. Remove The Punctuations And Stop Words From Corpus<\/h3>\n
Remove the punctuations, and the stop words, as below.<\/p>\n
<\/pre>\nstop_words = set(stopwords.words(\"english\"))\ntokenized_words = [word for word in tokenized_words if not word.lower() in stop_words and word.lower() not in string.punctuation] len(tokenized_words)<\/code><\/pre>\n<\/pre>\nExplanation of code block above:<\/p>\n
\n- Create a set with the \u201cstopwords.words(\u201cenglish\u201d)\u201d to include all the stop words in the English language. Python sets do not include duplicate values; thus, we used a set rather than a list to prevent any conflict.<\/li>\n
- Use list comprehension with \u201cif\u201d and \u201celse\u201d statements.<\/li>\n
- Use the \u201clower\u201d method to compare the \u201cAnd\u201d or \u201cTo\u201d types of words properly to their lowercase versions in the stop words list.<\/li>\n
- Use the \u201cstring\u201d module and include \u201cpunctuations.\u201d A note here is that the string module might not include all the punctuations that you might need. For these situations, create your own punctuation list and replace these characters with space using the regex, and \u201cregex.sub.\u201d<\/li>\n
- Optionally, to remove the punctuations or some other non-alphabetic and numeric values, you can use the \u201cisalnum\u201d method of Python strings. But, based on the phrases, it might give different results. For example, \u201cisalnum\u201d would remove a word such as \u201ckeyword-related\u201d since the \u201c-\u201d at the middle of the word is not alphanumeric. But, string.punctuation wouldn\u2019t remove it since \u201ckeyword-related\u201d is not punctuation, even if the \u201c-\u201d is.<\/li>\n
- Measure the length of the new list.<\/li>\n<\/ul>\n
The new length of our tokenized word list is \u201c5319\u201d. It shows that nearly half of the vocabulary of the document consists of stop words or punctuations.<\/p>\n
It might mean that only 54% of the words are contextual, and the rest is functional.<\/p>\n
7. Count The Number Of Occurrences Of The Words In The Content Of The Web Pages<\/h3>\n
To count the occurrences of the words from the corpus, the \u201cCounter\u201d object from the \u201cCollections\u201d module is used as below.<\/p>\n
counted_tokenized_words = Counter(tokenized_words) counts_of_words_df = pd.DataFrame.from_dict( counted_tokenized_words, orient=\"index\").reset_index() counts_of_words_df.sort_values(by=0, ascending=False, inplace=True) counts_of_words_df.head(50)<\/code><\/pre>\nAn explanation of the code block is below.<\/p>\n
\n- Create a variable such as \u201ccounted_tokenized_words\u201d to involve the Counter method results.<\/li>\n
- Use the \u201cDataFrame\u201d constructor from the Pandas to construct a new data frame from Counter method results for the tokenized and cleaned text.<\/li>\n
- Use the \u201cfrom_dict\u201d method because \u201cCounter\u201d gives a dictionary object.<\/li>\n
- Use \u201csort_values\u201d with \u201cby=0\u201d which means sort based on the rows, and \u201cascending=False\u201d means to put the highest value to the top. \u201cInpace=True\u201d is for making the new sorted version permanent.<\/li>\n
- Call the first 50 rows with the \u201chead()\u201d method of pandas to check the first look of the data frame.<\/li>\n<\/ul>\n
You can see the result below.<\/p>\n
![\"counts](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/09\/compare-vocabulary-differences-between-ranking-web-pages-on-serp-with-python-via-sejournal-koraygubur-5.png\")
Screenshot from author, August 2022<\/em><\/span><\/div>\nWe do not see a stop word on the results, but some interesting punctuation marks remain.<\/p>\n
That happens because some websites use different characters for the same purposes, such as curly quotes (smart quotes), straight single quotes, and double straight quotes.<\/p>\n
And string module\u2019s \u201cfunctions\u201d module doesn\u2019t involve those.<\/p>\n
Thus, to clean our data frame, we will use a custom lambda function as below.<\/p>\n
removed_curly_quotes = \"\u2019\u201c\u201d\" counts_of_words_df[\"index\"] = counts_of_words_df[\"index\"].apply(lambda x: float(\"NaN\") if x in removed_curly_quotes else x) counts_of_words_df.dropna(inplace=True) counts_of_words_df.head(50)<\/code><\/pre>\nExplanation of code block:<\/p>\n
\n- Created a variable named \u201cremoved_curly_quotes\u201d to involve a curly single, double quotes, and straight double quotes.<\/li>\n
- Used the \u201capply\u201d function in pandas to check all columns with those possible values.<\/li>\n
- Used the lambda function with \u201cfloat(\u201cNaN\u201d) so that we can use \u201cdropna\u201d method of Pandas.<\/li>\n
- Use \u201cdropna\u201d to drop any NaN value that replaces the specific curly quote versions. Add \u201cinplace=True\u201d to drop NaN values permanently.<\/li>\n
- Call the dataframe\u2019s new version and check it.<\/li>\n<\/ul>\n
You can see the result below.<\/p>\n
Screenshot from author, August 2022<\/em><\/span><\/div>\nWe see the most used words in the \u201cSearch Engine Optimization\u201d related ranking web document.<\/p>\n
With Panda\u2019s \u201cplot\u201d methodology, we can visualize it easily as below.<\/p>\n
counts_of_words_df.head(20).plot(kind=\"bar\",x=\"index\", orientation=\"vertical\", figsize=(15,10), xlabel=\"Tokens\", ylabel=\"Count\", colormap=\"viridis\", table=False, grid=True, fontsize=15, rot=35, position=1, title=\"Token Counts from a Website Content with Punctiation\", legend=True).legend([\"Tokens\"], loc=\"lower left\", prop={\"size\":15})<\/code><\/pre>\nExplanation of code block above:<\/p>\n
\n- Use the head method to see the first meaningful values to have a clean visualization.<\/li>\n
- Use \u201cplot\u201d with the \u201ckind\u201d attribute to have a \u201cbar plot.\u201d<\/li>\n
- Put the \u201cx\u201d axis with the columns that involve the words.<\/li>\n
- Use the orientation attribute to specify the direction of the plot.<\/li>\n
- Determine figsize with a tuple that specifies height and width.<\/li>\n
- Put x and y labels for x and y axis names.<\/li>\n
- Determine a colormap that has a construct such as \u201cviridis.\u201d<\/li>\n
- Determine font size, label rotation, label position, the title of plot, legend existence, legend title, location of legend, and size of the legend.<\/li>\n<\/ul>\n
The Pandas DataFrame Plotting is an extensive topic. If you want to use the \u201cPlotly\u201d as Pandas visualization back-end, check the Visualization of Hot Topics for News SEO<\/a>.<\/p>\nYou can see the result below.<\/p>\n
![\"Pandas](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/09\/compare-vocabulary-differences-between-ranking-web-pages-on-serp-with-python-via-sejournal-koraygubur-6.png\")
Image from author, August 2022<\/em><\/span><\/div>\nNow, we can choose our second URL to start our comparison of vocabulary size and occurrence of words.<\/p>\n
8. Choose The Second URL For Comparison Of The Vocabulary Size And Occurrences Of Words<\/h3>\n
To compare the previous SEO content to a competing web document, we will use SEJ\u2019s SEO guide<\/a>. You can see a compressed version of the steps followed until now for the second article.<\/p>\ndef tokenize_visualize(article:int): stop_words = set(stopwords.words(\"english\")) removed_curly_quotes = \"\u2019\u201c\u201d\" tokenized_words = word_tokenize(crawled_df[\"body_text\"][article]) print(\"Count of tokenized words:\", len(tokenized_words)) tokenized_words = [word for word in tokenized_words if not word.lower() in stop_words and word.lower() not in string.punctuation and word.lower() not in removed_curly_quotes] print(\"Count of tokenized words after removal punctations, and stop words:\", len(tokenized_words)) counted_tokenized_words = Counter(tokenized_words) counts_of_words_df = pd.DataFrame.from_dict( counted_tokenized_words, orient=\"index\").reset_index() counts_of_words_df.sort_values(by=0, ascending=False, inplace=True) #counts_of_words_df[\"index\"] = counts_of_words_df[\"index\"].apply(lambda x: float(\"NaN\") if x in removed_curly_quotes else x) counts_of_words_df.dropna(inplace=True) counts_of_words_df.head(20).plot(kind=\"bar\", x=\"index\", orientation=\"vertical\", figsize=(15,10), xlabel=\"Tokens\", ylabel=\"Count\", colormap=\"viridis\", table=False, grid=True, fontsize=15, rot=35, position=1, title=\"Token Counts from a Website Content with Punctiation\", legend=True).legend([\"Tokens\"], loc=\"lower left\", prop={\"size\":15})<\/code><\/pre>\nWe collected everything for tokenization, removal of stop words, punctations, replacing curly quotations, counting words, data frame construction, data frame sorting, and visualization.<\/p>\n
Below, you can see the result.<\/p>\n
![\"tokenize](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/09\/compare-vocabulary-differences-between-ranking-web-pages-on-serp-with-python-via-sejournal-koraygubur-7.png\")
Screenshot by author, August 2022<\/em><\/span><\/div>\nThe SEJ article is in the eighth ranking.<\/p>\n
tokenize_visualize(8)<\/code><\/pre>\nThe number eight means it ranks eighth on the crawl output data frame, equal to the SEJ article for SEO. You can see the result below.<\/p>\n
![\"token](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/09\/compare-vocabulary-differences-between-ranking-web-pages-on-serp-with-python-via-sejournal-koraygubur-8.png\")
Image from author, August 2022<\/em><\/span><\/div>\nWe see that the 20 most used words between the SEJ SEO article and other competing SEO articles differ.<\/p>\n
9. Create A Custom Function To Automate Word Occurrence Counts And Vocabulary Difference Visualization<\/strong><\/h3>\nThe fundamental step to automating any SEO task with Python is wrapping all the steps and necessities under a certain Python function with different possibilities.<\/p>\n
The function that you will see below has a conditional statement. If you pass a single article, it uses a single visualization call; for multiple ones, it creates sub-plots according to the sub-plot count.<\/p>\n
def tokenize_visualize(articles:list, article:int=None): if article: stop_words = set(stopwords.words(\"english\")) removed_curly_quotes = \"\u2019\u201c\u201d\" tokenized_words = word_tokenize(crawled_df[\"body_text\"][article]) print(\"Count of tokenized words:\", len(tokenized_words)) tokenized_words = [word for word in tokenized_words if not word.lower() in stop_words and word.lower() not in string.punctuation and word.lower() not in removed_curly_quotes] print(\"Count of tokenized words after removal punctations, and stop words:\", len(tokenized_words)) counted_tokenized_words = Counter(tokenized_words) counts_of_words_df = pd.DataFrame.from_dict( counted_tokenized_words, orient=\"index\").reset_index() counts_of_words_df.sort_values(by=0, ascending=False, inplace=True) #counts_of_words_df[\"index\"] = counts_of_words_df[\"index\"].apply(lambda x: float(\"NaN\") if x in removed_curly_quotes else x) counts_of_words_df.dropna(inplace=True) counts_of_words_df.head(20).plot(kind=\"bar\", x=\"index\", orientation=\"vertical\", figsize=(15,10), xlabel=\"Tokens\", ylabel=\"Count\", colormap=\"viridis\", table=False, grid=True, fontsize=15, rot=35, position=1, title=\"Token Counts from a Website Content with Punctiation\", legend=True).legend([\"Tokens\"], loc=\"lower left\", prop={\"size\":15}) if articles: source_names = [] for i in range(len(articles)): source_name = crawled_df[\"url\"][articles[i]] print(source_name) source_name = urlparse(source_name) print(source_name) source_name = source_name.netloc print(source_name) source_names.append(source_name) global dfs dfs = [] for i in articles: stop_words = set(stopwords.words(\"english\")) removed_curly_quotes = \"\u2019\u201c\u201d\" tokenized_words = word_tokenize(crawled_df[\"body_text\"][i]) print(\"Count of tokenized words:\", len(tokenized_words)) tokenized_words = [word for word in tokenized_words if not word.lower() in stop_words and word.lower() not in string.punctuation and word.lower() not in removed_curly_quotes] print(\"Count of tokenized words after removal punctations, and stop words:\", len(tokenized_words)) counted_tokenized_words = Counter(tokenized_words) counts_of_words_df = pd.DataFrame.from_dict( counted_tokenized_words, orient=\"index\").reset_index() counts_of_words_df.sort_values(by=0, ascending=False, inplace=True) #counts_of_words_df[\"index\"] = counts_of_words_df[\"index\"].apply(lambda x: float(\"NaN\") if x in removed_curly_quotes else x) counts_of_words_df.dropna(inplace=True) df_individual = counts_of_words_df dfs.append(df_individual) import matplotlib.pyplot as plt figure, axes = plt.subplots(len(articles), 1) for i in range(len(dfs) + 0): dfs[i].head(20).plot(ax = axes[i], kind=\"bar\", x=\"index\", orientation=\"vertical\", figsize=(len(articles) * 10, len(articles) * 10), xlabel=\"Tokens\", ylabel=\"Count\", colormap=\"viridis\", table=False, grid=True, fontsize=15, rot=35, position=1, title= f\"{source_names[i]} Token Counts\", legend=True).legend([\"Tokens\"], loc=\"lower left\", prop={\"size\":15})<\/code><\/pre>\nTo keep the article concise, I won\u2019t add an explanation for those. Still, if you check previous SEJ Python SEO tutorials I have written, you will realize similar wrapper functions.<\/p>\n
Let\u2019s use it.<\/p>\n
tokenize_visualize(articles=[1, 8, 4])<\/p>\n
We wanted to take the first, eighth, and fourth articles and visualize their top 20 words and their occurrences; you can see the result below.<\/p>\n
![\"visualization](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/09\/compare-vocabulary-differences-between-ranking-web-pages-on-serp-with-python-via-sejournal-koraygubur-9.png\")
Image from author, August 2022<\/em><\/span><\/div>\n10. Compare The Unique Word Count Between The Documents<\/h3>\n
Comparing the unique word count between the documents is quite easy, thanks to pandas. You can check the custom function below.<\/p>\n
def compare_unique_word_count(articles:list): source_names = [] for i in range(len(articles)): source_name = crawled_df[\"url\"][articles[i]] source_name = urlparse(source_name) source_name = source_name.netloc source_names.append(source_name) stop_words = set(stopwords.words(\"english\")) removed_curly_quotes = \"\u2019\u201c\u201d\" i = 0 for article in articles: text = crawled_df[\"body_text\"][article] tokenized_text = word_tokenize(text) tokenized_cleaned_text = [word for word in tokenized_text if not word.lower() in stop_words if not word.lower() in string.punctuation if not word.lower() in removed_curly_quotes] tokenized_cleanet_text_counts = Counter(tokenized_cleaned_text) tokenized_cleanet_text_counts_df = pd.DataFrame.from_dict(tokenized_cleanet_text_counts, orient=\"index\").reset_index().rename(columns={\"index\": source_names[i], 0: \"Counts\"}).sort_values(by=\"Counts\", ascending=False) i += 1 print(tokenized_cleanet_text_counts_df, \"Number of unique words: \", tokenized_cleanet_text_counts_df.nunique(), \"Total contextual word count: \", tokenized_cleanet_text_counts_df[\"Counts\"].sum(), \"Total word count: \", len(tokenized_text)) compare_unique_word_count(articles=[1, 8, 4])<\/code><\/pre>\nThe result is below.<\/p>\n
The bottom of the result shows the number of unique values, which shows the number of unique words in the document.<\/p>\n
www.wordstream.com Counts<\/p>\n
16 Google 71<\/p>\n
82 SEO 66<\/p>\n
186 search 43<\/p>\n
228 site 28<\/p>\n
274 page 27<\/p>\n
\u2026 \u2026 \u2026<\/p>\n
510 markup\/structured 1<\/p>\n
1 Recent 1<\/p>\n
514 mistake 1<\/p>\n
515 bottom 1<\/p>\n
1024 LinkedIn 1<\/p>\n
[1025 rows x 2 columns] Number of unique words:<\/p>\n
www.wordstream.com 1025<\/p>\n
Counts 24<\/p>\n
dtype: int64 Total contextual word count: 2399 Total word count: 4918<\/p>\n
www.searchenginejournal.com Counts<\/p>\n
9 SEO 93<\/p>\n
242 search 25<\/p>\n
64 Guide 23<\/p>\n
40 Content 17<\/p>\n
13 Google 17<\/p>\n
.. \u2026 \u2026<\/p>\n
229 Action 1<\/p>\n
228 Moving 1<\/p>\n
227 Agile 1<\/p>\n
226 32 1<\/p>\n
465 news 1<\/p>\n
[466 rows x 2 columns] Number of unique words:<\/p>\n
www.searchenginejournal.com 466<\/p>\n
Counts 16<\/p>\n
dtype: int64 Total contextual word count: 1019 Total word count: 1601<\/p>\n
blog.hubspot.com Counts<\/p>\n
166 SEO 86<\/p>\n
160 search 76<\/p>\n
32 content 46<\/p>\n
368 page 40<\/p>\n
327 links 39<\/p>\n
\u2026 \u2026 \u2026<\/p>\n
695 idea 1<\/p>\n
697 talked 1<\/p>\n
698 earlier 1<\/p>\n
699 Analyzing 1<\/p>\n
1326 Security 1<\/p>\n
[1327 rows x 2 columns] Number of unique words:<\/p>\n
blog.hubspot.com 1327<\/p>\n
Counts 31<\/p>\n
dtype: int64 Total contextual word count: 3418 Total word count: 6728<\/p>\n
There are 1025 unique words out of 2399 non-stopword and non-punctuation contextual words. The total word count is 4918.<\/p>\n
The most used five words are \u201cGoogle,\u201d \u201cSEO,\u201d \u201csearch,\u201d \u201csite,\u201d and \u201cpage\u201d for \u201cWordstream.\u201d You can see the others with the same numbers.<\/p>\n
11. Compare The Vocabulary Differences Between The Documents On The SERP<\/h3>\n
Auditing what distinctive words appear in competing documents helps you see where the document weighs more and how it creates a difference.<\/p>\n
The methodology is simple: \u201cset\u201d object type has a \u201cdifference\u201d method to show the different values between two sets.<\/p>\n
def audit_vocabulary_difference(articles:list): stop_words = set(stopwords.words(\"english\")) removed_curly_quotes = \"\u2019\u201c\u201d\" global dfs global source_names source_names = [] for i in range(len(articles)): source_name = crawled_df[\"url\"][articles[i]] source_name = urlparse(source_name) source_name = source_name.netloc source_names.append(source_name) i = 0 dfs = [] for article in articles: text = crawled_df[\"body_text\"][article] tokenized_text = word_tokenize(text) tokenized_cleaned_text = [word for word in tokenized_text if not word.lower() in stop_words if not word.lower() in string.punctuation if not word.lower() in removed_curly_quotes] tokenized_cleanet_text_counts = Counter(tokenized_cleaned_text) tokenized_cleanet_text_counts_df = pd.DataFrame.from_dict(tokenized_cleanet_text_counts, orient=\"index\").reset_index().rename(columns={\"index\": source_names[i], 0: \"Counts\"}).sort_values(by=\"Counts\", ascending=False) tokenized_cleanet_text_counts_df.dropna(inplace=True) i += 1 df_individual = tokenized_cleanet_text_counts_df dfs.append(df_individual) global vocabulary_difference vocabulary_difference = [] for i in dfs: vocabulary = set(i.iloc[:, 0].to_list()) vocabulary_difference.append(vocabulary) print( \"Words that appear on :\", source_names[0], \"but not on: \", source_names[1], \"are below: \\n\", vocabulary_difference[0].difference(vocabulary_difference[1]))<\/code><\/pre>\nTo keep things concise, I won\u2019t explain the function lines one by one, but basically, we take the unique words in multiple articles and compare them to each other.<\/p>\n
You can see the result below.<\/p>\n
Words that appear on: www.techtarget.com<\/a> but not on: moz.com<\/a> are below:<\/p>\n![\"Compare](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/09\/compare-vocabulary-differences-between-ranking-web-pages-on-serp-with-python-via-sejournal-koraygubur-10.png\")
Screenshot by author, August 2022<\/em><\/span><\/div>\nUse the custom function below to see how often these words are used in the specific document.<\/p>\n
def unique_vocabulry_weight(): audit_vocabulary_difference(articles=[3, 1]) vocabulary_difference_list = vocabulary_difference_df[0].to_list() return dfs[0][dfs[0].iloc[:, 0].isin(vocabulary_difference_list)] unique_vocabulry_weight()<\/code><\/pre>\nThe results are below.<\/p>\n
![\"Compare](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/09\/compare-vocabulary-differences-between-ranking-web-pages-on-serp-with-python-via-sejournal-koraygubur-11.png\")
Screenshot by author, August 2022<\/em><\/span><\/div>\nThe vocabulary difference between TechTarget and Moz for the \u201csearch engine optimization\u201d query from TechTarget\u2019s perspective is above. We can reverse it.<\/p>\n
def unique_vocabulry_weight(): audit_vocabulary_difference(articles=[1, 3]) vocabulary_difference_list = vocabulary_difference_df[0].to_list() return dfs[0][dfs[0].iloc[:, 0].isin(vocabulary_difference_list)] unique_vocabulry_weight()<\/code><\/pre>\nChange the order of numbers. Check from another perspective.<\/p>\n
![\"moz](\"https:\/\/marketingnewsbox.com\/wp-content\/uploads\/2022\/09\/compare-vocabulary-differences-between-ranking-web-pages-on-serp-with-python-via-sejournal-koraygubur-12.png\")
Screenshot by author, August 2022<\/em><\/span><\/div>\nYou can see that Wordstream has 868 unique words that do not appear on Boosmart, and the top five and tail five are given above with their occurrences.<\/p>\n
The vocabulary difference audit can be improved with \u201cweighted frequency\u201d by checking the query information and network.<\/p>\n
But, for teaching purposes, this is already a heavy, detailed, and advanced Python<\/a>, Data Science, and SEO intensive course<\/a>.<\/p>\nSee you in the next guides and tutorials.<\/p>\n
More resources:<\/strong><\/p>\n
\nFeatured Image: VectorMine\/Shutterstock<\/em><\/p>\n