2023 in SEO with Data Science

• Well it's 2023
• SEO in 2023: A Data Science Approach
• The question
• Getting the data
• Using Horseman
• Processing the data
• Answering Other Questions
• Content Analysis
• LDA Analysis
• What are the most common words in the listicles?
• I trained GPT-2 on the listicles and all I got was this lousy text
• What are the biggest SEO trends of 2023?

Well it's 2023 #

via patrick stox

I've been using Horseman, the tool developed by Chris Johnson for a couple of long term projects right now. Because I've been using it so much, I ended up kind of going down a rabbit hole. And that rabbit hole is the question of "what are the clickbait SEO trends of 2023?"

My mission was simple: get data, process it, and visualize the results. I'm going to show you how I did it, and what I found.

Also it's perfectly reasonable and smart for me to put this out in April, months after the year has started, during BrightonSEO, aka the week that nobody reads articles. I'm a smart guy.

SEO in 2023: A Data Science Approach #

The question #

What is SEO in 2023? Are people just regurgitating the same old SEO trends, or are they actually doing something new? I wanted to find out.

Getting the data #

The first step was of course: how get data? Well, this is for SEO, so obviously any SEO or SEO agency worth their salt would show up for a series of queries like "seo in 2023", "top * in SEO in 2023", and so on. After... analyzing... the Google results, I had a series of web pages with titles like "Is SEO Still relevant in 2023?", "SEO in 2023", and "15 SEO Trends for 2023 That You Need to Know." I ended up with around 1200 webpages, which I then used Horseman to scrape the data from.

Using Horseman #

Horseman is a tool that allows you to crawl websites. It's a bit like Scrapy, but it's a lot easier to use, and a lot more customizable. I used several Horseman features to make my life a lot easier.

• List Crawls
  

Starting with the most basic thing, I dragged my list of URLs over to the crawler and let it do its thing. This is the most basic way to use Horseman, and it's great for getting started. But before I started my crawl off, I enabled a couple of other features:

• Custom Snippets

Horseman allows you to create custom JS snippets-- this lets you do things like scrape data from a page using CSS selectors, or even use Regex. I created a custom snippet that would find any list elements on the page and surface them in an "explore" node, letting me see the number of results as a cell, and get the list items as a list.

/* horseman-config enable-explore */
var list_items = []
var list_numbers = []

// Get all the <ol> elements
var ordered_lists = document.querySelectorAll('ol')

// Loop through the <ol> elements
for (var i = 0; i < ordered_lists.length; i++) {
    var list_items_temp = ordered_lists[i].querySelectorAll('li')
    for (var j = 0; j < list_items_temp.length; j++) {
        list_items.push(list_items_temp[j].textContent)
        list_numbers.push(j + 1)
    }
}

const zip = (a, b) => a.map((k, i) => [k, b[i]]);
const extended = zip(list_numbers, list_items)
extended.unshift(['number', 'item'])

return {
    cell: list_items.length,
    extended: extended
}

This resulted in me being able to analyze my results in a dataframe, in much finer detail. Another useful tool for this came from:

• Pre-built Snippets

The content extractor tool in Horseman is legitimately so useful for any sort of NLP work. It uses Mozilla's Readability library to extract the main content from a page, which avoids a lot of the work and messiness of trying to do it yourself. I used this to extract the main content from each page, which let me go HAM on the NLP.

Additionally, Horseman has added an update allowing you to use GPT to generate snippets and analyze results. Basically automating a ton of the work I had to do on this project. I'm going to be using this a lot more in the future!

Processing the data #

I ended up with a dataframe that looked like this:

And a smaller dataframe of the lists I found, which looked like this:

Hooray! This is a ton of data, and I'm going to use it to answer my question: what are the clickbait SEO trends of 2023?

Answering Other Questions #

First off, the question all SEOs want the answer to: who is ranking the best? I used python to group together the sites by domain, and then sort them by the number of times they appeared in the list. This is what I got:

Themes in articles by Ranking #

I'm a huge dummy who totally forgot to get ranking data for these articles. :)

This is a pretty good list of sites that are ranking well for these queries. But what about the actual content? I used the content extractor to get the main content of each page. And then I performed ** Content Analysis**.

Content Analysis #

The first thing I did was run a TF:IDF on Tokenized versions of the content. This is a pretty standard NLP technique, and it's a good way to get a sense of what the content is about. I used the sklearn library to do this. Here's what I got:

Most common words:

Extremely expected results! Obviously SEO is showing up the most, but it's interesting to see that "content" is also showing up a lot. This is probably because a lot of the sites are talking about "making good content", which I've heard is pretty important for SEO.

From the top terms according to TF:IDF, I then went through the content and found the URLs that had the highest TF:IDF score for those terms. Here are the top 5:

I also made a pretty chart:

I then created a correlation matrix to see which terms were most correlated with each other. This is a good way to get a sense of what the content is about. Here's what I got:

Most Correlated Terms:

And some of the least....

Below in the iframe, you can explore some of this correlation data:

Out of the top 100 terms, for example, "rank" was most highly correlated with the term "high" and least correlated with the term "SEO."

Meanwhile, "research" was the least correlated with the term "2023". I'll let you know if that seems portentous.

I also ran a sentiment analysis on the content, using the TextBlob library. This is a pretty simple library, but it's a good way to get a sense of the overall sentiment of the content. Here's what I got:

Pages with the most positive sentiment:

And the most negative:

I think one of the things that's interesting about this kind of analysis is how you can see how flawed it can be while still getting an overview of the results.

Clustering by Number in the listicle: #

SEOs love listicles: they're the backbone of the web. Buzzfeed etc. So I wanted to cluster and look at, in these listicles, which terms were most common in the first place, second place, etc.

Using Horseman, I extracted different "places" in the listicle versions of the articles, 1-10 and 11+. I then used the KMeans clustering algorithm to cluster the articles using these numbers: basically, clustering everything in a listicle that was in first place, second place, etc. together.

I first figured it'd be prudent to ignore stopwords and some of the most common words, see below:

stop_words = set(stopwords.words('english'))
ignore_list = ['seo', 'search', 'optimization']

And these are my results!

I reran this again, this time ignoring "content", "quality", "user", "site", "voice", and "mobile" as well, to see if I could get some more variety in the results:

I used this colab to do some visualization of the clustering.

So here's another embedded iframe to play with:

LDA Analysis #

If you read my NLP Article for OnCrawl you probably read a little about LDA analysis.

LDA began as a way to analyze genetics: it's a way to look at a bunch of different genes and figure out what they have in common. Similarly, it's used in ML and text analysis to look at a bunch of different documents and figure out what they have in common.

You can explore the LDA analysis for this topic map here:

Named Entity Recognition #

I used Spacy to do some named entity recognition on the content. This is a way to extract entities from text, like people, places, and organizations.

import spacy
nlp = spacy.load('en_core_web_sm')

# Define a function to extract entities from text
def extract_entities(text):
    doc = nlp(text)
    entities = [ent.text for ent in doc.ents if ent.label_ in ['ORG', 'PERSON']]
    return entities

# Apply the function to each column of the dataframe
for column in df1.columns:
    entities = []
    for text in df1[column].dropna():
        entities += extract_entities(text)
    entities = list(set(entities)) # Remove duplicates
    print(f"Entities in {column}: {', '.join(entities)}")

And here are the results:

You can explore the most common entities by type of entity in the below iframe: obviously, Google and AI are the most common entities throughout these lists.

Hot Topic Mentions by Place in the listicle: #

Hot takes are for now: Google, AI, and Mobile-First Indexing are forever. One thing I thought would be fun to look at is how the hot topics are mentioned in the listicles. I used the same method as above to count the number of times each hot topic was mentioned in each listicle. I then plotted the results in a heatmap to see how often certain topics show up at certain points.

import pandas as pd

# Define the terms of interest
terms = ['AI', 'artificial intelligence', 'ML', 'Mobile', 'Content', 'Links', 'Visual', 'Voice']

# Iterate over the columns of df1
results = []
for col in df1.columns[1:]:
    col_docs = df1[col].astype(str).tolist()  # Extract the documents in the column
    col_total = len(col_docs)  # Calculate the total number of documents in the column
    term_counts = [sum([term.lower() in doc.lower() for doc in col_docs]) for term in terms]  # Count the number of documents that mention each term
    term_percents = [count / col_total * 100 for count in term_counts]  # Calculate the percentage of documents that mention each term
    for term, percent in zip(terms, term_percents):
        results.append({'Column': col, 'Term': term, 'Percentage': percent})  # Store the result in a dictionary

# Display the results in a table
results_df = pd.DataFrame(results)
print(results_df)

AI, Content, and Voice remain pretty strong: mentions of Links are way down. It's a bit cheap of me to put ML, AI, and Artificial Intelligence as three different things, but I was really curious to see what way people were talking about AI. It looks like they're talking about it mostly as a buzzword IMO.

What are the most common words in the listicles? #

Most common words in position 1 of the listicles:
content 62
Voice 38
Mobile 22

Position 2:
content
Voice
keywords

Position 3:
content
Voice
Mobile

So clearly, SEOs this year think that content, Voice, and Mobile are the most important things to focus on. Time for my opinion: if you don't already have a mobile friendly site, don't worry about it. you know? It's 2023.

Wordclouds by Number in the listicle: #

Wordclouds. Does anybody love them? Clients do, sometimes.
Most common words in position 1 of the listicles:

Most common words in position 2 of the listicles:

Most common words in position 3 of the listicles:

Most common words in position 4 of the listicles:

And these are the most common words in positions 11+:

I trained GPT-2 on the listicles and all I got was this lousy text #

sess = gpt2.start_tf_sess()

gpt2.finetune(sess,
              dataset=file_name,
              model_name='124M',
              steps=2000,
              restore_from='fresh',
              run_name='run2',
              print_every=10,
              sample_every=200,
              save_every=500
              )

I slammed all the content into GPT-2 and let it train for a while. Here's what it came up with:

In search engines, SERPs increase with the user’s willingness to engage with different content sources, from articles and blogs to blog posts and blogs. As these types of content are being searched, it’s a good idea to optimize your content for that type of search.

Search engines also rank pages based on factors such as keywords, ranking difficulty, and quality. As this type of content helps users find the right content, it should be optimized and optimized for organic search.

To make sure that your website ranks well in search engines, remember that it doesn’t necessarily mean that every keyword you have is the same, but rather that it’s a reflection of the content people are searching for.

What do you need to know about the different types of search results?

- Types of search results on Google
Google’s search engine is constantly changing. Its algorithms have been tweaked and improved recently. In order to keep up with the changes, Google continually updates its algorithms.

In order to avoid getting overwhelmed with new trends, it’s important to keep up with the latest SEO trends, using the latest tools and techniques to ensure that your website is SEO friendly.

Here are a few quick tips for taking your SEO on your own:
- Identify key keywords that attract people to your website
- Generate content with the proper keywords
- Use well researched language and content
- Provide a comprehensive description of yourself
- Provide a comprehensive explanation
- Provide a thorough tutorial

Take the next step and develop a strategy for improving your website’s performance.

1. What is the most vital SEO trend of the year?
The most essential SEO trends of the year are:
Mobile-first indexing
- Paid prioritization
- Mobile-first indexing is a way of ranking websites that helps people to get to a website without having to wait for the loading time.
In order to achieve this, mobile-first indexing will be made a priority.
That’s the reason behind the fact that Google’s AMP system is already working. As smartphones become increasingly smart as we age, Google is starting to get a little more comfortable with mobile-first indexing which is not only a factor in SEO (which is a huge shift), it also aims to make mobile-friendly websites easier to find.
If you use your mobile to search, then you’ll be able to rank on the first page of results. And if the site is already optimized for mobile, then you’ll notice that your ranking will become even more important. Because mobile will make up 99.99% of the number of searches on Google.
Additionally, it means that you’ll see more people going to your website when you rank highly. This means that Google’s mobile-responsive website will become a priority.
2. What is the most important SEO trend of the year?
SEO strategies to boost your website's ranking in search engines are not limited to keywords alone. There’s also the SEO marketing part too.
SEO strategies, like keyword optimisation, are one of the most important to keep up with.
What’s more, SEOs have an even more powerful SEO tool that you can use to boost your overall score in terms of clicks and conversions.
This is because the SERP is more important than ever in 2023, so focus on it when it’s important.
3. What is the biggest SEO trend of the year?
SEO is the most important game changer in 2023, and your SEO is still going strong. In fact, you already got a lot of SEO marketing to focus on this year, you can expect to see even more in the coming years.
Here’s what to make of the biggest SEO trends:
Mobile-first indexing
Paid prioritization
Mobile-first indexing is a great way to rank your website without waiting for the loading time to become an official ranking factor.
In fact, people prefer mobile devices to click on the top results, so you don’t have to focus on the most important one.
Moreover, mobile-first indexing is already the most important thing that people should look out for for any time. Therefore, it’s time to set up an effective practice to implement it.
Mobile-first indexing will help Google deliver high-quality traffic to its users through SERPs. If it’s going to be important to your website, you have to take full advantage of it to get it.
4. What is the biggest SEO trend of the year?
SEOs are one of the most important things on page one of search engines.
Why is this the biggest SEO trend?
SEO is about gaining clicks, leads (acquisition) and conversions.
As per the research, on page one, the most important result is to have a high conversion rate and lead generation

What are the biggest SEO trends of 2023? #

Well, Google seems to be a pretty big trend. It definitely shows up a lot in my text analysis if I don't specifically ignore it. So I think that's a pretty safe bet.

Based on the text analysis, the SEO trends SEOs were talking about are AI, keywords, and EAT, with some voice/vision sprinkled in for flavor. Classic SEO stuff.

What can this tell you about your SEO strategies?

Well, not much-- but it does show you some ways you can use NLP to analyze your content and competitors' content.

This article is a stopgap until I get some of my more interesting/in depth articles up on this site. It was fun to do, and I'll probably dump the code + data on github soon. Let me know if you have any questions about this supremely silly way to understand the SEO trends of 2023.

← Home