How to Spy on Competitors with Python & Data Studio

By olive 2 years ago

Whether you’re new to SEO or a seasoned pro, investigating random drops in keyword rankings is just part of the trade.

There are many methods to investigating why your targeted keywords may have dropped, but using Python and Data Studio together is a game changer!

A quick disclaimer: I did not create this script myself. A fellow SEO and developer, Evan from Architek, collaborated with me to solve a particular need I had. Evan was the brains behind creating the script.

I always wanted to be able to view a large set of Google results at scale, without having to conduct these searches manually one-by-one.

Evan mentioned that Python may be the ideal solution for solving my SEO investigation needs.

Why Python Is Relevant to SEO

Python is an incredibly powerful programing language that can do just about anything. One of the more common uses for Python is automating daily monotonous tasks.

One of the coolest things about Python is that there are several different ways to accomplish the same task. However, this also adds a new level of difficulty.

Most of the sample Python scripts out there can be a bit outdated, so you’ll find yourself doing a lot of trial and error.

Python has many applications for analyzing SEO data. The key is to have the right idea. If you have an idea to automate a task, chances are that there’s a script that can be created for it.

To stay up to date with the latest use cases for Python in SEO, Hamlet Batista has published some awesome articles here at Search Engine Journal.

What This Specific Script Does & Doesn’t Do

Most keyword ranking tools report an average ranking for a keyword over a specified timeframe. This Python script runs a single crawl at the time you run it from your IP address. This script is not meant to track keyword rankings.

The purpose of this script was to solve an issue I was having with investigating sudden drops of rankings across my client and their competition.

Most keyword ranking tools will tell you which pages your domain is ranking for, but not your competitor’s highest ranking pages per keyword.

So Why’s That Important?

In this scenario, we aren’t tacking page performance for the long-term. We’re simply trying to get quick data.

This script allows us to quickly identify trends across the organic landscape and see which pages are performing best.

What You’ll Need to Get Started

If you’re new to Python, I recommend checking out the official Python tutorial or Automate the Boring Stuff.

For this tutorial I’m using PyCharm CE, but you can use Sublime Text or whatever your preferred development environment is.

This script is written in Python 3 and may be a bit advanced for folks new to this programming language.

If you have not found an interpreter or set up your first virtual environment yet, this guide can help you get started.

Once you’re set up with a new virtual environment, you’ll need the following libraries:

  • urllib
  • lxml
  • requests

Now that you’re all set up, let’s dive into some research together.

1. Make a List of Keywords to Investigate

We’re going to be using some sample data to investigate some keywords we’re pretending to track.

Let’s pretend that you looked at your keyword tracking software and noticed that the following keywords dropped more than five positions:

  • SEO Tips
  • Local SEO Advice
  • Learn SEO
  • Search Engine Optimization Articles
  • SEO Blog
  • SEO Basics

Disclaimer: *Searching for too many keywords may result in your IP getting temporarily banned. Pinging Google for this many results at once may appear spammy and will drain their resources. Use with caution and moderation.

The first thing we will do is place these keywords in a simple text file. The keywords should be separated with a line break, as shown in the screenshot below.