HOW TO AUTOMATE YOUR KEYWORD RESEARCH

Paul Shapiro – Organic Search Director at CatalystSearchLove Boston – May 4th,

2016

@fighto

2

Without Automation

@fighto

3

With Automation

@fighto

WHO AM I?

4

@fighto

5

Tool Data Sources

Visualization

+ +

@fighto

WHAT YOU WILL LEARN

• The problems with the standard keyword research process• How you can save time on keyword research with automation• How to automate keyword research with a powerful, but easy-to-use

tool• How APIs work and how to leverage them• What data sources are useful for keyword research automation• How to use data visualization to make keyword research decisions

6

MANUAL KEYWORD RESEARCH IS

OFTEN BLAND &TIME CONSUMING

7

@fighto

8

@fighto

9

Most Keyword Research Looks Like This

@fighto

10

That’s just a Google

Keyword Planner Export

@fighto

TYPICAL TIME INVESTMENT FOR KEYWORD RESEARCH

Micro (0-49 pages) Small (50-99 pages) Medium (100-249 pages)

Large (250-499 pages) Extra Large (>500 pages)

0

5

10

15

20

25

30

58

12

21 21

6

10

18

22

28

Hours to Complete Keyword Research by Site Size

Average (Low End) Average (High End)

11

@fighto

12

AIN’T NOBODY GOT TIME FOR MANUAL KEYWORD RESEARCH!

@fighto

SIZE OF THE DATA SET

13

vs.

@fighto

14

We can do better!

WE CAN SAVE TIME WITH AUTOMATION

@fighto

TYPICAL TIME INVESTMENT FOR KEYWORD RESEARCH

Micro (0-49 pages) Small (50-99 pages) Medium (100-249 pages)

Large (250-499 pages) Extra Large (>500 pages)

0

5

10

15

20

25

30

58

12

21 21

6

10

18

22

28

Hours to Complete Keyword Research by Site Size

Average (Low End) Average (High End)

16

@fighto

17

If we do keyword research once per month, we will save around 24 to 45 full days over a 5-year period using automation.

http://xkcd.com/1205/

WE CAN DO BETTER THAN AN EXCEL SPREADSHEET

@fighto

THAT’S MORE LIKE IT!

19h/t @victorpan

@fighto

HOW TO USE FOR KEYWORD SELECTION / TARGETING

20

This top-right quadrant contains keywords with:

• Low competition• Good growth

Larger bubbles show higher search volumes.

You can alternatively use current rank on the x-axis to signal organic market share like a traditional growth-share matrix.

@fighto

21

That’s much better! actionability.

MY KEYWORD RESEARCH AUTOMATION PROCESS

@fighto

23

https://www.knime.orgIt starts with a tool…

@fighto

WHY KNIME?

• Singular tool for data collection, manipulation, analysis, and reporting

• Easy to use and no development skills necessary

• If you are a developer, it provides faster prototyping

• Large library of built-in “nodes”• Free/Open Source• Run on Windows/Mac/Linux• Infinitely expandable

24

THE BASICS OF KNIME

@fighto

WHAT’S A NODE?

• Nodes are prebuilt, drag and drop modules designed perform a singular task• Nodes are strung together like a chain to accomplish larger, more complex tasks• Nodes can be grouped together into “meta-nodes”, which can be configured in unison

26

@fighto

HOW DO YOU ADD NODES & HOW DO THEY CONNECT?

How do you add nodes to your “workflow”?

How do you string nodes together?

27

@fighto

HOW DO YOU CONFIGURE & RUN NODES?

Configuring Nodes

Running Workflows

28

OR

APIs 101

29

@fighto

EASY & APPLICABLE API: GOOGLE AUTOSUGGEST

http://suggestqueries.google.com/complete/search?output=toolbar&hl=en&q=ninja%20turtles

30

Simple API using “GET Request”

This is a variable that changes

@fighto

31

Output = XML

Parse me!

A KNIME-API EXAMPLE

32

@fighto

STEP 1: LIST OF KEYWORDS

33

@fighto

STEP 2: ENCODE SPACES IN KEYWORD PHRASES

34

Space Character

@fighto

STEP 3: CONCATENATE API URL + KEYWORDS

35

@fighto

STEP 4: DOWNLOAD API OUTPUT

36

@fighto

STEP 6: LOOP THROUGH EACH KEYWORD

37

@fighto

STEP 7: PARSE THE XML OUTPUT FORM THE API

38

@fighto

39

Auto-CompleteSuggestions

@fighto

40

@fighto

FYI: I’M NOT A KNIME SALESMAN

41

DATA INPUTS / SOURCES

@fighto

43# of primary data sources to aggregate

@fighto

44 http://searchwilderness.com/gwmt-data-python/

Google Search Analytics – Historical MySQL Database via Python Script

@fighto

45

AdWords - Search Query Report API or CSV

@fighto

46

SEMRush – API (if <100,000) or Domain vs. Domain:

Competitor Keywords (Advanced Mode)

https://www.semrush.com/info/domain_vs_domain

@fighto

47

GrepWords API – For Google Keyword Planner related query suggestions &

search volumes

http://grepwords.com/

@fighto

48

Google Autosuggest API or Emulation with Selenium

@fighto

49

Rank Checking/Full SERP API - GetSTAT

https://getstat.com/

@fighto

50

MarketMuse – Semantic Keyword Recommendations

https://www.marketmuse.com/

Alternative: http://pshapi.ro/SemanticKWR

@fighto

51

Moz API + Moz Nodes

https://tech.knime.org/book/mmi-labs-nodeshttps://moz.com/products/mozscape/access

@fighto

52

Google Trends – Unofficial API

https://github.com/GeneralMills/pytrendsScript soon to follow: http://pshapi.ro/slopetrends

@fighto

53

More data sources:http://pshapi.ro/kwmetrics

THE DATA-TO-KNIME PROCESS

54

@fighto

Filtering + Data

Manipula-tions

ONE BIG KEYWORD LIST

Seed Keywords- List- GWMT- SEMRush Comp. KWs- SQR Keywords

Keyword Planner Suggestions (via

GrepWords)Google

Autocomplete

Semantic Keyword Recommendations (via MarketMuse)

Google Autocomplete

SEMRush Domain vs.

Domain KeywordsGoogle

Autocomplete

55

@fighto

STRING ‘EM ALL TOGETHER AND THEN…

56

@fighto

DATA MANIPULATIONS / CALCULATIONS

• Get top 10 results from rank checking API (i.e., GetSTAT)

• Use Moz API nodes and find average PA to assess competiveness.

• Optionally, use SEMRush’s Keyword Difficulty API

Organic Competition

Search Volume

Keyword Trends

57

@fighto

DATA MANIPULATIONS / CALCULATIONS

• Get top 10 results from rank checking API (i.e., GetSTAT)

• Use Moz API nodes and find average PA to assess competiveness.

• Optionally, use SEMRush’s Keyword Difficulty API

Organic Competition

• Get Search Volumes via SEMRush API or via GrepWords APISearch Volume

Keyword Trends

58

@fighto

DATA MANIPULATIONS / CALCULATIONS

• Get top 10 results from rank checking API (i.e., GetSTAT)

• Use Moz API nodes and find average PA to assess competiveness.

• Optionally, use SEMRush’s Keyword Difficulty API

Organic Competition

• Get Search Volumes via SEMRush API or via GrepWords APISearch Volume

• Use 2 Years of Google Trends data to calculate slope and determine growing/declining keywords

Keyword Trends

59

@fighto

60

@fighto

61

Reusable

5-Minutes

@fighto

62

You can use this data to filter keywords and reduce massive list... OR, you can leverage data visualization!

VISUALIZING KEYWORD RESEARCH

@fighto

64

We have a few

options…

@fighto

OPTION: EXCEL + BUBBLE CHART

65

Not Recommended

@fighto

OPTION: TABLEAU

66

@fighto

OPTION: PYTHON + BOKEH CHARTS IN KNIME

67

@fighto

THEY’RE INTERACTIVE…

68

@fighto

69

Sample Python Code:http://pshapi.ro/SLBokeh

@fighto

SO WHAT…

• Free up time spend doing mundane, repetitive work and spend more doing impactful, strategic work• Free up staffing resources and utilize them in other ways• Extract and analyze a larger pool of data for greater data pool

70

From this… To this…

@fighto

SUMMARY

• You should rethink how your currently do keyword research.• Keyword research is often bland, time consuming, and not as actionable as it can be.• KNIME provides an easy-to-use interface for automating your keyword research• APIs are easier to use than they sound; Use them• There are great data sources you can leverage via APIs/built-in KNIME nodes• Visualize your keywords using a growth-share matrix like-graph to make keyword usage

decisions• Emojis are awesome

71

@fighto

RESOURCES: DATA SOURCES > KNIME

72

Data Source KNIME Node(s)Google Search Analytics Database Table Connector/ReaderAdWords Search Query Report XLS ReaderSEMRush Domain vs. Domain (or API if less than 100,000 keyword)

XLS/CSV Reader or GET Resource & Read REST Representation

Google Keyword Planner/GrepWords GET Resource & Read REST Representation

Google Autosuggest GET Resource & Read REST Representation (or Selenium nodes)

Rank Checking (via GetSTAT) GET Resource & Read REST Representation

Semantic Keywords/MarketMuse POST ResourceMOZ Moz API Connector/URL Metrics nodesGoogle Trends TBD – Either CSV or Python Script nodes

@fighto

ADDITIONAL RESOURCES• KNIME Version to install: “KNIME Analytics Platform + all free extensions”• To install KNIME Add-ons, Choose Help->Install New Software

• Moz nodes• Should come with Nightly Builds. If not can add

http://update.knime.org/community-contributions/3.1 as a repository.• Add-on Selenium nodes

• http://seleniumnodes.com/• Search Engine Land article addressing additional data inputs: http://pshapi.ro/kwmetrics• Information about Growth-share matrices: https://

en.wikipedia.org/wiki/Growth%E2%80%93share_matrix• Sanple Python + Bokeh script for an interactive keyword research visualization in KNIME:

http://pshapi.ro/SLBokeh

73

@fighto

Questions?

74

THANK YOU!@fightohttp://

searchwilderness.comhttp://catalystdigital.com