SEO Data Scraping

Web data extraction, also known as scraping, is widely used by marketers to collect prices from competitors’ websites. However, it can also be useful for other digital marketing specialists, including PPC, SEO, and content marketers. To prove this point, we’ll show you five ways to use scraping for comprehensive competitor analysis. We will use Netpeak Spider as a web scraping tool.

Leveraging Scraping for Market Trend Analysis

Understanding market trends is crucial for digital marketers who want to stay ahead of the competition. Scraping data from various sources allows marketers to gain insights into emerging trends, customer preferences, and industry developments. By collecting data from blogs, news sites, and social media platforms, marketers can identify trending topics, popular hashtags, and emerging industry buzzwords. This information can inform content creation, advertising strategies, and product development.

To perform market trend analysis, follow these steps with Netpeak Spider:

Identify Sources: List relevant websites, blogs, and social media platforms where your audience frequently engages.
Set Up Scraping: Configure Netpeak Spider to extract content headlines, publication dates, and engagement metrics.
Data Analysis: Use the extracted data to identify patterns and trends, such as frequently mentioned topics or spikes in engagement around specific issues.
Strategize: Leverage insights to refine your marketing strategies, align content with audience interests, and anticipate future trends.

By systematically analyzing market trends, businesses can make informed decisions, optimize their marketing efforts, and stay competitive. Scraping offers a data-driven approach to understanding the dynamic market landscape, ensuring that your strategies are both proactive and responsive to the ever-changing digital environment.

1. Scraping Prices from Competitors’ Websites

Competitors’ price scraping is one of the main routine tasks digital marketers and SEO specialists face in eCommerce projects. The process consists of two parts: finding an element with the appropriate value (data) and extracting it. You can also select the necessary categories or pages for extraction.
In most cases, you need to perform the following actions to scrape prices from a website:

Open product page
Find the price and hover over it.
Right-click on it and select ‘Inspect element’.
Scroll to a highlighted line and right-click on it.
Select ‘Copy’ → ‘Copy XPath’.

Launch Netpeak Spider.
Open ‘Settings’ → ‘Scraping’.
Turn on the ‘Use HTML scraping’ option.
Choose the ‘Xpath’ type of search and enter the code you’ve copied in the ‘Search expressions’ box. Then choose ‘Data extraction’ mode → ‘Inner text’.

Click the ‘ОК’ button to save settings and close the window.
Put the website URL into the address bar and launch scanning with the ‘Start’ button.
After scanning is finished, go to the sidebar and open the ‘Reports’ → ‘Scraping’ tab.
Click on the line with the number of pages containing the requested data.
Click on the ‘Show selected’ button.
Look up the report in a new window and export scraping data with the ‘Export’ button.

You can also use this method to scrape all information about product lines, special purchase conditions, and any other product specifications described on product pages.

By the way, scraping can be used for competitor analysis and creating Google Adwords product feeds.

2. Most Popular Competitors’ Content Analysis

While creating content in a highly competitive environment, you cannot ignore your competitors’ moves. You can use scraping to find the most viral competitors’ publications. This will give you an overall picture and help you identify common factors of successful content.
If there are any views, shares, likes or repost counters open for public viewing, you can scrape their values the following way:

Open a page with a competitor’s publication.
Find a counter with an indicator you’re interested in.
Hover over its value.
Left-click on it and select ‘Inspect element’.

Copy Xpath.
Set and launch the scraping procedure as shown above.
Export scraping data.

3. Google SERP Scraping

You can automatically get a piece of information about the top of Google SERP using scraping. You can scrape up to 100 snippets with URLs, Titles, and Descriptions. Perform scraping in the following way:

Open the Google search page and type your query.
Go to a search settings page.
Set the right amount of search results per page.
Save settings and go back to the SERP page.
Copy the SERP page URL.
Launch Netpeak Spider.
Select «List of URLs» → «Enter Manually» in the Netpeak Spider main menu.
Paste the copied URL in a new window.

You can simultaneously paste as many URLs as search queries you’re interested in.

Open ‘Settings’ → ‘Scraping’.
Turn on the ‘Use HTML scraping’ option.
Set names for searches, such as SERP Title, SERP Description, and SERP URL.
Choose the ‘Xpath’ search type. Then choose ‘Data extraction’ mode → ‘Inner text’.

Enter in the search box the following pieces of code:

for Title scraping—
```
//*[@id="rso"]//div[1]/div/div/div/h3/a
```

for Description scraping—

//*[@id="rso"]//div[1]/div/div/div/div/div/span

for URL scraping —

//*[@id="rso"]//div[1]/div/div/div/h3//@href

Do not close the current window. Move to the ‘User-Agent’ tab and choose Chrome as a user agent.
Open the ‘Advanced’ tab and turn off all parameters.
Save settings.
Go to the sidebar and open the ‘Parameters’ tab. Turn off all parameters except ‘Scraping’.
Start scanning.
At the end of scanning, you will see a few columns that match searches you’ve set before. A number of values will be found.
To see the results of each search, double-click on the value you’re interested in.
Look at the scraping data in an opened table.
Left-click on a ‘Report’ button for a quick switch between the results of each search.
To download the results table, click the ‘Export’ button and save it as a file.

4. SEO Competitor Analysis

While testing new methods for improving website optimization, you must wonder what SEO strategy your competitors have. What technologies do they use? What methods helped them achieve current results?
Scraping will help you quickly get answers to the following questions:

Do your competitors use some specific structured data?
What elements of structured data are used on competitors’ pages with rich snippets?
Do your competitors use external media content (from YouTube or other audio/video platforms) on their product pages, for example?
What kind of metadata do your competitors use?

You can get all answers to these questions using simultaneous searches for different competitors’ websites in Netpeak Spider.

4.1. Structured Data Scraping

If you want to know if your competitor uses some specific Schema element, you can perform the following actions:

Open ‘Settings’ → ‘Scraping’.
Turn on the ‘Use HTML scraping’ option.
Choose the ‘Contains’ search type. Then choose ‘Data extraction’ mode → ‘All source code’.
If you need to check if there are some specific elements in the page code, enter itemprop=”name” in the search box (you can set any other element of the Schema glossary instead of the name).
If you need to see each page’s entire structured data, choose the ‘XPath’ search type and ‘Data extraction’ mode → ‘Entire HTML element’.
Enter
```
//*[@itemtype]/@itemtype
```
in the search box.
Save settings and start scanning.

4.2. Media Content Search

If you want to find out if there are any pages with content embedded with iFrame (YouTube and Vimeo videos, audio tracks from Soundcloud, Bandcamp, etc.), follow these instructions:

Open ‘Settings’ → ‘Scraping’.
Turn on the ‘Use HTML scraping’ option.
Choose the ‘Contains’ search type. Then choose ‘Data extraction’ mode → ‘All source code’.
Launch scraping.
Export scraping data.

5. Scraping Customer Reviews

If you work in a niche where review platforms have a big influence, it’s important to monitor your company reviews and your competitors. To scrape reviews automatically, you can use Netpeak Spider and perform data extraction with regular expressions. For example, to scrape reviews from G2 Crowd, you need to:

Open ‘Settings’ → ‘Scraping’.
Turn on the ‘Use HTML scraping’ option.
Create searches to extract positive and negative review parts.
Choose the ‘RegExp’ search type. Then choose ‘Data extraction’ mode → ‘All source code.’
Use regular expression
```
(?
```
Save settings, enter a list of G2 Crowd pages with reviews, and start scraping.
Export results just like we’ve explained in paragraph 3.

Summary

The practical use of scraping is not limited to price extraction. We’ve described the scraping procedure with Netpeak Spider to show you that it can be used for solving a lot of different daily marketing tasks:

Scraping prices and product data from competitors’ websites.
Analyzing most popular competitors’ content.
Google SERP scraping.
SEO competitor analysis.
Scraping customer reviews from specialized niche websites.

By the way, all Plerdy blog readers can buy Netpeak Software products, including Netpeak Spider, with a 10% discount. Follow this link or use discount code 26618a85 to activate it.