Mastering Python Selenium Web Scraping: A detailed guide to perform Web Scraping

In the last decade or so, data has become the most valuable resource in the world. The influence of data has become so prominent in our society and industry that experts have now coined the term “data economy” to emphasize it. Undoubtedly, data has also made a huge impact on how the software industry and businesses operate. To stay competitive, most companies/marketers need to gather data and analyze it to come up with trends, predictions, customer behavior segmentation, classification, etc. and take action based on them.

But data, in its raw form, isn’t of much help for most of their use cases. Data needs to be structured, cleansed, refined, arranged and then connected or measured with other data to become a valuable resource. The first and foremost step is to collect this data quickly, reliably, and correctly from different sources. And what can be the best source other than the internet that hosts a giant pool of easily accessible data?

To leverage this huge amount of available data from the internet, we need tools that can go inside it, launch websites, and collect, process and store those data for us in an automated way. Sometimes in a scheduled manner, but always as per our expectations. Though we can collect data manually, the process can be error-prone and time-consuming. Imagine thousands and millions of data.

Surely it will be a cumbersome task for any human to obtain data manually at this scale. Here, automation comes to the rescue. One of the most efficient tools to do such automated task is the combination of Python (a versatile scripting language) and Selenium (a popular browser automation tool). This article will discuss the importance of this combination for web scraping.

Web Scraping and its Applications

Firstly, let’s understand what web scraping is. It is a generic term used for the various methods one can apply to extract data from websites in an automated way. Sometimes the companies, behind those websites, make it easier for people to collect data by providing Web APIs (Application Programming Interfaces). To collect the data, you can call those APIs using a tool or code at your end. 

But, in most cases, you will not have any dedicated APIs to hit and collect the data you need. For that, you have to take the help of the web scraping process from the UI (User Interface) itself, which is far slower compared to the API route, but can be as effective as the latter if you know how to optimize and effectively do it. 

The data received from the scraping process in UI is mostly in an unstructured format. With the help of various tools, we can make it structured and then store them in data repositories like databases or spreadsheets for further analysis. The analysis will depend on what we try to achieve with the gathered data:

• To keep it as general information purpose (as records)


• To use it for complex tasks: feeding machine learning models for prediction or feeding recommendation engines to recommend products to its end customers.

Web scraping methods also act as enablers to facilitate other important activities like researching market trends, monitoring e-commerce applications/prices/consumer sentiments, gathering social media feeds, extracting new research in particular fields, finding investment opportunities, generating sales leads and so on.

What is Web Scraping with Selenium and why is it used?

Selenium is a popular open-source umbrella project consisting of several tools, servers and libraries that help to automate different browsers. The main remote control interface- “Selenium WebDriver” –  enables the control of user agents and making connections with browsers to mimic the different actions humans can perform on them. The tool is quite popular in the web application testing world. But it can go beyond the testing realm and help people scrape websites. 

Selenium has bindings for different programming languages (like Python, Java, C#, JavaScript etc.). Since it supports multiple languages, people can easily use it to scrape websites by writing code of their choice. Also, it has a great vibrant community surrounding it; if any problem occurs, the user can easily get an efficient solution. 

Before we move on to understand how to perform scraping using Selenium, know that scraping can be of two types:

Traditional Data Scraping

In traditional data scraping, the user does not necessarily need to connect to the internet or open websites to collect data. The data can be collected from different sources such as

• databases (relational or non-relational) e.g. MongoDB, SQL etc.,


• spreadsheets (e.g. Excel, CSV)


• various reports/charts (e.g. HTML reports, Email reports, bar charts, pie charts etc.)

This data is aggregated to form a common source of informational records. 

Browser-based Scraping

In browser-based scraping, the user needs to open/launch a website inside a browser (e.g. Chrome, Safari, Firefox etc.), utilize an internet connection, and then either copy data manually or use an automated tool like Selenium to scrape data from it. 

In this article, we will be talking about browser-based scraping.

Is it legal to scrape a website?

Before you start scraping websites for your needs, it would be crucial to know whether it is legal to scrape any website. In general, scraping websites is not an illegal activity, but its legality also depends on various other factors:-

• What are you going to use the data for?
• How are you going to use the data?
• Has any “Terms and Conditions” been violated?
• Is there any rule imposed by the hosting website on data scraping?
• For how long are you going to hold the data?
• Is the data publicly available on the internet?

Suggestions would be to have reasonable answers to all the questions mentioned above and to stay away from scraping any data that holds personal information or is intellectual property.

Role of Python Selenium web scraping in detail

As mentioned earlier, Selenium WebDriver is a browser automation tool having bindings for programming languages like Python. So, Selenium can be used as a library in Python, along with many other open-sourced Python libraries that make extracting data from websites and arranging and storing them easier. 

Let’s see an example of how we can scrape data using Selenium and Python. To begin with, first, you need to set up a python environment using any IDE tool that supports Python e.g. PyCharm, Spyder, VS Code, etc.

For our example, before writing your first code, you need to install the required python packages using the below commands in the Python environment:

pip install selenium

pip install pandas

pip install openpyxl

You also need to download chromedriver from the below link (according to the chrome browser version you are using in your system):

https://chromedriver.chromium.org/downloads

Then you need to create a python project in the IDE, create a “driver” folder inside it and place the downloaded chromedriver file within the “driver” folder.

Then, you can create a python file inside the project and give any name to the created module. I have given it the name “scraper_main.py”. Inside it, I have created a class name “Scraper” (with a function called “main”) and wrote the below code:

Code

Code Explanation and Output

Let’s go through this code and see what each of its lines does:

First, I have imported the required packages and modules to drive the scraping process:

Then, inside the main method, I am launching the Testsigma blog website taking the help of Selenium WebDriver. There I am also checking whether the URL has successfully launched or not:

After that, I am listing down the total number of blogs getting displayed on that page:

Then, I am going through each of the displayed blogs and extract the blog title, blog author and blog published date information from them (as a python dictionary and using Xpath) and put them inside a python list.

After that, I am converting that list into a pandas dataframe before inserting the captured data into a newly created excel file named “Blog_Info.xlsx”

The excel file will look like below:

Summary

In this article, we have seen step-by-step how to scrape a website using Python Selenium web scraping method. We have understood the different types of scraping processes and circumstances when you might need to check whether scraping is legal. We have also discussed how data is becoming more valuable to the world because of its applications in numerous fields and use cases. Hope this article has provided you with all the information you were looking for related to Web Scraping with Selenium and Python.

FAQs

Is Selenium good for web scraping?

Selenium is indeed a very good tool for performing web scraping. To take out the best from Selenium, you have to know a supported programming language (like Python) and utilize various useful techniques like synchronization, multi-threading, etc. Besides, you have a helpful community you can reach out to regarding any queries or suggestions.

How to use Selenium to scrape a website?

To use Selenium to scrape a website, you first need to know a supported programming language (e.g. Python). You need to then install the required packages and write code (using the IDE of your choice) to launch the website and scrape data from it.

Which Python IDE is best for web scraping?

Any known Python IDEs can be used for web scraping and the speed of execution will not depend on the IDE being used. This includes PyCharm, Spyder, VS Code, Jupyter notebook etc. However, nowadays, most people’s preferred choice is PyCharm IDE or VS Code, since code development using these two IDEs is much faster and more convenient.

Is Scrapy better than Selenium?

There is no right or wrong answer to whether Scrapy is better than Python regarding web scraping. For some use cases where you would like to scrape data very quickly, Scrapy may perform better, but in cases wherein you have to deal with variations in Javascript in UI and AJAX calls, Selenium would do the tasks a lot better.