Python Selenium Web Scraping: How-To Guide With Examples

How to set up logging using Log4j in Selenium?

<div id="toc_container" class="no_bullets">Table Of Contents<ul class="toc_list"><li><a href="#When_do_we_want_to_run_tests_on_existing_browser_instances">1 When do we want to run tests on existing browser instances?</a><ul><li><a href="#Prerequisites">1.1 Prerequisites</a></li><li><a href="#How_to_launch_Chrome_with_remote_debugging_enabled">1.2 How to launch Chrome with remote debugging enabled?</a></li><li><a href="#How_to_Connect_to_an_Existing_Chrome_Browser_from_Selenium">1.3 How to Connect to an Existing Chrome Browser from Selenium?</a></li></ul></li></ul></div>
<h2 id="When-do-we-want-to-run-tests-on-existing-browser-instances?">When do we want to run tests on existing browser instances?</h2>



There are many scenarios where we want Selenium to connect to and use an existing browser that was previously opened manually or by some other program.



Some of the typical use cases are as below.



<ul><li>When the user does not want to log in every time, they add a test step while drafting the <a href="https://testsigma.com/docs/projects/settings/test-case-types/" target="_blank" rel="noreferrer noopener">test case </a>or when they want a plugin to be added in the automation-controlled browser, the Chrome browser launched using the Selenium web driver will not have their log-in information or plugins that they might have installed.</li><li>When the user has logged into an account that requires two-factor authentication.</li><li>When the user has done a sequence of manual operations that might be hard to automate have already been done.</li><li>When the application has a bug, and the user does a manual intervention to bypass a few steps.</li></ul>



Example



The sample sequence of test steps to add a beneficiary and make a transaction are



<ol><li>Go to the net banking website login page.</li><li>Enter username and password.</li><li>Go to the money transfer page.</li><li>Add the beneficiary details.</li><li>Verify the details of the bank account and person.</li><li>Select from the various methods of online money transfer including RTGS, IMPS, NEFT, UPI, and digital wallet.</li><li>Select the beneficiary name, and the amount, and then submit the details.</li><li>Verify the amount and other information on the verification page.</li><li>Click submit for the final action of the fund transfer.</li></ol>



If you want to add an assertion case where you want to verify the text message displayed after a transaction, you will be looking for an option to run ONLY the test where you assert the success/failure message displayed on the page without repeating the entire test steps from logging into the website.



The default behavior in Selenium is that a new driver instance will be initiated every time when you write a new test. In the above example, if you add the 10th test step to an existing test or write a new test in Selenium, you will be able to verify it only by running from step 1.



You can run only the assertion use case from the above example in a previously opened chrome browser where the transaction case is tested either manually or using selenium script when the browser is opened in debugging mode.



When the Chrome browser application is opened in a remote debugger port, we could easily connect a new Selenium Chrome Driver to that existing chrome instance.



<h3 id="Prerequisites">Prerequisites</h3>



<ol><li>Chrome Browser — Version 63 or higher</li><li>Add Chrome Browser installation directory to Path variable (if you are using windows) — preferable as it would allow opening chrome browser from any directory using terminal/command prompt.</li></ol>



<h3 id="How-to-launch-Chrome-with-remote-debugging-enabled?">How to launch Chrome with remote debugging enabled?</h3>



ChromeDriver will communicate with Chrome via the <a href="https://chromedevtools.github.io/devtools-protocol/" target="_blank" rel="noreferrer noopener">remote DevTools Protocol</a>, which is not usually enabled when you open Chrome from the list of applications in the system.



So, you will need to enable the protocol by starting Chrome with the &#8211;remote-debugging-port command-line switch and choosing a port for it. Port <code>9222</code> is used by convention, but you can use any non-reserved port.



On Windows, you can run from the command prompt by appending a custom flag:



<styled-code>
start chrome.exe &#8211;remote-debugging-port=9222
</styled-code>



or



<styled-code>
<path to="" chrome="">/chrome.exe &#8211;remote-debugging-port=9222
</path></styled-code>



when the path is not set.



On macOS, you will need to find the path to the Chrome executable. Here&#8217;s an example to launch a Chrome executable that is in the default path:



<styled-code>
/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome &#8211;remote-debugging-port=9222
</styled-code>



On Linux, launch the browser by running <code>google-chrome --remote-debugging-port=9222</code>



To launch Chrome with the remote DevTools protocol enabled without the need to use the command line:



<ul><li>On Windows,<ul> <li>Right-click the Chrome shortcut, and select <code>Properties</code></li><li>In the &#8220;target&#8221; field, append <code>--remote-debugging-port=9222</code> </li></ul></li><li>On macOS, either of the 3 ways mentioned below works<ul> <li>Create an <a href="https://support.apple.com/guide/automator/welcome/mac" target="_blank" rel="noreferrer noopener">Automator</a> workflow that runs a bash script to open the browser with the command line switch enabled.</li><li>Use the <code>.dmg</code> file that you can obtain from <a href="https://github.com/natkuhn/Chrome-debug/" target="_blank" rel="noreferrer noopener">this GitHub repository</a>.</li><li>You could create your executable with a tool like <a href="https://sveinbjorn.org/platypus" target="_blank" rel="noreferrer noopener">Platypus</a>.</li></ul></li></ul>



If you want to launch the browser using a custom chrome profile, you can do it by adding <code>--user-data-dir</code> flag when you launch the browser.



To enable Chrome to open a port for remote debugging in the new chrome profile, we need to launch it with a custom flag.



On Windows,



<styled-code>
start chrome.exe &#8211;remote-debugging-port=9222 &#8211;user-data-dir=&#8221;C:\selenum\ChromeProfile&#8221;
</styled-code>



On Mac,



<styled-code>
/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome &#8211;remote-debugging-port=9222 &#8211;user-data-dir=&#8221;~/ChromeProfile&#8221;
</styled-code>



For <code>--user-data-dir</code> flag you can either pass the directory where a new folder will be created for storing details of the chrome profile or you can create a folder and pass it as a value for the flag. You can launch the browser in a new profile when you don’t want to pollute your default profile.



<h3 id="How-to-Connect-to-an-Existing-Chrome-Browser-from-Selenium?">How to Connect to an Existing Chrome Browser from Selenium?</h3>



Here is the sample snippet to initialize Chrome Driver in Selenium with Java/Python<code>1//Sample Java </code>



<styled-code>
//Sample Java Code
ChromeOptions options = new ChromeOptions();
options.setExperimentalOption(&#8220;debuggerAddress&#8221;,&#8221;localhost:<remote-port>&#8220;);
ChromeDriver driver = new ChromeDriver(options);
</remote-port></styled-code>



<styled-code>
#Sample Python Code
chrome_options = Options()
chrome_options.add_experimental_option(&#8220;debuggerAddress&#8221;, &#8220;127.0.0.1:9222&#8221;)
#Change chrome driver path accordingly
chrome_driver = &#8220;C:\chromedriver.exe&#8221;
driver = webdriver.Chrome(chrome_driver, chrome_options=chrome_options)
print driver.title
</styled-code>



*Replace remote-port with the port where you have launched chrome with the remote DevTools protocol enabled



A working example in a sample application is below &#8212;



<styled-code>
package com.sample.web;

import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;

import org.openqa.selenium.By;
import org.openqa.selenium.Capabilities;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;

import org.openqa.selenium.remote.*;
import org.testng.Assert;

import java.util.List;
import java.util.Map;
import java.util.concurrent.TimeUnit;

public class DebuggingMode {
 private WebDriver driver;

 @BeforeClass
 public void setUp() {
 System.setProperty(&#8220;webdriver.chrome.driver&#8221;, &#8220;resources/chromedriver&#8221;);
 ChromeOptions options = new ChromeOptions();
 options.setExperimentalOption(&#8220;debuggerAddress&#8221;, &#8220;127.0.0.1:9222&#8221;);

 driver = new ChromeDriver(options);
 driver.manage().window().maximize();
 driver.manage().timeouts().pageLoadTimeout(120, TimeUnit.SECONDS);
 driver.manage().timeouts().implicitlyWait(20, TimeUnit.SECONDS);

 Capabilities capability = ((RemoteWebDriver) driver).getCapabilities();
 Map<string, object=""> map = capability.asMap();
 for (Map.Entry<string, object=""> entry : map.entrySet()) {
 System.out.println(entry.getKey() + &#8220;:&#8221; + entry.getValue());
 }
 }

 @Test()
 public void testLoginWithWrongCredentials() {
 driver.get(&#8220;https://travel.testsigma.com/login&#8221;);
 driver.findElement(By.xpath(&#8220;//input[@name=&#8217;name&#8217;]&#8221;)).sendKeys(&#8220;admin&#8221;);
 driver.findElement(By.xpath(&#8220;//input[@name=&#8217;password&#8217;]&#8221;)).sendKeys(&#8220;abcde&#8221;);
 driver.findElement(By.xpath(&#8220;//button[@type=&#8217;submit&#8217;]&#8221;)).click();
 Assert.assertTrue(driver.findElement(By.xpath(&#8220;div[text()=&#8217;Wrong username/password&#8217;]&#8221;)).isDisplayed());

 }

 //Below test does not launch the new browser session, and you are not entering the username again
 @Test()
 public void testLoginWithCorrectPassword() {
 driver.findElement(By.xpath(&#8220;//input[@name=&#8217;password&#8217;]&#8221;)).clear();
 driver.findElement(By.xpath(&#8220;//input[@name=&#8217;password&#8217;]&#8221;)).sendKeys(&#8220;12345&#8221;);
 driver.findElement(By.xpath(&#8220;//button[@type=&#8217;submit&#8217;]&#8221;)).click();
 List<webelement> message = driver.findElements(By.xpath(&#8220;div[text()=&#8217;Wrong username/password&#8217;]&#8221;));
 Assert.assertTrue(message.isEmpty());
 }
}
</webelement></string,></string,></styled-code>

Executing Tests on existing Chrome Browser Instance

Executing Tests on Existing Chrome Browser Instances

When you encounter 404/Page Not Found/Dead Hyperlinks on a website, what do you think of it? You would find it irritating to encounter broken links, so you should continuously concentrate on eliminating their presence in your web product (or website).



The rankings of your web product on search engines (like Google) will also suffer if it contains numerous pages (or links) that lead to a 404 error (or page not found). Dead links could damage the credibility of your product as the user needed to get the information for which they have visited the page.&nbsp;



In this blog, we will look into finding and locating broken links using Selenium WebDriver; 



Check out this detailed tutorial on how to use <a href="https://testsigma.com/blog/selenium-based-automated-testing/#Selenium_WebDriver" target="_blank" rel="noreferrer noopener">Selenium WebDriver</a> to know more about what and how to use Selenium WebDriver.



<div id="toc_container" class="no_bullets">Table Of Contents<ul class="toc_list"><li><a href="#What_are_Broken_links">1 What are Broken links?</a></li><li><a href="#Common_Reasons_for_Broken_Links">2 Common Reasons for Broken Links.</a></li><li><a href="#Reasons_to_Check_for_Broken_Links">3 Reasons to Check for Broken Links</a></li><li><a href="#Different_HTTP_Status_Codes">4 Different HTTP Status Codes</a></li><li><a href="#How_to_Find_Broken_Links_in_the_Selenium_WebDriver">5 How to Find Broken Links in the Selenium WebDriver?</a><ul><li><a href="#1_Retrieve_all_links_from_a_web_page">5.1 1. Retrieve all links from a web page: </a></li><li><a href="#2_Verify_each_link">5.2 2. Verify each link: </a></li><li><a href="#3_Check_the_status_code">5.3 3. Check the status code: </a></li><li><a href="#4_Repeat_for_each_link">5.4 4. Repeat for each link: </a></li><li><a href="#5_Report_the_Results">5.5 5. Report the Results: </a></li></ul></li><li><a href="#Conclusion">6 Conclusion</a></li><li><a href="#Frequently_Asked_Questions">7 Frequently Asked Questions</a><ul><li><a href="#How_do_you_handle_broken_links">7.1 How do you handle broken links?</a></li><li><a href="#What_causes_a_broken_link">7.2 What causes a broken link?</a></li><li><a href="#How_do_I_track_broken_links">7.3 How do I track broken links?</a></li></ul></li></ul></div>
<h2 id="h-what-are-broken-links">What are Broken links?</h2>



A broken link on a website leads to a page or resource that is no longer available. This can happen for various reasons, such as the page being deleted, moved to a different URL, or the website being temporarily down. Broken links can occur on any website and can be found on pages, images, videos, and other resources.



Broken links can be a problem for website visitors and search engines, leading to a poor user experience and negatively impacting search engine optimization (SEO). When a user clicks on a broken link, they will typically be taken to a &#8220;404 not found&#8221; page, which can be frustrating and cause them to leave the website. Additionally, search engines may view broken links as a sign of a poorly maintained website, which can lead to a lower search engine ranking.



To avoid these issues, it&#8217;s essential to regularly check your website for broken links and fix them as soon as they are discovered. This can be done manually by visiting each page of your website and clicking on each link to see if it leads to a &#8220;404 not found&#8221; page or by using automated tools such as <a href="https://www.selenium.dev/" target="_blank" rel="noreferrer noopener">Selenium</a> or other link checker software.



<h2 id="common-reasons-for-broken-links">Common Reasons for Broken Links.</h2>



There are several reasons why broken links may appear on a website, including.&nbsp;



<ul><li>Page Deletion: A previously linked page may have been deleted, causing the link to lead to a &#8220;404 not found&#8221; page.</li><li>URL Changes: If a page&#8217;s URL is changed, any links pointing to the old URL will become broken.</li><li>Website Redesign: During a <a href="https://www.webcitz.com/website-design/redesign.html" target="_blank" rel="noreferrer noopener nofollow">website redesign</a>, URLs may change, causing previously valid links to become broken.</li><li>Moving to a New Domain: If a website is moved to a new domain, all links pointing to the old domain will become broken.</li><li>Typos: Mistyping a URL or copying and pasting it wrongly can result in a broken link.</li><li>Third-Party Content: If a website links to content hosted on another website, and that content is removed or moved, the link will become broken.</li><li>Server Outages: If a website is temporarily down, links to that site will become broken.</li></ul>



By regularly checking for and fixing broken links, website owners can ensure that their website provides a good user experience and positively impacts their SEO.



<h2 id="reasons-to-check-for-broken-links">Reasons to Check for Broken Links</h2>



There are several reasons why it&#8217;s important to check for and fix broken links on a website:



<ul><li>User Experience: Broken links can frustrate website visitors and lead to a poor user experience. If a user clicks on a broken link, they will be taken to a &#8220;404 not found&#8221; page, which can be confusing and cause them to leave the website.</li><li>Search Engine Optimization (SEO): Search engines view broken links as a sign of a poorly maintained website, which can negatively impact the website&#8217;s search engine ranking.</li><li>Maintenance: Broken links can signal that a website needs to be regularly maintained and updated, leading to other issues, such as outdated content and security vulnerabilities.</li><li>Data Accuracy: Broken links can lead to incorrect or outdated information being displayed on a website, which can harm the website&#8217;s reputation.</li><li>Link Equity: When a website links to another page, it passes on some of its link equity, a measure of the value that search engines place on a link. If the linked page is no longer available, the link equity is lost, which can negatively impact the linking website&#8217;s search engine ranking. Make sure to improve your metrics with the correct use of the <a href="https://buzznberry.com/link-building-services/" target="_blank" rel="noreferrer noopener nofollow">white hat link building</a> so your website can have better rankings.</li><li>By regularly checking for and fixing broken links, website owners can ensure that their website provides a good user experience, positively impacts their SEO, and is regularly maintained and updated.</li></ul>



<h2 id="different-http-status-codes">Different HTTP Status Codes</h2>



HTTP (Hypertext Transfer Protocol) status codes are numerical codes that are returned by a web server to indicate the status of a request made by a client (e.g., a web browser). The most encountered HTTP status codes are



<ul><li>200 OK: The request was successful, and the server has returned the requested content.</li><li>201 Created: The request was successful, and the server has created a new resource.</li><li>204 No Content: The request was successful, but there needs to be a representation to return (i.e., the response is empty).</li><li>300 Multiple Choices: The request has multiple options, and the server directs the client to choose one.</li><li>301 Moved Permanently: The requested resource has been moved to a new URL, and future requests should use the new URL.</li><li>302 Found (Previously &#8220;Moved temporarily&#8221;): The requested resource has been temporarily moved to a new URL.</li><li>304 Not Modified: The client&#8217;s cached copy of the resource is up to date, and the server indicates that the client should use its cached copy.</li><li>400 Bad Request: The request must be understood or include the required parameters.</li><li>401 Unauthorized: Authentication failed, or the user does not have permission for the requested operation.</li><li><a href="https://www.cloudways.com/blog/wordpress-403-forbidden-error/" target="_blank" rel="noreferrer noopener">403 Forbidden</a>: Authentication succeeded, but the authenticated user cannot access the requested resource.</li><li>404 Not Found: The requested resource could not be found on the server.</li><li>500 Internal Server Error: An error occurred on the server while processing the request.</li></ul>



<h2 id="how-to-find-broken-links-in-the-selenium-webdriver">How to Find Broken Links in the Selenium WebDriver?</h2>



To find broken links in Selenium WebDriver, you can use the following steps:



<h3 id="1-retrieve-all-links-from-a-web-page">1. Retrieve all links from a web page: </h3>



You can use Selenium WebDriver&#8217;s &#8220;findElements&#8221; method to find all the links (i.e., &#8220;a&#8221; elements) on a web page. You can store the links in a list for later processing.



<h3 id="2-verify-each-link">2. Verify each link: </h3>



For each link, you can use the &#8220;getAttribute&#8221; method to retrieve the &#8220;href&#8221; attribute of the link, which represents the URL that the link points to. You can then use the &#8220;HttpURLConnection&#8221; class in Java to check the status code of the URL.



<h3 id="3-check-the-status-code">3. Check the status code: </h3>



You can use the &#8220;getResponseCode&#8221; method of the &#8220;HttpURLConnection&#8221; class to retrieve the HTTP status code of the URL. The link is broken if the status code is not in the 200 or 300 range. You can store the broken links in a separate list for later reporting.



<h3 id="4-repeat-for-each-link">4. Repeat for each link: </h3>



Repeat the above steps on the page until all links have been checked.



<h3 id="5-report-the-results">5. Report the Results: </h3>



After all, links have been checked, you can use Selenium WebDriver&#8217;s &#8220;assert&#8221; method to check that the list of broken links is empty. If the list is not empty, you can report the broken links to the console or write them to a file for later analysis.



Let&#8217;s understand the code.&nbsp;



Code Snippet:




package demo;

import java.io.IOException;
import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.Iterator;
import java.util.List;

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;

public class Demo {

	

	public static void main(String[] args) {
// TODO Auto-generated method stub

	 System.setProperty(&#8220;webdriver.chrome.driver&#8221;,&#8221;.\\chromedriver.exe&#8221;);
	 	 
	 //Initialising the driver
	 WebDriver driver = new ChromeDriver();

		String homePage = &#8220;https://google.com/&#8221;;
		String url = &#8220;&#8221;;
		HttpURLConnection con = null;
		int responseCode = 200;

		driver = new ChromeDriver();

		driver.manage().window().maximize();

		driver.get(homePage);

		List links = driver.findElements(By.tagName(&#8220;a&#8221;));

		Iterator it = links.iterator();

		while (it.hasNext()) {

			url = it.next().getAttribute(&#8220;href&#8221;);


			try {
				con = (HttpURLConnection) (new URL(url).openConnection());

				con.setRequestMethod(&#8220;HEAD&#8221;);

				con.connect();

				responseCode = con.getResponseCode();

				if (responseCode &gt;= 400) {
					System.out.println(url + &#8221; is a broken link&#8221;);
				} else {
					System.out.println(url + &#8221; is a valid link&#8221;);
				}

			} catch (MalformedURLException e) {

				e.printStackTrace();
			} catch (IOException e) {

				e.printStackTrace();
			}
		}

		driver.quit();

	}
}




Output:



The output indicates all valid links except one broken link.



<div class="wp-block-image"><figure class="aligncenter is-resized"><img loading="lazy" src="https://lh5.googleusercontent.com/2i3tK4HGsyPniQ0F5u4Q7SCx0No9kH-G6qH2fOnJEdwznyWojGHHLXauAs782RMXo4Hkd3EjILGy7LBVeKJBbEVHrGW4M3G0kviDv7PPbD9XF4XWZ4KfhmK8PnxS46Is_G_FIdrL1XewzlsZcKepiw" alt="Output - Find Broken Links in the Selenium WebDriver" width="600" height="300" /></figure></div>



<h2 id="conclusion">Conclusion</h2>



Broken links are a common issue in web development and can negatively impact the user experience. Regular checking for broken links is essential for maintaining a functional and professional-looking website. Using tools such as Selenium WebDriver, developers can automate the process of checking for broken links, saving time and effort while ensuring a higher level of website quality. Fixing broken links provides users with a seamless and error-free browsing experience. In addition, checking for broken links can help identify other issues, such as outdated or misconfigured URLs, which can be corrected to improve the website&#8217;s overall functionality.



<h2 id="frequently-asked-questions">Frequently Asked Questions</h2>



<h3 id="how-do-you-handle-broken-links">How do you handle broken links?</h3>



Handling broken links means finding and fixing the underlying causes of the broken links. Here are some steps you can follow to manage broken links:



<ul><li>Identify the broken links: Use tools like Selenium WebDriver or online broken link checkers to identify all broken links on your website.</li><li>Analyze the cause: Determine why the link is broken. Is it due to a misconfigured URL, a page that has been moved or deleted, or a server-side error?</li><li>Fix or replace the link: If the cause of the broken link is a misconfigured URL or a page that has been moved, update the link to the correct URL. If the reason is a server-side error, contact the website owner or webmaster to resolve the issue.</li><li>Monitor the links: Regularly check for broken links to ensure they do not reappear. You can automate this process using tools like Selenium WebDriver to save time and effort.</li><li>Implement a redirect: If a page has been permanently removed, consider implementing a redirect to a related page to maintain a positive user experience.</li></ul>



<h3 id="what-causes-a-broken-link">What causes a broken link?</h3>



A variety of reasons can cause broken links. A few are listed below:



<ul><li>Moved or deleted pages: If a page has been moved or deleted, the link pointing to it will become broken.</li><li>Misconfigured URLs: Typing errors, incorrect link syntax, or incorrect casing can cause links to be misconfigured and lead to broken links.</li><li>Server-side errors: Broken links can also be caused by server-side errors, such as a misconfigured server, a server that is down, or a server that is experiencing high traffic.</li><li>Outdated links: Over time, some links may become outdated as the information they point to changes or becomes unavailable.</li><li>Third-party services: If a third-party service that your website depends on is down or unavailable, links that point to it may become broken.</li><li>Change in website structure: If the website&#8217;s structure changes, links that point to pages within the website may become broken.</li></ul>



<h3 id="how-do-i-track-broken-links">How do I track broken links?</h3>



There are several ways to track broken links on your website, including



<ul><li>Manual checking: Check each link on your website to see if it works correctly. This method can be time-consuming but is useful for small websites.</li><li>Online broken link checkers: Use online tools, such as W3C Link Checker, Dead Link Checker, or Broken Link Checker, to scan your website for broken links. These tools will identify broken links and report the issue for you.</li><li>Automated testing with Selenium WebDriver: Use Selenium WebDriver to automate checking for broken links. You can write a script that visits each page on your website, follows all the links, and checks the response code of each link to determine if it is broken.</li><li>Monitoring tools: Use monitoring tools, such as Google Analytics or Google Search Console, to monitor your website&#8217;s performance and receive notifications when broken links are detected</li></ul>

How to Find Broken links using Selenium Webdriver

How to Find Broken Links in Selenium Webdriver?

How To Scroll a Page Using Selenium WebDriver with Java?

How to Handle Multiple Tabs and Windows Using Selenium

A Complete Guide to Selenium UI testing

10 Best Incident Management Systems Of 2023

10 Best Incident Management Software List to Look For

Selenium is one of the leading automation platforms in web application testing. Using Selenium, you can automate your testing process efficiently and deliver a bug-free product quickly. This article will guide you in building and executing Selenium projects in Java. According to <a href="https://www.selenium.dev/blog/2021/selenium-survey-results/" target="_blank" rel="noreferrer noopener">reports</a>, 67% of Selenium users prefer Java for writing test scripts. Before you learn to build Selenium projects, let&#8217;s get to know the basics of Selenium and why it is popular among developers.



<div id="toc_container" class="no_bullets">Table Of Contents<ul class="toc_list"><li><a href="#Insights_Into_The_Open_Source_Selenium_Project">1 Insights Into The Open Source Selenium Project</a></li><li><a href="#Why_Are_Selenium_Project_Ideas_Popular">2 Why Are Selenium Project Ideas Popular?</a></li><li><a href="#How_to_Build_a_Selenium_Project">3 How to Build a Selenium Project</a></li><li><a href="#H">4 H</a></li><li><a href="#How_To_Create_A_Selenium_Project_With_Eclipse_IDE">5 How To Create A Selenium Project With Eclipse IDE?</a></li><li><a href="#Sample_Selenium_Project_using_Java">6 Sample Selenium Project using Java</a></li><li><a href="#Conclusion">7 Conclusion</a></li><li><a href="#Frequently_Asked_Questions">8 Frequently Asked Questions</a><ul><li><a href="#How_to_Write_Test_Scripts_in_Selenium">8.1 How to Write Test Scripts in Selenium?</a></li><li><a href="#Which_Language_is_Best_for_Selenium">8.2 Which Language is Best for Selenium?</a></li><li><a href="#Is_Selenium_Testing_Easy">8.3 Is Selenium Testing Easy?</a></li></ul></li></ul></div>
<h2 id="insights-into-the-open-source-selenium-project">Insights Into The Open Source Selenium Project</h2>



Selenium is one of the top open-source test automation frameworks that has been in the automation space for a very long time. It can be used to automate web testing across different browsers and platforms.



The latest version is Selenium 4 (4.20.1 as of April 26, 2024). The market share of Selenium is 28.19% in the software testing space.



Selenium is still widely used by many companies (approximately 68,253 companies)



<figure class="wp-block-image"><img src="https://lh7-us.googleusercontent.com/WWhcR4l9D5CSh8CHgEyKW52ADNEaCVNvog3M_Nn4ZFsN5hJBzBBfg6AzxAWKVDTds778U3hb_0zMD_SbX64bu2n7pycwLm_OGjAbYOI6Gl0e88OixU3GRPLeW_3ah4F3vXhhVgnvLLXhT7c004WX-xQ" alt="" /></figure>



<a href="https://enlyft.com/tech/products/selenium" target="_blank" rel="noreferrer noopener nofollow">Credits</a>



<h2 id="why-are-selenium-project-ideas-popular">Why Are Selenium Project Ideas Popular?</h2>



Selenium has been one of the most widely used open-source testing frameworks by web developers and testers since its launch in 2004. It supports diverse programming languages like Java, C#, Python, Perl, PHP, Ruby, and JavaScript, giving testers more flexibility in creating test scripts, which is the key reason for the success of Selenium projects.&nbsp;



Selenium is also compatible with various top browsers and platforms. This compatibility allows students to build full-stack and MERN stack projects, which is also a main reason behind Selenium&#8217;s success. It is used by top global companies like Facebook, Amazon, etc.



<h2 id="h-how-to-build-a-selenium-project">How to Build a Selenium Project</h2>



You can build Selenium projects in multiple ways. For example, you can build and configure Selenium Projects using Java, Maven, and more. Here, we will see how to build a Selenium project using Java. You need to download the following software to build and configure Selenium projects:



<ul><li>Java: Install Java or JDK as we use Java to build a Selenium project.</li><li>Eclipse: You have to <a href="https://www.eclipse.org/downloads/" target="_blank" rel="noreferrer noopener">install Eclipse IDE</a> (Integrated Development Environment). It helps write, run and maintain tests or projects.</li></ul>



Note &#8211; You can also use some other IDE like IntelliJ IDEA or NetBeans.



<h2 id="h-h">H</h2>



<h2 id="how-to-create-a-selenium-project-with-eclipse-ide">How To Create A Selenium Project With Eclipse IDE?</h2>



You can avoid all this complexity and simplify the test automation process using low-code testing tools like Testsigma. Unlike Selenium, which requires you to create test scripts using programming languages, Testsigma simplifies automated testing using NLPs, Recorder, or Data-driven testing.&nbsp;



You can create tests similar to Natural Language, making it a very user-friendly and powerful tool for testers, developers, and stakeholders. Take a quick look at the detailed comparison between <a href="https://testsigma.com/comparison-automated-testing-tools-selenium-testsigma">Selenium vs Testsigma</a>.



The most simple way to build a Selenium project is by using Java. To do this, first, you must manually download all the required Selenium jars and the <a href="https://testsigma.com/blog/test-automation-frameworks/" target="_blank" rel="noreferrer noopener">testing frameworks</a>. 



Note &#8211; We are using Eclipse to execute Selenium test scripts in this example.



Steps to Build and Configure a Selenium Project



Step 1 &#8211; Click on the File menu from the navigation. Select New, then click on Java Project. (File -&gt; New -&gt; Java Project)



<div class="wp-block-image"><figure class="aligncenter is-resized"><img loading="lazy" src="https://images.storychief.com/account_36412/Screenshot2023-01-22021634_61697469ecd98961a9f64298341fbd73_2000.png" alt="Selenium projects" width="600" height="400" /></figure></div>



If you&#8217;re using an old version of Eclipse, and if you don&#8217;t find Java Project under New, follow these steps,



<ul><li>Select New and click on Project.</li><li>Now, select Java Project and click Next.</li></ul>



Step 2 &#8211; Enter the Project Name building-selenium-project-Testsigma and click Finish.



<div class="wp-block-image"><figure class="aligncenter is-resized"><img loading="lazy" src="https://images.storychief.com/account_36412/Screenshot2023-01-22022645_58840e512d47246bdc14d2e2ac326a6d_2000.png" alt="Create a Java project" width="600" height="400" /></figure></div>



The Java project building-selenium-project-Testsigma is successfully created in the package explorer of Eclipse.



<div class="wp-block-image"><figure class="aligncenter is-resized"><img loading="lazy" src="https://images.storychief.com/account_36412/Screenshot2023-01-22022816_828ce0f38799fc3c741833acf80c2403_2000.png" alt="Selenium projects" width="300" height="300" /></figure></div>



Step 3 &#8211; Now, you need to manually install the necessary Selenium jars and a unit testing framework like TestNG jars. To download these, go to the <a href="https://www.selenium.dev/downloads/" target="_blank" rel="noreferrer noopener">Selenium downloads page</a> and check on the latest versions.



<div class="wp-block-image"><figure class="aligncenter is-resized"><img loading="lazy" src="https://images.storychief.com/account_36412/Screenshot2023-01-22022852_485e15f42496d7a5b08663cbfa2cf3bc_2000.png" alt="Selenium jars" width="600" height="400" /></figure></div>



Step 4 &#8211; Let&#8217;s install the TestNG jars now.



<div class="wp-block-image"><figure class="aligncenter is-resized"><img loading="lazy" src="https://images.storychief.com/account_36412/Screenshot2023-01-22023134_4cb18157c08940ac6f8149135aeca498_2000.png" alt="TestNG" width="600" height="400" /></figure></div>



Step 5 &#8211; After you download Selenium server standalone jar, you should add it to the Selenium project. To do so, right-click on the Java Project created, then hover your mouse on Build Path and click on Configure Build Path.



<div class="wp-block-image"><figure class="aligncenter is-resized"><img loading="lazy" src="https://images.storychief.com/account_36412/Screenshot2023-01-22023730_9d9e162fccbb8cc618335846b6b703b7_2000.png" alt="Build Path" width="600" height="400" /></figure></div>



Step 6 &#8211; Now, select Add External JARS&#8230; option to add the downloaded jar file to the Selenium projects.



<div class="wp-block-image"><figure class="aligncenter is-resized"><img loading="lazy" src="https://images.storychief.com/account_36412/Screenshot2023-01-22023918_1e2fae6c8507ec65796c6ff9a6dfa453_2000.png" alt="Add External Jars" width="600" height="400" /></figure></div>



Step 7 &#8211; Select the jar selenium-server-standalone and TestNG and click Open.



<div class="wp-block-image"><figure class="aligncenter is-resized"><img loading="lazy" src="https://images.storychief.com/account_36412/Screenshot2023-01-22023944_a1258f10c6a83f91b79afbc38517ea79_2000.png" alt="Add file" width="300" height="100" /></figure></div>



Note &#8211; This Jar file will help start the Selenium Server.



Step 8 &#8211; Verify if the jar files are added to the Build Path and click on Apply and Close.



<div class="wp-block-image"><figure class="aligncenter is-resized"><img loading="lazy" src="https://images.storychief.com/account_36412/Screenshot2023-01-22024013_07982699209540db48ab65c9939c9f5a_2000.png" alt="Verify the build path" width="600" height="400" /></figure></div>



Note &#8211; Check if the downloaded jar files are under Referenced Libraries 



Step 9&nbsp;– Now, right-click on the src and create a package&nbsp;building-selenium.



<div class="wp-block-image"><figure class="aligncenter is-resized"><img loading="lazy" src="https://images.storychief.com/account_36412/Screenshot2023-01-22024218_142b94ca06f605fa2f8db014eb5b3936_2000.png" alt="Create a Java Package" width="600" height="400" /></figure></div>



As you can see, the building-selenium is created.



<div class="wp-block-image"><figure class="aligncenter is-resized"><img loading="lazy" src="https://images.storychief.com/account_36412/Screenshot2023-01-22030709_86879842d5b86c43fd85c8850ed65615_2000.png" alt="" width="300" height="300" /></figure></div>



Step 10 &#8211; Next, right-click on the package created. Select New, then Class, to create a Java Class. Now, enter the Java Class name and click Finish.



<div class="wp-block-image"><figure class="aligncenter is-resized"><img loading="lazy" src="https://images.storychief.com/account_36412/Screenshot2023-01-22025809_da59738cc1fbde92f6ff23b551bab208_2000.png" alt="Create a Java Class" width="600" height="400" /></figure></div>



This will create a BuildingSeleniumProject.java file, where you can type your test script.Now that we have configured Selenium, let&#8217;s write a sample test case in Java.



<h2 id="h-sample-selenium-project-using-java">Sample Selenium Project using Java</h2>



The following is a sample Selenium project created using Java in Eclipse,




package building_selenium;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
public class BuildingSeleniumProject {
	
	 public static void main(String[] args) {
	
	 // declaration and instantiation of objects/variables
	 System.setProperty(&#8220;webdriver.chrome.driver&#8221;, &#8220;C:\\Users\\Yamini\\Downloads\\chromedriver.exe&#8221;);
	 WebDriver driver=new ChromeDriver();
	
	 // Launch website
	 driver.navigate().to(&#8220;http://www.google.com/&#8221;);
	
	 // Click on the search text box and send value
	 driver.findElement(By.name(&#8220;q&#8221;)).sendKeys(&#8220;Testsigma tool&#8221;);
	
	 // Click on the search button
	 driver.findElement(By.name(&#8220;btnK&#8221;)).submit();
	}
}




Code Explanation



This program, on successful execution, will open Google search, type the search text used in the search bar, and display the result page.



Output



When executed, the above code will open Google on the Chrome browser, automatically type the text &#8220;Testsigma tool&#8221; in the search bar and then display the top search results catering to the search.



<div class="wp-block-image"><figure class="aligncenter is-resized"><img loading="lazy" src="https://images.storychief.com/account_36412/Screenshot2023-01-22194958_e8cb3aaf6fcd14ffe50c7d3023ae4e14_2000.png" alt="Selenium Test Output" width="600" height="400" /></figure></div>



The above step-by-step process will help you to build and configure Selenium projects in Eclipse using Java. However, this will require testers to have coding knowledge as they will have to write test scripts using any programming language supported by Selenium. 



Selenium supports automation testing only for web applications. But if you&#8217;re looking to perform web, mobile app or API testing without needing to program test cases, try out <a href="https://testsigma.com/" target="_blank" rel="noreferrer noopener">Testsigma</a>. 



Testsigma is a cloud-automation testing platform that lets you write tests using simple English. It is a low-code automation platform that does not require much coding knowledge. So, it can be used by anyone on your team. It is a cloud-based tool, so you don&#8217;t have to install anything to use the software.



<h2 id="h-conclusion">Conclusion</h2>



Building and executing a Selenium project may be a bit complicated, but I hope the above step-by-step instructions will help you build your Selenium projects in no time. 



<a href="https://testsigma.com/blog/automated-test-scripts/" target="_blank" rel="noreferrer noopener">Automating test scripts</a> using Selenium for your web application makes testing easy. This helps the developers to identify bugs and solve them quickly. However, if you are looking to test web apps, mobile apps or API without writing a single line of code, you should prefer our <a href="https://testsigma.com/signup" target="_blank" rel="noreferrer noopener">cloud automation platform &#8211; Testsigma</a>.



<h2 id="h-frequently-asked-questions">Frequently Asked Questions</h2>



<h3 id="h-how-to-write-test-scripts-in-selenium">How to Write Test Scripts in Selenium?</h3>



Before you start writing test scripts in Selenium, you need to download the necessary components such as Java, IDE, Drivers, Selenium Jars, and, TestNG required to build the test environment. Once you download and complete the setup process, you can then easily write test scripts in Selenium.



<h3 id="h-which-language-is-best-for-selenium">Which Language is Best for Selenium?</h3>



Java is one of the top programming languages used by testers worldwide. But you can also use other languages like C#, Ruby, JavaScript, and Python. Also, there are a few factors you need to consider when you try to learn a language, like the project requirements and the client&#8217;s preference.



<h3 id="h-is-selenium-testing-easy">Is Selenium Testing Easy?</h3>



Automating testing using Selenium is quite easy, but you need to have prior knowledge of programming languages like Java, Ruby, and Python to write, run, and execute test scripts.

An Easy Way to Build and Execute Selenium Projects