How To Collect Network Traffic Automatically With Selenium WebDriver

If you need to collect performance data and HTTP web traffic between your browser and a web server. You can use the following method to make the process automatically use selenium webdriver. This article will tell you how to capture network data to a har file in selenium webdriver.

1. What Is HAR File?

HAR is the abbreviation of HTTP Archive. You can make HTTP web traffic visualized with this format files created by HTTP communication tracking tools. This file includes detailed information about the communication between a web browser and a web server. You can also make an analysis with all this information. The har file format is similar to JSON, you can use jackson or gson libraries to parse the HTTP web communication data in the har log file.

2. How To Use BrowserMob Proxy Server To Capture Web Traffic.

The BrowserMob proxy server can capture and save the communication traffic data between the client browser and web server in HAR format. The BrowserMob Proxy server can be used as a standalone or embedded proxy server.

The code below will use BrowserMob as an embedded proxy server to capture HTTP traffic. Create an eclipse maven project and add the below dependency in the project pom.xml file <dependencies> XML element. You can read the article How To Add Selenium Server Standalone Jar File Into Eclipse Java Project And Maven Project to learn more.

<dependency>
   <groupId>net.lightbody.bmp</groupId>
   <artifactId>browsermob-core</artifactId>
   <version>2.1.5</version>
   <scope>test</scope>
</dependency>

Save the eclipse maven project will download the BrowserMob proxy server used client jar files into your eclipse maven local repository. It is recommended to use the maven to add the BrowserMob proxy server jar files into the eclipse java maven project because it will download other required jar files automatically.

Create a java class file in eclipse, the java class file path is \src\main\java\com\dev2qa\webdriver\WatchTraffic.java, below is the java project structure.

> tree /F ./
│   pom.xml
│   yahoo.har
│
├───.settings
│       org.eclipse.jdt.core.prefs
│       org.eclipse.m2e.core.prefs
│
├───src
│   ├───main
│   │   ├───java
│   │   │   └───com
│   │   │       └───dev2qa
│   │   │           └───webdriver
│   │   │               │
│   │   │               WatchTraffic.java

Add the below code in the WatchTraffic.java file. When you run the below code in eclipse, it will generate a yahoo.har file in the eclipse maven project root folder.

package com.dev2qa.webdriver;

import java.io.FileOutputStream;

import org.openqa.selenium.Proxy;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.firefox.FirefoxOptions;
import org.openqa.selenium.remote.CapabilityType;
import net.lightbody.bmp.BrowserMobProxy;
import net.lightbody.bmp.BrowserMobProxyServer;
import net.lightbody.bmp.client.ClientUtil;
import net.lightbody.bmp.core.har.Har;
import net.lightbody.bmp.proxy.CaptureType;

public class WatchNetworkTraffic {
    
    
    
    public static void CreateHarByBrowserMobProxy() {
        
        try {
            
            // Create an instance of BrowserMob proxy server.
            BrowserMobProxy proxy = new BrowserMobProxyServer();
            
            // Start the BrowserMob proxy server.
            proxy.start(0);
            
            // Get the BrowserMob proxy server listening port.
            int port = proxy.getPort();
            
            System.out.println("BrowserMob proxy server port number is : " + port);
            
            // Get the selenium proxy object that communicate with the BrowserMob proxy server. 
            Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);
            
            // Because we use Firefox to communicate through the proxy server so add some Firefox options.
            FirefoxOptions firefoxOptions = new FirefoxOptions();

            // Set Firefox to use the BrowserMob proxy server.
            firefoxOptions.setCapability( CapabilityType.PROXY, seleniumProxy);
           
            // Avoid the Warning: Potential Security Risk Ahead error when use Firefox to get the har file. 
            // It can also avoid the unsecured massage page when use google chrome to invoke the BrowserMob proxy server.
            firefoxOptions.setAcceptInsecureCerts(true);
            
            // Because Firefox use the geckodriver, so set the geckodriver.exe path to the system property webdriver.gecko.driver's value.
            System.setProperty("webdriver.gecko.driver", "C:\\WorkSpace\\Tool\\geckodriver-v0.29.1-win64\\geckodriver.exe");
                    
            // Create the FirefoxDriver object with the above FirefoxOptions object and start the Firefox web browser.
            WebDriver driver = new FirefoxDriver(firefoxOptions);
    
            // Enable the HAR capture content type. The CaptureType class contains a list of the content type.
            proxy.enableHarCaptureTypes(CaptureType.REQUEST_CONTENT, CaptureType.RESPONSE_CONTENT);

            // Give the new HAR a name "www.yahoo.com".
            proxy.newHar("www.yahoo.com");
    
            // Open yahoo home page in the Firefox.
            driver.get("https://www.yahoo.com");
    
            // Retrieve the HAR data for the communication between yahoo.com.
            Har har = proxy.getHar();
            
            // Create a new FileOutputStream object.
            FileOutputStream fos = new FileOutputStream("./yahoo.har");
            
            // Save the HAR data to the file.
            har.writeTo(fos);
            
        }catch(Exception ex) {
            ex.printStackTrace();
        }
        
        

    }
    

    public static void main(String[] args) {
        
        WatchNetworkTraffic.CreateHarByBrowserMobProxy();
        

    }

}

Below is the pom.xml file content.

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>2021JavaMavenProject</groupId>
  <artifactId>2021JavaMavenProject</artifactId>
  <version>0.0.1-SNAPSHOT</version>

  <dependencies>
  
    <dependency>
        <groupId>org.seleniumhq.selenium</groupId>
        <artifactId>selenium-java</artifactId>
        <version>3.141.59</version>
    </dependency>
    
    <dependency>
        <groupId>net.lightbody.bmp</groupId>
        <artifactId>browsermob-core</artifactId>
        <version>2.1.5</version>
    </dependency>
    

  </dependencies>
</project>

3. How To Pass The Warning: Potential Security Risk Ahead Web Page Returned From Selenium Browsermob Proxy.

When you comment on the code line firefoxOptions.setAcceptInsecureCerts(true); in the above source code and run the example, you may encounter the error page with the message Warning: Potential Security Risk Ahead in the Firefox web browser.

When you use google chrome to run this example, it will also show a similar web page that displays unsecured massage. This issue will stop the example to run, but as you can see, you just need to add the code firefoxOptions.setAcceptInsecureCerts(true); to pass this issue.

4. How To Visualize The HAR File.

You can use chrome extension har file viewer to visualize the content in the generated har file.