Load Testing Of WebRTC Media Server

Testing WebRTC applications can be a challenging task. From simple one-to-one calls to optimizing for large group video calls, it requires efficient and effective automation.

WebRTC media servers are essential components in real-time communication applications as they are responsible for managing media streams. It is imperative to verify that these servers have the capability to support numerous connections and media streams concurrently in order to uphold optimal performance and dependability. Performing load testing is a crucial measure in evaluating the capacity and efficiency of WebRTC media servers when faced with demanding circumstances.

WebRTC media servers are available in different types, each offering unique functionalities:

1. SFU (Selective Forwarding Unit):
   • Forwards media streams from multiple clients with minimal processing involved
   • Examples: Janus, Jitsi Videobridge, Mediasoup
2. MCU (Multipoint Control Unit):
   • Mixes media streams from multiple clients before sending them back
   • Examples: Kurento, OpenVidu
3. Hybrid Solutions:
   • Combine the features of SFUs and MCUs
   • Examples: Wowza Streaming Engine, Red5 Pro

Conducting load testing for any application involves the creation of a group of virtual users, replicating user behaviors, and replicating the server load. The process of load testing differs significantly between traditional web applications and WebRTC applications. While JMeter is a widely used tool for testing traditional web applications, testing WebRTC applications, which involve peer-to-peer media communication through the browser, requires a unique approach to simulate these usage scenarios. The complexity increases when testing involves the transmission and reception of Audio and Video streams, necessitating the measurement of stream quality, frame rate, and latency between endpoints, in addition to analyzing client actions, server performance, load balancers, and media server efficiency. Therefore, selecting the appropriate technology stack is crucial.

To conduct Load testing on a media server, it is essential to automate the test execution by utilizing actual web browsers such as Chrome, Firefox, Safari, etc. These browsers come equipped with built-in support for WebRTC. Additionally, multiple browser instances need to be running simultaneously on various machines. Selenium Grid, a component of the Selenium Suite, is designed specifically for executing numerous tests concurrently on different browsers, operating systems, and machines. It facilitates distributed test execution and enables tests to be run in a distributed testing environment.

Set Up the Test Environment:

1. Realistic Scenario Simulation:
   • Mimic real-world usage patterns to get accurate results.
   • Consider different user behaviors, such as joining, leaving, and active participation in calls.
2. Isolate the Environment:
   • Ensure that the test environment is isolated from production to avoid impacting real users.
   • Use similar hardware and network conditions to the production environment.

Selenium Grid has 2 basic entities :

Hub/Master

• A Selenium Hub is a central point (which can be a local machine) that receives all the test requests and distributes them to the right nodes. The machine which actually triggers the test case if known as the Hub.
• There can be only one hub in a Selenium grid.
• The machine containing the hub triggers the test case, but the actual test execution is carried out on Node machines.

Node/Slaves

• Nodes are the Selenium instances that will execute the tests that are loaded on the hub.
• There can be one or more nodes in a grid.
• Nodes can be launched on multiple machines with different operating systems and browsers.

The test bed can be setup as an on-premises solution or a cloud solution. There are several players in the public cloud / SaaS market such as Saucelabs and Browserstack. Using a cloud service is a good approach, as these providers offer the entire infrastructure with a large range of browser & OS combinations.

In this article, we will look at a Selenium grid and node setup on AWS cloud. The entire solution needs to be hosted in a single Region, within a single custom VPC. All selenium grid components – HUB & Nodes should fall within the same subnet.​ Hub machine will act as a Master and will require a GUI to watch the node instances.​ Testing nodes can be operated in Non-GUI mode and will act as Slaves. Test needs to be executed in Headless Mode which facilitates faster test execution and less usage of resources on client machines.

 

Test setup is categorized into 3 phases:

1. Number of meetings in progress

• This phase includes creating and setting up test environment, software installations, setting up security policies and opening the required Outgoing/Incoming ports.
• While forming the grid proper timeouts and configurations should be specified.
• Modify the system configuration to increase file descriptors to allow many socket + API connections on backend application server.

2. Test Execution phase:

• This phase includes creating the necessary test data that acts as an input for the test scripts.
• Specify IP’s in Nodelist file and provide appropriate values in config file.
• Tests can either be executed either sequentially or in parallel depending on the test scenarios.
• Session duration can be set from few minutes to hours. Certain action will be triggered on client browsers to avoid timeout. All such configuration should be configured in the property file.
• Resource profiling of server should be done while test execution is in progress.

3. Test Analysis phase:

• This phase includes analyzing the outcome of test execution. Test reports will be created to show the status of test execution.
• Analyzing screenshots and client browser logs in case of failure or Audio/Video loss.
• Creating detailed test report with status of test execution, server statistics and resource metrics graphs.

 

Types of Testing

Tests are broadly classified on the basis of their objectives :

1. Load Testing

• Evaluate the number of meetings which can be accommodated on a particular media server instance by monitoring the Memory and CPU utilization while gradually increasing the load during testing.
• Ensure that all meetings can work as functionally intended without causing any performance degradation by capturing screenshots of the admin portal.
• Perform test actions based on usage and simulate various user patterns, for example, users can perform actions such as Audio calls, Video Calls, Audio calls and Screenshare, Video calls and Screenshare, Annotations and Messages etc.
• Ensure there are no memory leaks on the backend application server by comparing memory usage pre and post execution of load test.
• Ensure that all Websockets are released by the backend application server after the test completion.

2. Scalability Tests

Due to the above constraints, server scalability is achieved by implementing a dedicated Seed Node server which is responsible for redirecting users to the appropriate Media server instance as well as manages Scale-in and Scale-out of the instances. Additionally, there should be a portal to monitor Media server instances and their status like CPU, Memory consumption, no. of meetings etc.

The Seed Node Server has configurations like threshold CPU / Memory / no. of meetings etc. and it starts with a batch of media server instances. Following scenarios provides an insight to Scalability testing:

Profiling to determine benchmark of Media Server

Perform Profiling activity of media server by increasing 1 meeting at a time and monitoring CPU and Memory usage as shown in the diagram below. This test helps to determine the the threshold value which acts as a pivot point Seed Node configuration.

Profiling Media server by executing a load test

• Scale OUT

Increase the load of meetings and users using load script so that the threshold value on Seed Node server is crossed. Verify that beyond this no further meetings will be hosted on the current a server instance and that the Seed Node server spins up a new server instance.

• Scale IN

Stop the currently running meetings using the load test script. Ensure that when the meetings count on a particular media server instance reaches zero, it is marked as idle and stopped after a predefined time.

Results

By following the above steps, we are able to determine the performance of the WebRTC application at various load scenarios and verify whether the application adheres to the expected performance requirements for the given load.