Try Agora for Free
Sign up and start building! You don’t pay until you scale.
Summary: In Part 1 and Part 2 of this multi-part blog, Agora examined a recent study from Zoom that suggested the Zoom Video SDK outperformed Agora’s Video SDK on several fronts. However, in our follow-up analysis, Agora’s detailed testing demonstrated the performance advantages of its SDK in 1:1 PC calling scenarios and Multi-Host Video Calls (Desktop PCs).
In our third installment, we are looking at how the Agora and Zoom video SDKs performed side-by-side during Multi-Party Mobile Video Calls with up to 32 participants, under varying real-world network conditions.
First, a quick refresher on the genesis of this debate: In a series of benchmark tests commissioned by Zoom late last year, the video telephony company dipped its toes further into the video SDK space, comparing its offering to several high-profile competitors, including Agora.
Obviously, after nearly a decade of providing best-in-class video SDKs to thousands of customers and hundreds of use cases, the Agora team was more than confident to take up the challenge.
As you can read here and here, rounds 1 and 2 went to Agora. You’ll have to read on to see if Agora makes it three in a row!
Before diving into the results let’s examine the conditions and configurations deployed in testing.
Table 1 – Test Setup
Table 2 - Test Case Scenarios
Although this is a 32-video host test on mobile, there is little to no user experience value for rendering 32 video tiles simultaneously on a mobile screen. That is why, for a realistic user experience, pagination was implemented to solve this problem.
For Agora, we tested using our Agora Video Call V6.2.26.2533 and SDK version 3.5.2. For Zoom, we tested the recent Android Version: 5.12.8, to ensure Zoom’s best practices were properly implemented for comparison.
* All results were observed and measured on a Galaxy S10+ receiving video from a 32-video conference meeting.
Whether under optimal network conditions or during significant packet loss, Agora outperformed Zoom from top to bottom. If you’re a CEO in the money-making business, having a reliable video SDK that can function while others struggle to deliver content in adverse conditions can mean the difference between success for failure.
Consider these results:
In this test, we focused primarily on the effect of packet loss and limiting network conditions on the send and receive frame rates.
As we discussed in previous blogs, Agora has several ways to combat network disrupts that often result in costly packet loss by leveraging our SD-RTN™ network as an overlay to the public internet, implementing technologies that optimize performance over last-mile connections, and device optimizations.
By routing traffic around impairments on the internet using AI algorithms and optimally shapes real-time traffic for the optimal performance, Agora directly combats possible consumer churn.
Under normal network conditions, we see the average frame rate received during Multi-Party Mobile Video Calls Agora maintains fluent video and audio with an average received frame rate of 26 FPS. Zoom lags behind at 24 FPS.
With 25% uplink packet loss, Agora held steady, maintaining an FPS advantage of 2 to 3 at 25 FPS. Zoom dropped to a frame rate of 23.
When testing introduced downlink packet loss of 25%, Agora continued to put distance between it and the competition. While Zoom dropped to 19 FPS, Agora held steady with a healthy 25 FPS. This difference is significant enough for users to notice issues with the call.
This variation in latency occurs when sending data over a network. 600ms is considered high and will likely cause noticeable performance issues. If the jitter is consistently at 600ms or higher, it is likely due to an issue with the network connection, such as congestion or interference.
Once the network in our testing reached uplink 600ms jitter, Zoom’s FPS dropped precipitously to a low of 4 FPS.
Here Zoom’s frame rate dropped to 4 FPS generating choppy video. Once again, Agora held steady at 23 FPS. The testing again demonstrated the divide between the Video SDKs’ performance.
Much like the uplink jitter, the downlink 600ms jitter yielded similar results. Nearly identical results occur when conditions were created for downlink 600ms jitter, with Agora again at 23 FPS, while Zoom bottoms out at 4 to 5 FPS.
Again, when the frame rate is in the mid-20s, the users will experience no lag or disruption. A frame rate below 5 will begin to freeze the video.
When the test starts with no bandwidth limitation and then a low bandwidth of 500Kbps is applied, first on the send side, then applied on the receiver side, Agora performs much better than Zoom. Zoom dropped to 0 FPS for over 10 seconds total, freezing the video, and then struggled for 20 seconds to recover to 25 FPS.
Agora dips to 6 FPS but quickly rebounds to the mid-20s in under 2 seconds.
As you can see in the graph below, the initial frame rate drops to zero, rebounds briefly to 5 then struggles for 30 seconds.
For mobile multi-party video calls, we monitored CPU Octa-core, 2.80 GHz, Kryo 485.
Table 3 – CPU Comparison
For this use case, we would not expect a user experience impact for either Zoom or Agora, even with multiple applications running. Agora averaged 10.63%, while Zoom averaged 7.47%.
Agora consumed less RAM than Zoom, with an average of 254MB vs. Zoom’s average of 293MB.
Table 4 – RAM Comparison
Once again, Agora squared off with Zoom under varying real-world network conditions and came away the clear-cut winner. In every test, Agora’s video SDKs outperformed Zoom’s.
For anyone integrating video capabilities into an application, Agora provides tools that allow developers to quickly and easily add video features to any app, more reliably and efficiently than others, including Zoom. As the tests have shown, under any condition, especially in more challenging environments, Agora delivers fluent, natural audio and video, while Zoom freezes or delivers choppy video.
Tune in for our next testing updates, covering a breakdown of how Agora compares to Zoom when leveraging Video SDKs for web.
In the meantime, sign up for free to test out Agora’s performance for yourself.
