Quality of Experience tests

Measuring the performance of real applications over real infrastructure

Gaming

Good internet performance is particularly important for gaming. Beyond downloading your 100GB game, gamers need fast latency and low packet loss to stay competitive when gaming online. Gaming servers are spread out all over the world, but the physical distance and performance to each of them is unique to every gamer. 

We’ve built tests for the following popular games, to test the round-trip latency and packet loss performance to their real-world gaming servers. More games will be added over time.

Supported agents:

Whiteboxes, Routers, Android, iOS

Measurements

The IP address of the node that the user was directed to.

The average, minimum and maximum round-trip times of the ten packets.

The number of packets sent and received, allowing packet loss to be calculated.

Social Media

This social media tests measure the round-trip latency and reachability of a selection of major social media services.

A single social media site can use a variety of endpoints for different content types (e.g. audio, video) and different activities (downloading and uploading). These tests capture the round-trip latency to all of the supported combinations.

Supported agents:

Whiteboxes, Routers

Measurements

Average round trip latency for 10 packets (by default).

Minimum, maximum, median and standard deviations for the latency measurements.

The number of sent and received packets.

The number of hops to the endpoint (performed using a traceroute).

The resolver IP address of the endpoint.

Video conferencing

Our video conferencing tests measure round-trip latency and packet loss to the most popular online video conferencing services.

Many different protocols and communications are used during video conferencing. Our measurement focuses on traffic to the video and audio relay servers, as problems on this link will differently manifest as connectivity and quality issues to end users.

Supported agents:

Whiteboxes, Routers

Measurements

Average round trip latency for 10 UDP packets (by default).

Minimum, maximum, meditation and standard deviations for the latency measurements.

The number of sent and received packets.

The IP address of the endpoint.

Video streaming

We have developed a variety of video streaming measurements that stream real content from major video streaming providers. This allows you to gain an accurate view of how video is delivered to customers.

Supported agents:

Whiteboxes, Routers, Android, iOS

Measurements

The bitrate reliably streamed

The startup delay (the time taken to download two seconds of video)

The TCP connection time

The number of stalls and their duration (this is only applicable if the test is not running in the ‘bitrate reliably streamed’ mode)

The downstream throughput achieved

CDN measurements

CDNs are widely used by large websites to distribute static content near to end users. This reduces the burden on the content providers' servers and improves web page load times. CDN operators will place caches inside ISP networks so that traffic does not even need to leave the ISP network. This is beneficial for both the ISPs (it reduces the load on their transit and peering links) and the CDN operators too (it spreads the content serving burden to a wider array of servers).

The SamKnows CDN test measures performance to multiple CDNs by fetching a small object over HTTP. Care is taken to ensure that the same object (or a very similar one) is fetched from each CDN, to provide comparability between the CDNs themselves. Currently an ~80KiB JavaScript library is fetched from each CDN.

Supported agents:

Whiteboxes, Routers

Measurements

TCP connection time

Download speed

Object fetch time (lower is better)

The IP address of the CDN node

Web browsing

Load real pages from real websites to assess browsing performance that users would actually see in their homes rather than to a centralised test server.

Supported agents:

Whiteboxes, Routers, Android, iOS

Website Performance Test

This test uses a full headless web browser to accurately capture the user experience of browsing a website from within a user’s network. The measurements are based on what is observable on the screen, the goal is to highlight how much waiting time is needed for a user to be able to start accessing content on the page.

Web browsing lite

Measures the time taken to fetch the HTML and referenced resources from a page of a popular website. This test does not test against centralised testing nodes; instead it tests against real websites, allowing for content distribution networks and other performance enhancing factors to be considered.