Tests & Metrics

Research and data available to download


 

Video Streaming Measurements

Video streaming has exploded in popularity in recent years, with some estimates putting it at over 60% of peak-time internet traffic. Global services such as Netflix and YouTube dominate the market, but there are also smaller country-level services that still have tens of hundreds of millions of users such as Hulu and BBC iPlayer. SamKnows has developed a methodology for streaming real videos from these services, thus measuring the true quality of experience (QoE) that a user received in the home. This approach is extensible to new video streaming services that arise or an ISP's own internal service.

 

Supported metrics for on-demand/OTT video streaming:

  • Bitrate reliably streamed (Highest bitrate that can be streamed without rebuffering)
  • Startup delay of video
  • Number of rebuffering events
  • Duration of rebuffering events
  • Length of video
  • Video codec
  • Video resolution
  • Receive throughput of video
  • TCP connection time to content server

Supported metrics for multicast IPTV:

  • Channel change time (Multicast group join time)
  • Jitter of video frames
  • Packet loss
  • Multicast group leave time
  • Out of order video frames
 

Speed Measurements

Speed is the primary differentiator that ISPs use when advertising their broadband products, so it is clearly an important metric to monitor. SamKnows has standardised on using TCP-based speed tests using multiple concurrent connections for its measurement projects worldwide. However, SamKnows also supports a variety of other mechanisms that can be useful in niche cases or alongside the standard measure for comparative purposes.

 

Supported tests:

  • TCP download speed (from 1 to 128 concurrent TCP connections)
  • TCP upload speed (from 1 to 128 concurrent TCP connections)
  • UDP download speed
  • UDP upload speed
  • Hardware accelerated UDP download speed (Broadcom and Qualcomm/Ikanos)
  • Hardware accelerated UDP upload speed (Broadcom and Qualcomm/Ikanos)

Supported metrics:

  • Average speed across the entire test
  • Peak speed
  • Speed at fixed intervals during the test
  • Test duration
  • Bytes transferred
  • TCP-level debug information, including RTT estimates, retransmissions and more
 

 

Low-level Measurements

Latency, packet loss and DNS measurements are fundamental to any kind of network measurement infrastructure. All internet based services rely upon being able to send packets reliably and being able to resolve hostnames into IP addresses. Issues seen in these metrics will manifest themselves in a variety of ways across other metrics (e.g. delays in DNS resolution will cause delays in web browsing, but have minimal impact on speed tests).

Supported tests:

  • UDP round-trip latency and packet loss
  • UDP one-way jitter
  • ICMP round-trip latency and packet loss
  • Disconnection events
  • DNS response time
  • DNS failure rate
 

 

Web Browsing Measurements

Web browsing continues to be the activity that users most commonly associated internet usage with. Studies have shown that delays of just a second when loading a web page can result in millions of dollars of lost revenue to large online stores, purely due to user dissatisfaction. Clearly measuring web browsing performance is fundamental to an internet measurement platform. Websites have exploded in complexity too; nowadays a website is composed of dozens or hundreds of objects, perhaps served partially or entirely by a CDN. SamKnows is well placed to measure all of this in a reliable and consistent manner.

Supported tests:

  • Websites being tested
  • Number of elements on the web page
  • Number of elements successfully retrieved
  • Number of redirects
  • Total page load time
  • TCP connection established time
  • DNS resolution time
  • Time to first byte
  • Page load failure rate
  • CDNs in use, including TCP connection time and request failure rate for each
 

 

VoIP?!

Network operators are increasingly moving voice services to an IP based approach, with many housing developments being built without wiring for analogue phone lines. End-to-end measurement of call quality over IP networks is essential, particularly for the incumbent network operators who often have a minimum service level agreement for voice services.

SamKnows has a comprehensive set of voice measurements that can be configured to test a variety of different voice codecs and even interact with an ISPs own voice service.

Supported tests:

  • Round-trip latency
  • Upstream packet loss
  • Downstream packet loss
  • Upstream jitter
  • Downstream jitter
  • Mean Opinion Score (MOS)
  • Packet Loss Rate
  • Packet Discard Rate
  • Bust Density
  • Gap Density
  • Round-trip Delay
  • End-system Delay
  • Signal Level
  • Noise Level
  • R-Factor
  • MOS for listening quality
  • MOS for conversation quality
  • Current and maximum jitter delay
 

Generic information available across all measurements

 

All active measurements that SamKnows carries out include at least the following.

  • Whitebox ID
  • Date and timestamp of the test (UTC)
  • Test server hostname and IP address

Additionally, SamKnows has standardised on the following when reporting measurement data in raw form.

  • All timestamps are in UTC and in ISO8601 format
  • All measured transmission speeds are expressed bytes per second
  • All durations are expressed in microseconds
  • All data volumes are expressed in bytes
 
 

Find out what we can do for you

 

To discuss your measurement requirements and what we can do to help you please get in touch.