CPE Embedded Client SDK


 

Designed to enable the easy integration of SamKnows testing into hardware devices, the Embedded Client SDK gives you everything you need to build internet performance measurement into your products.

The SDK currently supports the core SamKnows measurements, including:

  • Download speed
  • Upload speed
  • Latency
  • Packet loss
  • Jitter

By incorporating the SamKnows Embedded Client SDK your device will become immediately compliant with the latest legislation, including the EU Net Neutrality regulation and others from around the world.

Future versions of the SDK will incorporate more of the SamKnows test suite. Notifications of updates to the SDK will be sent to you via email. To request for additional email addresses to receive notifications, please contact your SamKnows account manager.

The SDK documentation presented here and the accompanying source code provide everything you need in order to integrate the SamKnows measurements inside your device.


1What's included

/sdk-release/clients

Contains the sources of the software to run in your device. Specifically:

Package Path Version
Download and Upload TCP Speed Test http_test 3.2.0
Latency, Packet Loss and Jitter Test jitter_client 1.0.2

/sdk-release/servers

Contains the sources of the software running on SamKnows servers. This is provided only for testing purposes, since SamKnows will take care of the servers infrastructure.

Package Path Version
TCP Speed Test Server skhttp_server 1.2.2
Latency, Packet Loss and Jitter Test Server jitter_server 1.0

/sdk-release/docs

This documentation.

/sdk-release/scripts

Shell scripts with the logic to find nearby test servers, execute the tests, output the results and send a copy of these to SamKnows.


2How it works

Integrating the SDK into an embedded device is a straightforward process.

  1. Compile the test clients (found in the /sdk-release/clients) directory for your platform.
  2. Copy the compiled code, plus the contents of the /sdk-release/scripts directory to a device.
  3. You may need to modify the /sdk-release/scripts/run_test.sh file to specify revised paths to the binaries.
  4. To execute the tests, either according to a schedule or on an ad-hoc basis, simply run the /sdk-release/scripts/run_test.sh script. This script will perform the following actions
    • Contact a SamKnows API which will perform a geo-IP lookup on the client’s source address, locate nearby measurement servers, and return a list of candidates
    • Run a brief latency test to each candidate test server, ultimately determining the nearest server in terms of latency
    • Run all of the test clients against the nearest server
    • Submit the results from the tests to SamKnows, along with a unique device identifier
    • Report the results to standard output
  5. Interpret the results using the output formats described in the Test Clients documentation linked above.

Memory:

The SDK takes up approximately 1MB of flash space, depending on libraries already available in the CPE.

We only consume RAM when tests are running, which is very infrequently. It will use up to ~3MB RAM when running a full rate speed test at over 100Mbps.


3Building the SDK

You will need to build the client binaries for your devices. The file clients/ReadMe explains how to do this.

Documentation for the individual test clients can be found from the Test Clients link at the top of this page.

When submitting the results to SamKnows a device unique id will need to be used. script/run_test.sh contains a get_id() function supposing a standard Linux kernel with a eth0 interface. This function may need to be modified to generate a better ID using the specifics of your platform.

In case you need to build the servers for testing, they use the common Autotools build system. The following will allow you to build them:

autoreconf -fi && ./configure && make

4Submitting Results to SamKnows

Part of the SDK agreement involves ensuring that all measurement results are transmitted back to SamKnows. The code in the /sdk-release/scripts/ takes care of this automatically. If you need to modify this to cater for your integration, you need to ensure that the following three things are transmitted to SamKnows each time a measurement is run:

  1. The measurement results in raw form, as returned by the test clients
  2. The Internet-facing IP address of the user
  3. A persistent and unique device identifier, such as a MAC address or a hash of one
  • TCP Speed Test (http_test)
    • The HTTP test can create GET and POST HTTP requests with multiple parallel connections to measure the available downstream and upstream bandwidth.

      The test consists of two phases: a warmup period and a measurement period. The warmup period is there purely to let TCP reach the line speed. The variables that will give the test result are the bytes transferred during the measurement period and the length of time the test lasts. The warmup period can be configured to wait for a stable speed, stable being defined by the command line options.

      The results can consist of a single output line, or of multiple lines. In the later case each line additionally reports the speed during a specific time period of the test run.

      Command line options

      <-n|--node> <server hostname>
      <-s|--service> <server port>
      <-p|--path> <file path>
      <-c|--connections> <number of concurrent connections>
      <-m|--mode> <GET|POST>
      [<-a|--aggregate-warmup>]
      [<-b|--tcp-trace-interval> <trace interval in ms>]
      [<-d|--no-cache-control>]
      [<-e|--tcp-no-delay>]
      [<-F|--socket-send-buffer> <socket send buffer in bytes>]
      [<-g|--socket-recv-buffer> <socket receive buffer in bytes>]
      [<-h|--interface> <network interface to bind to>]
      [<-i|--protocol> <IPv4|IPv6>]
      [<-j|--results-interval-period> <results interval period in seconds>]
      [<-k|--result-num-intervals> <number of interval results>]
      [<-l|--minimum-payload> <minimum number of bytes transferred in total after warmup to consider the test valid>]
      [<-P|--pipeline-depth> <number of GET requests to pipeline>]
      [<-r|--send-random-data> [<period_in_bytes>]]
      [<-S|--server-side>]
      [<-t|--test-data-cap>] <data cap for the whole test in bytes>
      [<-T|--transfer-max-size>] <consider the test finished once this number of bytes has been reached during the transfer period>
      [<-u|--cubic>]
      [<-U|--network-timeout>]
      [<-v|--warmup-sample-time> <warmup sample time in ms>]
      [<-w|--num-warmup-samples> <number of samples to check in warmup period>]
      [<-W|--max-warmup-size> <maximum number of bytes downloaded between all the connections during warmup>]
      [<-x|--max-warmup-samples> <maximum number of samples to check in warmup period>]
      [<-y|--warmup-dont-fail-on-max>]
      [<-z|--warmup-tolerance> <error margin when checking for constant speed in %>]
      

      Notes:

      • aggregate-warmup
        During the warmup period wait for the sum of the speeds of all the connections to be stable, instead of all the speeds of each connection independently
      • tcp-trace-interval
        Record TCP statistics (cwnd, ssthresh, etc.). This will record trace files on the server side, if enabled at the server.
      • –server-side
        Try to use the extended protocol to communicate with the SamKnows HTTP server. This allows to ask the server to use a specific congestion control algorithm in downstream tests, do server side measurements in upstream tests, etc.This option is strongly recommended.
      • –cubic
        Force TCP Cubic instead of whatever is the default in the client/server

      Recommended usage:

      Download:

      ./new_http_test -n <server hostname> -s <server port> -p /1000MB.bin -c 3 -m GET -j 10 -k 1 -v 5000 -w 1 -x 1 -y -a -S -u
      

      Upload:

      ./new_http_test -n <server hostname> -s <server port> -p /1000MB.bin -c 3 -m POST -j 10 -k 1 -v 5000 -w 1 -x 1 -y -a -S -u
      

      Things to keep in mind when looking at results

      We are looking at user-space stream socket data. If there are losses and duplicate packets, what we will see is one sample with a lower than average value, and the next sample with a higher than average value. The average of both samples will be the normal average. If the losses happen to be in the last warm-up sample some of the warm-up speed will be “transferred” to the first result sample. So the global reported speed will be too high. If the losses happen to be in the last sample, and we stop before receiving the duplicates, the average of the whole test will be lower.

      Protocol selection

      The test is protocol agnostic. By default it will select the first address getaddrinfo() returns (based on RFC6724 in uClibc and glibc). To force the use of IPv4 or IPv6 use the –protocol option.

      Examples of usage

      Download

      3 connections, 6 MiB max transfer (+512 KiB warmup)

      ./new_http_test --node 192.168.1.9 --service 8080 --path /1000MB.bin --mode GET --connections 3 --aggregate-warmup --warmup-sample-time 60000 --num-warmup-samples 1 --max-warmup-samples 1 --max-warmup-size $((512 * 1024)) --no-cache-control --result-num-intervals 1 --results-interval-period 300 --transfer-max-size $((6 * 1024 * 1024)) --cubic --warmup-dont-fail-on-max
      HTTPGETMT.3;1381740757;OK;192.168.1.9;192.168.1.9;70472;6421880;91126689;91126689;8102;712473;0;3;0
      

      3 connections, 12 MiB max transfer (+512 KiB warmup)

      ./new_http_test --node 192.168.1.9 --service 8080 --path /1000MB.bin --mode GET --connections 3 --aggregate-warmup --warmup-sample-time 60000 --num-warmup-samples 1 --max-warmup-samples 1 --max-warmup-size $((512 * 1024)) --no-cache-control --result-num-intervals 1 --results-interval-period 300 --transfer-max-size $((12 * 1024 * 1024)) --cubic --warmup-dont-fail-on-max
      HTTPGETMT.3;1381741126;OK;192.168.1.9;192.168.1.9;133524;12800320;95865313;95865313;6711;586497;0;3;0
      

      3 connections, 10 sec max duration (+between 1.5 sec and 5 sec warmup), the warmup checks three 500 ms samples (base speed plus two more with a 15% tolerance)

      ./new_http_test --node 192.168.1.9 --service 8080 --path /1000MB.bin --mode GET --connections 3 --aggregate-warmup --warmup-sample-time 500 --num-warmup-samples 3 --max-warmup-samples 10 --warmup-tolerance 15 --no-cache-control --result-num-intervals 1 --results-interval-period 10 --cubic --warmup-dont-fail-on-max
      HTTPGETMT.3;1381741272;OK;192.168.1.9;192.168.1.9;10000474;1011756696;101170874;101170874;2502346;248623105;0;3;0
      

      3 connections, 30 seconds split in 6 intervals (+between 1.5 sec and 5 sec warmup), the warmup checks three 500 ms samples (base speed plus two more with a 15% tolerance)

      ./new_http_test --node 192.168.1.9 --service 8080 --path /1000MB.bin --mode GET --connections 3 --aggregate-warmup --warmup-sample-time 500 --num-warmup-samples 3 --max-warmup-samples 10 --warmup-tolerance 15 --no-cache-control --result-num-intervals 6 --results-interval-period 5 --cubic --warmup-dont-fail-on-max
      HTTPGETMT.3;1381997771;OK;192.168.1.9;192.168.1.9;5000774;508993720;101782988;101782988;2500730;249740961;0;3;0
      HTTPGETMT.3;1381997771;OK;192.168.1.9;192.168.1.9;10002108;1020540264;102032518;102282020;2500730;249740961;1;3;0
      HTTPGETMT.3;1381997771;OK;192.168.1.9;192.168.1.9;15000373;1535440376;102360146;103015789;2500730;249740961;2;3;0
      HTTPGETMT.3;1381997771;OK;192.168.1.9;192.168.1.9;20001675;2051063040;102544564;103097686;2500730;249740961;3;3;0
      HTTPGETMT.3;1381997771;OK;192.168.1.9;192.168.1.9;25001667;2565014712;102593748;102790519;2500730;249740961;4;3;0
      HTTPGETMT.3;1381997771;OK;192.168.1.9;192.168.1.9;30000555;3077209960;102571768;102461837;2500730;249740961;5;3;0
      

      Upload

      3 connections, 3 MiB max transfer (+512 KiB warmup)

      ./new_http_test --node 192.168.1.9 --service 8080 --path /1000MB.bin --mode POST --connections 3 --aggregate-warmup --warmup-sample-time 60000 --num-warmup-samples 1 --max-warmup-samples 1 --max-warmup-size $((512 * 1024)) --no-cache-control --result-num-intervals 1 --results-interval-period 300 --transfer-max-size $((3 * 1024 * 1024)) --cubic`<span style="color:rgb(0,96,0);font-family:monospace"> --warmup-dont-fail-on-max
      HTTPPOSTMT.3;1381741392;OK;192.168.1.9;192.168.1.9;40720;4754552;116762083;116762083;5788;676744;0;3;0
      

      3 connections, 6 MiB max transfer (+512 KiB warmup)

      ./new_http_test --node 192.168.1.9 --service 8080 --path /1000MB.bin --mode POST --connections 3 --aggregate-warmup --warmup-sample-time 60000 --num-warmup-samples 1 --max-warmup-samples 1 --max-warmup-size $((512 * 1024)) --no-cache-control --result-num-intervals 1 --results-interval-period 300 --transfer-max-size $((6 * 1024 * 1024)) --cubic`<span style="color:rgb(0,96,0);font-family:monospace"> --warmup-dont-fail-on-max
      HTTPPOSTMT.3;1381741439;OK;192.168.1.9;192.168.1.9;66804;7734864;115784444;115784444;6045;616904;0;3;0
      

      3 connections, 10 sec max duration (+between 1.5 sec and 5 sec warmup), the warmup checks three 500 ms samples (base speed plus two more with a 15% tolerance)

      ./new_http_test --node 192.168.1.9 --service 8080 --path /1000MB.bin --mode POST --connections 3 --aggregate-warmup --warmup-sample-time 500 --num-warmup-samples 3 --max-warmup-samples 10 --warmup-tolerance 15 --no-cache-control --result-num-intervals 1 --results-interval-period 10 --cubic`<span style="color:rgb(0,96,0);font-family:monospace"> --warmup-dont-fail-on-max
      HTTPPOSTMT.3;1381741497;OK;192.168.1.9;192.168.1.9;10024271;1140527104;113776563;113776563;2063472;238022656;0;3;0
      

      Output Format

      The http_test client reports one or more lines to stdout in the following format once the test is complete:

      HTTP<TYPE>.3;<Unix Timestamp>;<OK|FAIL>;<Server_Hostname>;<Server_address>;<Duration usec>;<Bytes Transferred>;<Speed Bytes/sec>;<SpeedAtInterval>;<WarmupBytes>;<WarmupTime>;<IntervalNumber(Sequence)>;<Threads>;<NumRetrans>
      

      These fields have the following definitions:

      Number Field Type Notes
      1 Test Type String The possible values are:
      • GET - download, single thread, IPv4
      • GETMT - download, multi thread, IPv4
      • POST - upload, single thread, IPv4
      • POSTMT - upload, single thread, IPv4
      • GET6 - download, single thread, IPv6
      • GETMT6 - download, multi thread, IPv6
      • POST6 - upload, single thread, IPv6
      • POSTMT6 - upload, single thread, IPv6
      2 Unix Timestamp Integer Epoch in seconds
      3 Test result OK / FAIL
      4 Server Hostname String Name of the server
      5 Server Address IPv4/IPv6 Resolved IP address
      6 Duration Integer Duration of the measurement period in microseconds
      7 Bytes transferred Integer Bytes transferred during the measurement period
      8 Speed Integer Bytes transferred / Duration in bytes/sec
      9 Interval Speed Integer Speed during the last interval
      10 WarmUp Bytes Integer Bytes transferred during the warmup period
      11 WarmUp Time Integer Duration of the warmup period in microseconds
      12 Interval Number Integer
      13 Number of Threads Integer Number of parallel connections
      14 Numer of Retransmissions Integer Number of retransmitted packets
  • Latency/Loss Client (jitter_client)
    • The test measures round-trip latency, one-way jitter and one-way packet loss over a UDP connection between this client and a measurement server.

      The test starts with the client establishing a TCP connection and sending the test parameters to the server. The parameters include the UDP packet stream rate, the number of packets to be transmitted, whether the test is bidirectional, the packet size, the inter-packet interval at which jitter is calculated and the percentile over which the jitter calculation is made.

      The server responds with the UDP port number it is listening on. The client then sends the UDP packets at the given rate (the server does the same for bidirectional tests).

      The receivers timestamp the UDP packets and calculate the jitter values.

      At regular ping intervals, the client sets an ‘echo flag’ which cause the server to acknowledge some UDP packets. The echo packets are used to calculate RTT values. A specially flagged packet will signal the end of the UDP transmissions to the server. At this point, the server sends its statistics to the client over TCP. Then the client calculates the MOS values (Pingman formula) and outputs the results.

      Command line options

         <-u|--server> <Server hostname>
         <-p|--serverport> <Server TCP port>
         [<-j|--jitter_interval_packets> <Jitter interval packets> <Default value: 100>]
         [<-l|--percentile> <Percentile> <Default value: 99>]
         [<-B|--bidirectional> <Bidirectional> <Default value: single direction>]
         [<-d|--debug_level> <Debug level> <Default value: disabled>]
         [<-h|--help> <Print this message>]
         [<-k|--packets> <Number of UDP packets> <Default value: 500>]
         [<-z|--packet_size> <UDP packet size> <Default value: 160>]
         [<-r|--stream_rate> <Packet stream rate> <Default value: 64K>]]
         [<-t|--ping_interval> <Ping_interval (us)> <Default value: 50000>]
         [<-i|--host_interface> <Network interface> <Default value: default host interface>]
         [<-4|--ipv4_flag> <Do not use ipv6 socket> <Default value: from getaddrinfo>]
         [<-6|--ipv6_flag> <Only use ipv6 (AF_INET6) socket> <Default value: from getaddrinfo>]
         [<-V|--version> <Print program version>]
      

      Suggested execution command:

      For latency and bidirectional testing

      ./jitter_client -B -u server_hostname -p server_tcp_port
      

      Protocol selection

      The test is protocol agnostic. By default it will select the first address getaddrinfo() returns (based on RFC6724 in uClibc and glibc). To force the use of IPv4 or IPv6 use the –ipv4_flag or –ipv6_flag option.

      Examples of usage

      Latency and bidirectional jitter with default options

      ./jitter_client -u n1-the1.samknows.com -p 6003 -B
      UDPJITTER.3;1477324588;OK;n1-the1.samknows.com;46.17.56.234;160;64000;10049656;500;500;500;500;245;473;4539;4.40;0;NO_ERROR
      

      Latency and bidirectional jitter at 1 Mbps transmission rate

      ./jitter_client -u n1-the1.samknows.com -p 6003 -B -r 1M
      UDPJITTER.3;1477324710;OK;n1-the1.samknows.com;46.17.56.234;160;1000000;678727;500;500;500;500;219;369;4832;4.40;0;NO_ERROR
      

      Latency and bidirectional jitter over 1000 packets

      ./jitter_client -u n1-the1.samknows.com -p 6003 -B -k 1000
      UDPJITTER.3;1477324884;OK;n1-the1.samknows.com;46.17.56.234;160;64000;20077984;1000;1000;1000;1000;305;391;4589;4.40;0;NO_ERROR
      

      Bidirectional jitter with latency packets every 10 ms

      ./jitter_client -u n1-the1.samknows.com -p 6003 -B -t 10000
      UDPJITTER.3;1477325004;OK;n1-the1.samknows.com;46.17.56.234;160;64000;10038874;500;500;500;500;233;301;4389;4.40;0;NO_ERROR
      

      Upstream jitter at 100 Kbps transmission rate

      ./jitter_client -u n1-the1.samknows.com -p 6003 -r 100K
      UDPJITTER.3;1477325741;OK;n1-the1.samknows.com;46.17.56.234;160;100000;6450228;500;500;500;499;285;3182;7398;4.39;0;NO_ERROR
      

      Latency and bidirectional jitter with IPv6

      ./jitter_client -u v6-n1-the1.samknows.com -p 6003 -B -6
      UDPJITTER6.3;1477325879;OK;v6-n1-the1.samknows.com;2a01:348:0:21::2;160;64000;10057523;500;500;500;500;654;345;19529;4.39;0;NO_ERROR
      

      Output Format

      The client reports one or more lines to stdout in the following format once the test is complete:

      UDPJITTER.3;<Unix Timestamp>;<OK|FAIL>;<Server hostname>;<Server IP>;<Packet size>;<Stream rate>;<Duration>;<Tx upstream>;<Tx downstream>;<Rx upstream>;<Rx downstream>;<Up jitter>;<Down jitter>;<Latency>;<MOS>;<Error code>;<Error code label>
      

      These fields have the following definitions:

      Number Field Type Notes
      1 Unix Timestamp Integer Epoch in seconds
      2 Test result OK / FAIL
      3 Server hostname String Name of the server
      4 Server IP IPv4/IPv6 Resolved IP address
      5 Packet size Integer UDP packet size
      6 Stream rate String Transmission rate
      7 Duration Integer Duration of the measurement period in microseconds
      8 Tx upstream Integer Number of sent packets upstream
      9 Tx downstream Integer Number of sent packets downstream
      10 Rx upstream Integer Number of received packets upstream
      11 Rx downstream Integer Number of received packets downstream
      12 Up jitter Integer Upstream jitter in us
      13 Down jitter Integer Downstream jitter in us
      14 Latency Integer RTT in us
      15 MOS Integer Mean Opinion Score
      16 Error code Integer Error code number such as 0 for success
      17 Error code label String Error code label such as NO_ERROR for success
 

Find out what we can do for you

 

To discuss your measurement requirements and what we can do to help you please get in touch.