How we test

Transparency

TrueNet Performance Measurements use equipment installed between the home or business modem/router and their network of computers to test to fixed TrueNet owned servers.  Tests are completed hourly using data allocated to TrueNet by the volunteer Panelist, the amount is controlled by the Panelist at any time.  Panelists get access to charts of their results.

At the bottom of the page are the installation instructions and some instructions on how to turn on Wifi once a Panelist has a probe installed.  

For the technically minded, the probe is a "Bridged Router" with software control managed by the database - this is why the Wifi must be turned on from our website.

This page is intended to be a transparent description of our methodology and techniques, we welcome discussion on the contents.

How the results are obtained

TrueNet measures Broadband performance from volunteer sites in people's homes and businesses (Panelists).  TrueNet provides one testing device per Panelist.
 
Our probes are simply a bridged router which are connected between the Local network and the Internet, with all Wifi traffic going through the probe.  We offer the probes with an extended 802.11n Wifi as a service to encourage this.  Faster technologies such as VDSL, Cable and Fibre are provided with long-reach wifi and the fastest technologies (over 50Mb/s are provided with GigE probes
For the technically minded:

  • The probe requests an IP address from your router/modem's DHCP
  • Being a bridged router, it passes all DHCP requests, files etc as though it were not there.
  • Before a test run, the probe checks if you're using the Internet, if you are using more than 100kb/s then it waits or pauses until you've finished.  100kb/s is quite slow, faster than a voice call so VoIP is ignored, but less than most uses.
  • While performing a test, the probe detects if you start using the Internet, when it detects this it pauses any tests it is running and waits until you finish.  There may be a short delay between you using the Internet and the probe detecting this.
  • The probe will attempt to detect the brand/model name of your router or ADSL modem to enable future reporting comparing modems - if we get enough reliable details.  This may appear in the security logs on your device, but we never try and login to your router or ADSL modem.
  • We do not view or save any data passing through your network.
  • The Wifi login enables your own SSID (network name) and password from our website, but because the probe only calls home once an hour to receive configuration settings from the TrueNet server, this could take an hour to change, although restarting your TrueNet probe will load the settings immediately.
  • We prefer you use our Wifi to ensure that traffic is included in its detection routine and tests are avoided while you're using the Wifi. If you use the Wifi on your router or ADSL modem then the TrueNet probe is unable to detect this traffic.
  • VoIP is unlikely to have any impact, either through traffic levels or on call quality.

Measuring from the LAN and identifying traffic before and during tests avoids interference from the home testers computer or network activities.  It also enables some very useful attributes, such as the ability to change tests each time the probe initiates a new set of tests, usually every hour.

Why do we test this way?

We reviewed alternative methods of testing and found the probe based method most reliable for comparison of results because it has the least variables.  Other testing methods rely on software or PC based probes, where the PC can be a portable machine, testing via it's professed ADSL line while actually testing via wifi, bluetooth, mobile or even a fibre connection in "the office".  The use of common locations for probes presents different challenges but much smaller.

We know where our probes are, we know what software contention they have and we know the route from the probe to the ISP, reducing variables by a significant margin.  We also use the volunteer's own modem, ensuring that ISP supplied modems are a factor in the results.

The probe used sits between the testers modem and the home network (LAN), testing only when the network is quiet and ceasing testing if network activity begins to avoid interference between the two functions.  Streaming video does not necessarily stop all testing; at say 30Mb/s available speed, a video plays at about 5Mb/s, leaving 25Mb/s, or almost 80% of the time, available for a test which may take less time than the gap between streaming sections.

The probe runs a suite of tests every hour to measure the performance of the connection, immediately sending the results to our servers.  
 

What We Test

Our testing is based on simple concepts that mirror users experience to the maximum extent possible, eg we use a test website that is the weighted average size of the top ten NZ Hitwise sites.   We download a random file that behaves like a typical file download, say a data, music or movie file.

The tests we run include:

  1. Latency using the Ping command - measures the time it takes for the request to go to a selected network node and back
  2. Jitter - is the standard deviation of the variation of latency ie the measurement of many pings to calculate the variations in speed.
  3. Packet loss - loss of packets in a transmitted file ( a packet is the smallest amount of data that is sent over the internet for a specific transmission)
  4. DNS time - the time taken to find an IP address at the ISPs own domain name server (DNS). We use the best of each pair of DNS server tests.
  5. File downloads using the CURL command - these include:
    • File downloads to test Throughput. eg the download speed of software updates or other commonly used files from the following sites
      • ICONZ, Auckland, peered over GigE to APE
      • ACSData, Wellington, peered over GigE to WIX
    • The same file uploaded to test upload speed (opposite direction to 1) eg saving a file on a work server, or sending an email.  This is done to Catalyst Wellington or Sydney, depending on the connection country location
    • Similar random files downloaded from our International sites measuring non-cached download - Dallas and Sydney
    • Other public files, eg pictures from various sites
  6. Video - we download a video from selected sites (say youtube).  This is done in parts with gaps so that the average download speed is about that of a streamed video in 4K or HD.  The size of the gap depends on the speed of your service and we measure how often buffering would occur if you were watching this video - along with some other stats about startup etc.

Also, URL Downloads using the WGET command, these include;

  • Various live websites that are popular in NZ
  • Australian websites
  • USA Websites

Installation Instructions

How to connect our probes is explained in the following files downloaded by clicking on the names below.  The first two are for the older D-Link probes, the update is for the newer MikroTik Probes:

Truenet Probe Installation and Instructions 

Updated 20th December 2016

Subscription

Subscribe to our mailing list

* indicates required



Email Format

View previous campaigns.