Game and shame: Internet providers called out again on NBN speed claims

Internet providers are under renewed pressure to scale back their advertised “typical” evening speed claims for NBN services following the release of a new metric that shames alleged under-performers.

Go slow, perform better?

The fifth instalment of the Australian Competition and Consumer Commission’s (ACCC) NBN speed monitoring report [pdf], released a fortnight ago, included a new chart showing the “proportion of busy hours where advertised speed was achieved” for each major retail service provider (RSP).

TPG was shown to meet its advertised busy hour speeds 83.4 percent of the time, Telstra 76.3 percent of the time and Optus 74.6 percent of the time.

However, the chart produced two surprises.

First, it showed Aussie Broadband users in the ACCC speedtest sample hit advertised evening peak speeds just 25.9 percent of the time. MyRepublic also fared relatively poorly on the measure, achieving 37.1 percent.

Those numbers came with little context from the ACCC, particularly to explain why two sizable RSPs could allegedly under-perform their rivals by such a large margin.

Following an investigation by iTnews, there are several possible explanations.

The first explanation - and indeed the one the ACCC is pushing - is that the measure exposes truth in NBN advertising.

Certainly, some of the typical busy hour claims made by RSPs on their websites are hard to correlate with reality.

iTnews, together with comparison sites, have been chronicling these speed claims and their movements over the past couple of years.

There is typically no explanation from RSPs about how the numbers were reached and to what extent they represent the common user experience on the network.

So the ACCC’s introduction of a standardised measure on RSP speed claims could reduce any residual uncertainty in NBN advertising.

“This additional metric was introduced to further encourage RSPs to align their typical busy hour speed claims with what their plans deliver on average when their networks are under most stress,” an ACCC spokesperson told iTnews.

“An RSP that makes its claims on this basis will achieve a result that is at or close to 100 percent.”

To bundle or not to bundle in the bad services?

Some of the opacity in claimed NBN speeds comes from whether or not the results of so-called under-performing services are included or excluded from the advertised number.

There remains a not insignificant number of mostly fibre-to-the-node (FTTN) services that have poor line speeds.

The performance of these services drags down the average.

A customer whose line is capable of 65Mbps might still buy a 100Mbps retail plan to try and get every last megabit per second out of their line.

However, the service is technically incapable of achieving even a baseline typical evening speed, and the customer knows this when their plan is finalised (courtesy of negotiated standard operating procedures between the ACCC and RSPs).

Should a known under-performing service be included in the bundle of speedtest results that make up an RSP’s advertised claim, thus making their services seem slower than they actually are for people with regularly functioning lines? Or can the result be safely excluded?

The new ACCC measure shows how much underperforming services can weigh down an RSP’s overall performance.

“When underperforming services are excluded, average plan speeds met or exceeded advertised speed claims in over 92 percent of busy hours for all RSPs except Aussie Broadband and Exetel where speed claims were met in 67 percent and 84 percent of busy hours,” the ACCC said in its report.

In other words, Telstra users see advertised evening speeds 76.3 percent of the time overall, or over 92 percent of the time when the poorly-performing services are excluded.

Optus users see the advertised speeds 74.6 percent of the time - or over 92 percent of the time if they have regularly performing lines.

If nothing else, this illustrates the temptation to present the best possible picture of NBN performance out to the market, particularly during a period where there is still competition for new sign-ups.

Aiming low

Of course, setting up a standardised calculation only works if it can’t be gamed. Unfortunately, this cannot be ruled out.

It isn’t described in the ACCC’s report, but iTnews has confirmed how the “proportion of busy hours where advertised speed was achieved” metric is put together.

It’s a four-step process:

• Identify the relevant speed claims for each RSP’s 50Mbps and 100Mbps plans. These numbers are drawn from the NBN fact sheets that RSPs publish under Australian Communications and Media Authority (ACMA) rules.
• Convert that into a percentage e.g. if an RSP markets a typical busy hour speed claim of 80Mbps for a 100Mbps plan, that translates as an 80 percent of maximum speed claim.
• This average maximum speed claim is then compared to the average plan speed observed in each of the busy hours in the sample period of the ACCC report. The same period in this instance is February 2019.
• The result for each busy hour is based on a binary analysis: if your average speed in the hour exceeded the claim, you get a 1; if it fell short, you get a 0. The results for the 50Mbps and 100Mbps plans are then combined and converted to a single percentage based on the number of busy hours in the test period.

Knowing that the metric uses the NBN fact sheet speeds as a baseline is important. These speeds vary widely.

Some RSPs, like Telstra and Optus, market a very basic typical evening peak speed based on old ACCC guidance. That means 40Mbps for a 50Mbps plan, and 80Mbps for a 100Mbps plan.

These standardised speeds came about as a result of earlier efforts by the ACCC to set some baseline claims around NBN speeds as part of a truth in marketing effort.

Others, like Aussie Broadband and MyRepublic, advertise performance above these basic numbers. MyRepublic advertises 43Mbps on NBN50 and 83Mbps on NBN100 plans; Aussie Broadband claims 45Mbps on NBN50 and 90Mbps on NBN100.

Telstra and Optus do reasonably well on the ACCC’s “proportion of busy hours where advertised speed was achieved” metric; MyRepublic and Aussie Broadband do not.

The simplest explanation is that Aussie Broadband and MyRepublic set themselves a very high bar in their advertising to match up to, by relying on “actual” speeds instead of a standardised minimum number.

RSPs that market with lower speed numbers will naturally do better in the ACCC test because their bar for success or failure is set much lower.

As an RSP, an uncontrolled sample of your customers  - which is what the ACCC’s speed monitoring program amounts to - have a higher chance of meeting or exceeding 80Mbps in the evening peak than 90Mbps.

Therefore, if the ACCC’s new “proportion of busy hours where advertised speed was achieved” metric catches on, one might expect RSPs that currently market high - “actual” - evening speeds to scale back their claims to look better on the ACCC metric.

It is certainly safer for the RSP, since they would instantly appear to have better performance.

It would come down to whether the ACCC measure becomes as influential in impacting sign-ups as the advertised measure on the RSP’s website.

The ACCC acknowledged that advertising low speeds meant a better mark on its test.

However, it implored consumers not to consider the new metric in isolation, and to look at all the different performance measures in the ACCC report.

“Reducing a speed claim to an artificially low level would be one means to secure full marks on this measure, but an RSP who chose to do this would never score more than 100 percent on this metric or gain a lift on its headline download speed metrics,” the spokesperson said.

“Consumers should consider all the speed metrics we include in the reports in deciding which RSP best suits their needs.

“An RSP could be delivering high speeds to its NBN customers as shown in our headline speed metrics, even though it does poorly in this new metric due to setting an overly ambitious advertised speed.”

The spokesperson added that if the ACCC “thought RSPs were trying to game this particular metric in this way then we would reconsider it”.

“But we haven’t seen it to date.”

TPG set to challenge the theory

The ACCC may have some support in this regard.

TPG - whose users experience the highest “proportion of busy hours where advertised speed was achieved” - has recently raised its advertised speeds.

It’s worth noting that at the time of the ACCC speed monitoring test - February 2019 - TPG advertised NBN50 plans at 42.7Mbps and NBN100 plans at 71.4Mbps.

While the NBN50 plan speed is more or less on par with others RSPs, the NBN100 speed is very low - in fact, the lowest of any of the major RSPs servicing 100Mbps users.

This is likely to be a factor in TPG being able to top the “proportion of busy hours where advertised speed was achieved” measure.

It’s worth noting here that any RSPs that performed well on the ACCC metric because they had a low advertised speed were simply in the right place at the right time.

The ACCC runs the broadband monitoring scheme independently of the RSPs. They don’t get any advance warning on what is going to be in there, or even of their results - even when the results cast the RSP in a significantly bad light.

RSPs had no advance warning that the ACCC would introduce the “proportion of busy hours where advertised speed was achieved” measure in last week’s report, therefore could not have done anything to prepare for it.

What RSPs do now that they know about the measure and how it is calculated is an open question.

TPG has recently raised significantly the typical evening speeds in its 100Mbps plan advertising, from 71.4 Mbps to 88.1Mbps.

That would appear to indicate it is confident that the sample of users in the ACCC test can hit these much higher speeds with frequency.

Aussie Broadband’s CTO John Reisinger told iTnews his company is still exploring the new measure and what it means, given the wide disparity with Aussie’s own numbers and understanding of network performance.

“We haven’t really had an opportunity to talk that through,” Reisinger said.

“There may be merit to bringing [the advertised speed number down] to what the ACCC considers realistic versus what we consider realistic.

“Either way, though, we would still be basing it on real data rather than picking a number because it fits.”

A MyRepublic spokesperson could not be reached for comment.

An aside on outage stats

The May 2019 ACCC speed monitoring report contains a second new measure, charting the frequency of daily outages per RSP.

Outages are only tracked and included if the duration is “over 30 seconds”.

“The 30 second benchmark is also used by UK telecommunications regulator, Ofcom,” an ACCC spokesperson told iTnews.

“The ACCC considers that this is a reasonable threshold for when consumers may be noticeably impacted by an outage.”

Optus is by far the worst afflicted, according to the inaugural set of numbers.

But this information by itself is of unknown utility; it encompasses outages ranging from vaguely annoying to productivity-killing, and there’s no context on the severity of the outages being experienced on a per-RSP basis.

Like Aussie Broadband on the peak performance measure, Optus is still working out what to do about the result.

“This is a new data point in the ACCC reporting and we are assessing its implications,” an Optus spokesperson said.  “Optus is committed to improving customer service and experience.”