ABS apologises for AU$30m Census bungle, ISPs say IBM had standard DDoS protections disabled

Head of the Australian Bureau of Statistics (ABS) David Kalisch has conceded that the agency tested the patience of Australians with its decision to take down the Census website on the night of August 9.

"We made a difficult decision to take the Census system offline on 9 August to ensure the security of Census data, but we should not have got to that point, and the system should have been robust to DDoS events," Kalisch told Senate Estimates on Wednesday night.

"I apologise to the community on behalf of the ABS."

Under questioning, Kalisch revealed it had cost the agency AU$30 million to fix its systems.

"We did spend a little bit more money than we were expecting as a result of some of the remedial activities we took terms of the Census," he said. "We have, to date, probably incurred additional costs of around AU$20 million; these are rough figures, and we anticipate possibly spending another AU$10 million."

The ABS said 58 percent of Australian households had completed Census forms online, with the agency receiving a total of 4.9 million online forms and 3.5 million paper forms. The agency admitted this was shy of its 65 percent target for the online form.

The agency said it has thus far sent out 1,800 refusal letters, and 239 direction notices to complete the Census. Approximately 10,000 Australians have refused to complete the Census, which is down on the 13,000 refusals for the 2011 survey.

Next week, hearings for the Census Inquiry by the Senate Standing Committee on Economics are set to begin.

In a submission to the inquiry last month, ABS said IBM failed to adequately address the risk posed to the Census systems it was under contract to provide.

"The online Census system was hosted by IBM under contract to the ABS, and the DDoS attack should not have been able to disrupt the system," the ABS said. "Despite extensive planning and preparation by the ABS for the 2016 Census, this risk was not adequately addressed by IBM and the ABS will be more comprehensive in its management of risk in the future."

Kalisch revealed on Wednesday night that once the system was restored, it suffered another DDoS attack, but this time it was able to handle the increased traffic.

In its own submission to the inquiry, IBM pointed the finger at NextGen Networks and its upstream provider Vocus for not geoblocking traffic effectively.

"The attack was foreign-sourced and hit the eCensus site via the NextGen link at a time when IBM had already directed NextGen (and Telstra) that Island Australia [geoblocking non-Australian traffic] was to be in place and in circumstances where NextGen had provided repeated assurances to IBM prior to the attack that it had done so," Big Blue said.

"In fact, the assurances were incorrect. IBM was informed -- later that day after the attack had passed -- that a Singapore link operated by one of NextGen's upstream suppliers (Vocus Communications or Vocus) had not been closed off, and this was the route through which the attack traffic had entered the NextGen link to the eCensus site.

"Vocus admitted the error in a teleconference with IBM, NextGen, and Telstra around 11.00pm on 9 August 2016."

IBM said it accepts responsibility as head contractor for the Census site, and said its employees were the ones who had noticed monitoring data leaving the system and decided to pull the plug.

"Out of an abundance of caution, IBM shut down access to the site and assessed the situation. The cause of the problem was identified. No data exfiltration occurred," it said.

IBM said ABS asked the Australian Signals Directorate to review security for the Census site, but ASD declined to do a detailed review. The IT giant said it still believes that if the geoblocking had worked as intended, the Census site would have stayed up.

"Had Island Australia been properly implemented by Vocus, the fourth DDoS attack would have been prevented, and the site would not have become unavailable to the public as a result," it said.

In response, Nextgen said IBM refused its offer of DDoS protection for the Census, and said it was only told about the "Island Australia" strategy six days ahead of Census date.

"Although Nextgen strongly recommended to IBM to take up an internet DDoS protection option for the purposes of the 2016 Census, it was declined by IBM," Nextgen said.

Nextgen said IBM had tested its Island Australia arrangements on August 5, and gave it the tick of approval.

"After becoming aware of 'Island Australia', Nextgen advised IBM that the IP address range requested by IBM was part of a larger aggregate network, and therefore it was not possible to provide specific international routing restrictions for this range. Nextgen recommended using an alternative IP address range, which would give IBM better control, but this was rejected by IBM."

After the fourth DDoS attack of the night, Nextgen once again proposed implementing its DDoS protection, which it did on August 13.

In its submission, Vocus said the fourth DDoS attack peaked at 563Mbps, which it said is not considered significant by industry, and lasted 14 minutes.

"Such attacks would not usually bring down the Census website, which should have had relevant preparations in place to enable it to cater for the expected traffic from users as well as high likelihood of DDoS attacks," Vocus said.

"The Island Australia approach does not consider the reality of overseas network operators connecting to Australian service providers inside Australian borders. In fact, during the fourth DDoS attack, Vocus had blocked the vast majority of DDoS traffic, only passing on a small percentage of the total traffic from botnet hosts in Asia and Australia.

"Once Vocus was made aware of the fourth DDoS Attack, it implemented a static null route to block additional DDoS traffic at its international border routers within 15 minutes."

Vocus pointed out that it was requested to disable its DDoS protection for the Census IP range.

"If Vocus DDoS protection product was left in place, the eCensus website would have been appropriately shielded from DDoS attacks."

Despite the debacle of Census night, Kalisch said he still believes the online Census approach was the right one.

"My hope is that the 2016 Census experience will not reduce the public appetite for innovation across the broader public sector. The lesson for the ABS and for others is not to stop innovating, but to innovate more successfully," he said.

"Given some of the public reaction to the 2016 Census experience, and the need for the ABS to restore high levels of public trust in the Census process, planning for the 2021 Census will necessarily adopt a more rigorous approach.

"We will have the advantage of all the learnings from the new approach first adopted in the 2016 Census, and desirably have five clear years to plan and implement a successful 2021 Census."