Vodafone Australia provisioned extra cloud infrastructure and set up new monitoring systems for Apple’s iPhone XS and XS Max launch to avoid a repeat of last year when its sites fell over due to the load.
Digital senior test manager Senthil Rajendran provided a rare and candid look inside telco preparations for an Apple product launch at last month’s FutureStack summit hosted by New Relic.
Rajendran said that the past two years of Apple launches had been “good … but not so great” for Vodafone Australia.
“Last year we had a problem with our platform for a few minutes,” he recalled.
“We went down for a few minutes and then we recovered, but it put a black mark on our launch.”
The digital team was keen not to have a repeat performance on what is a massive sales opportunity for all telcos.
“This time we wanted to make sure there is no outage, that there is 100 percent reliability on our site, and there is no impact to any customer experience,” Rajendran said.
“We did work seven to eight weeks prior to the launch and engaged with various vendor partners - New Relic, Splunk, Amazon - to build up our infrastructure.”
Vodafone put together a core internal project team “of around 10 to 20 people” and ran workshops with its vendor partners to determine the right architecture.
Rajendran also attempted to understand just how much capacity Vodafone would need to comfortably see out the peak in interest when the new iPhones launched on Friday September 14 at 5.01pm.
The company took various sales numbers from previous launches and combined them with estimates from Vodafone marketing to try to estimate the level of infrastructure required.
“We have to crunch through a lot of numbers,” Rajendran said.
“The marketing team always comes up with crazy numbers - double the numbers from last year.
“They always say last year we did 300,000, this time it’s going to be double that. We just have to sit them down and cool them down.”
Rajendran said the team applied a high discount to the marketing projections.
“They either say a high watermark or low watermark, but they don’t give you the optimum. So whatever numbers they were giving us, we’d plus or minus 50 to 70 percent,” he said.
With numbers set, Rajendran’s team worked with Amazon to scale up its cloud infrastructure and with New Relic to create a “full monitoring setup and dashboard.”
As in past years, Vodafone set up a bank of monitors in its office. This time, there were 14 monitors displaying statistics from Amazon, Splunk, New Relic and Adobe, which provided website data on what iPhone models people were buying.
Vodafone would get an early taste of the interest level in the XS and XS Max, by jumping the gun on the 5.01pm launch time mandated by Apple.
“Actually we launched two minutes early this time,” Rajendran said, confirming the telco had worn Apple’s wrath for going early.
This year, the Adobe stats dashboard crashed almost immediately.
“It went completely blank,” Rajendran recalled.
“So I had my GM of digital, GM of marketing, and 30 to 40 people on the floor turn around and start looking at the Amazon and New Relic monitoring and graphs [instead].
“It was a very pleasing experience when you see all the execs and GMs really happy looking at the [new] graphs.”
Rajendran’s team was still nervous, however.
“At the same time, my core team and I kept looking at the graphs as they kept going up and up. We were like ‘OK, drop. Hit the peak and then come back’.
“We had 200 to 250 hits in the first two minutes, and then it stabilised. There was a big sigh of relief. We sustained the initial hit of users.
“It was the smoothest release we’ve had. We didn’t have one error. All customers able to log into the site could purchase devices.”
Rajendran said that Vodafone also typically watched its telco rivals closely during Apple launches to attempt to gauge how they also handled the influx of prospective device purchasers.
“We had two laptops to check our competitors’ sites, and they didn’t do so well,” he claimed.
“We’ve got the bragging rights now until the next launch.”
Rajendran expects future Apple iPhone launches to go more smoothly because Vodafone now has a wealth of statistics it can use to properly provision its sites to handle the extra load.
“When we prepare for the next iPhone launch I just have to look at the New Relic stats for this year. That will give me how many users came and were serviced,” he said.
“This year we didn’t have anything like that because we didn’t have a monitoring setup so we struggled and did a lot of intelligent guesses and assumptions to come up with the numbers.”
Rajendran indicated that Vodafone Australia would now focus on additional digital transformation projects in the pipeline, now that a proper monitoring setup was sorted.
“Pretty soon we’ll be internally launching new software and also we’ll be going into a serverless architecture and microservices,” he said.
“There are a lot of challenging projects coming up.”