Alibaba Group Holding’s cloud computing services subsidiary has described the system breakdown it experienced on December 18 as “the longest major-scale failure” at its Hong Kong and Macau operations for more than a decade, following a review of the incident.
“We want to apologise to all of the customers affected by this event, and we will handle the compensation as soon as we can,” Alibaba Cloud said in a statement published on its website on Sunday. “This event has caused a severe impact on the business of our customers.”
The incident, which happened at the internet data centre of an Alibaba Cloud partner, resulted in a lengthy service outage that stretched for more than 24 hours at some customer sites. It suspended withdrawals at major cryptocurrency exchange OKX and disabled the website of the Monetary Authority of Macau.
Alibaba Cloud blamed the system failure to “a malfunction of the data centre’s chillers”. It said the problem was detected at 8.56am on December 18, but engineers were not able to fix the problem expeditiously.
The engineers were alerted of “rising corridor temperatures in [the partner facility’s] data centre rooms in Zone C of the China (Hong Kong) region”, Alibaba Cloud said in the statement. “At [2.47pm], the fire sprinkler system of a room was triggered automatically due to high temperature, which increased the difficulty in troubleshooting.”
It took the engineers nearly 11 hours before the engine room’s temperature became stable enough for them to start service recovery.
“At [10.50pm], the data security check and risk assessment were completed,” Alibaba Cloud said. “Then, the power supply was restored to the last room and all servers were successfully restarted.”
The outage affected the use of various Alibaba Cloud offerings, including its elastic compute service, database, storage and cloud network products at the “Zone C” set-up covering Hong Kong and Macau. Other services covered by other zones in the China (Hong Kong) region were not affected by the incident, according to the company.
Alibaba Cloud’s system failure reflects how internet infrastructure remains vulnerable to the occasional breakdown even as a growing number of companies in the industrial, manufacturing, financial services, utilities and other sectors shift their information technology workloads to the cloud.
Cloud computing services enable companies to buy, sell, lease or distribute a range of software and other digital resources as an on-demand service over the internet, just like electricity from a power grid. These resources are managed inside data centres.
Alibaba Cloud has experienced outages before. In March 2019, a breakdown at one of the company’s facilities in northern China stretched for about six hours. Another incident in June 2018 caused certain websites and apps to go offline for about an hour.
As part of Alibaba, the company is considered a major growth driver for the e-commerce giant, which reported a profit of 1.1 billion yuan (US$164 million) from its cloud business in the 12 months ended March 31 – the subsidiary’s first profit since 2009. Alibaba owns the South China Morning Post.
Still, the cloud unit has seen its revenue growth slow down in recent months. Its sales rose by just 4 per cent in the September quarter, down from the 12 per cent and 10 per cent growth recorded in the June and March quarters, respectively.
