Google search and cloud services were unavailable for many customers on Monday after an incident in which the company's routing information was leaked, via a Nigerian service provider and China Telecom. All those involved say the incident was due to an accidental misconfiguration of the border gateway protocol (BGP), which exchanges routing information amongst internet nodes.
The problem started at MainOne in Nigeria, and was rapidly followed by a similar leak of routing information for content delivery and DDoS mitigation provider CloudFlare, with similar results. IP addresses belonging to both companies (totaling 180, according to ThousandEyes) were rerouted via China Telecom's infrastructure - rousing suspicions of foul play, as the provider is known for its affiliations with the Chinese government.
Whoopsidaisies
from Youtube
BGPmon, which assesses BGP routing information in real time, discovered that the IP ranges that were redirected contained highly sensitive information belonging to Google, including its VPN and its own corporate WAN infrastructure, reports Arstechnica. After MainOne advertized the addresses, they were picked up by China Telecom and then by Russian service providers.
Shortly thereafter, CloudFlare's IP addresses were leaked in a similar fashion.
According to CloudFlare CEO Matthew Prince, MainOne hosted a meeting of the Nigeria Network Operators' Group (ngNOG) last week during which new peerings were established, to build fresh network connections.
Prince says the Nigerian telecommunications provider inadvertently leaked the information by improperly declaring the wrong route to Google IP prefixes. China Telecom improperly accepted the routes, and announced them worldwide, with Russian provider Transtelecom accepting the route, among others.
Despite reassurances from CloudFlare, Google and MainOne that the incident was accidental, the Chinese telecommunications' provider's connections to the Chinese government caused some alarm, because the provider is known for many incidences of corporate surveillance, and because of a recent case of traffic diversion.
For thirty months, between 2015 and 2017, large amounts of US Internet traffic was routed via a China Telecom data center in Hangzhou, according to ArsTechnica. It is not clear whether this was malicious or the results of accidental mishandling of BGP by autonomous systems belonging to Verizon's Asia Pacific and China Telecom.
That incident went undetected for months, because the traffic all reached its destination. In Monday's event, however, the result was lost connections, because the traffic terminated at an edge router within China Telecom. The obviously lost traffic is evidence this was accidental rather than malicious Prince told Ars: "If there was something nefarious afoot there would have been a lot more direct, and potentially less disruptive/detectable, ways to reroute traffic. This was a big, ugly screw up. Intentional route leaks we’ve seen to do things like steal cryptocurrency are typically far more targeted."
Additionally, according to MainOne, both CloudFlare and Google are present on the Nigerian Internet Exchange (NIX), which would explain why both providers were affected.
BGP is one of the foundations of the Internet and is also one of its core weaknesses, according to Prince and others. Routes are advertised and accepted on trust, while many network operators are arguing for a cryptographically signed alternative
Update: Responding to a comment request from DCD, a MainOne representative issued the following statement:
"In the early hours of Tuesday, MainOne experienced an administrative error in the configuration of our edge router during a planned network upgrade which impacted traffic to our internet peering partners.
Specifically, the configuration on our Border Gateway Protocol (BGP) filters led to the inadvertent advertisement of Google prefixes through one of our upstream partners, China Telecoms. This leak majorly impacted traffic to Google because the return traffic for the leaked prefixes was dropped by ChinaTel. Therefore, during this period, some Google services were not accessible to some of its customers who preferred to use the leaked routes.
These routes were immediately shut down by MainOne as soon as we discovered the error.
It is important to note the BGP technology that routes Internet traffic from systems around the globe is predicated on trust within peering partners and encourages route sharing. While some providers have implemented some filtering on their networks to prevent the re-advertisement of routes that should not be propagated, implementation across networks differ, as evidenced in this instance.
In a statement, Réseaux IP Européens Network Coordination Centre (RIPE NCC), a non-profit organization responsible for allocating and registering IP numbers to ISPs in Europe has weighed in on this issue and concluded it was purely an error in network reconfiguration. "These reconfigurations happen every day in global routing and mistakes can happen, especially since the configuration of routers is error-prone and often still requires manual input which is prone to fat fingers”.
In order to prevent a re-occurrence, we have reviewed our supervisory and configuration policies to ensure these kinds of errors are avoided in the future.
MainOne appreciates our partners for their patience and cooperation through this brief period of internet disruption and we assure you of our continued commitment to the high degree of trust that makes the internet work efficiently."
