The Butterfly Effect: How a BGP Leak Slowed Down the Internet

Incident Report

Why is my Ping 400ms?

In August 2024, gamers, sysadmins, and cloud engineers noticed something weird. It wasn't a total outage, but packet loss to US-East regions spiked to 20%, and latency doubled. It felt like the internet was running through molasses.

The culprit? BGP (Border Gateway Protocol). Again.

The Technical Root Cause: Route Leaking

The internet is held together by trust. ISPs tell each other, "I can send traffic to Google." A Route Leak happens when an ISP accidentally says, "I can send traffic to Google," but they actually can't handle the load, or worse, they send it in a circle.

In this incident, a Tier-1 provider misconfigured their Route Filters. They accidentally advertised a "more specific" prefix for a major backbone route.

Rule of the Internet: Routers always choose the most specific path (Longest Prefix Match).

So, instead of traffic flowing through the super-fast backbone:

1. Traffic was sucked into a smaller ISP's network.
2. That ISP tried to route it back to the Tier-1.
3. The Tier-1 sent it back to the smaller ISP.

This created a Routing Loop. Packets bounced back and forth until their TTL (Time To Live) expired.

Troubleshooting: Seeing the Loop

Network engineers diagnosing this saw the dreaded "Star Pattern" in traceroutes:

$ traceroute 8.8.8.8
 1  192.168.1.1 (0.4 ms)
 ...
 6  core-router.isp-A.net (20 ms)
 7  border-router.isp-B.net (22 ms)
 8  core-router.isp-A.net (25 ms) <--- LOOP!
 9  border-router.isp-B.net (26 ms)
10  * * * (Request Timed Out)

The Fix: RPKI and Filtering

The incident was resolved when the ISP flushed their BGP tables and applied correct route-map filters.

! Cisco IOS-XR Example of preventing leaks
router bgp 65000
 neighbor 192.0.2.1
  address-family ipv4
   route-policy REJECT-UNWANTED-TRANSIT out
  !
 !
!
route-policy REJECT-UNWANTED-TRANSIT
  if destination in PREFIX-LIST-INTERNAL then
    pass
  else
    drop
  endif
end-policy

The Takeaway: This is why we need RPKI (Resource Public Key Infrastructure) everywhere. RPKI allows routers to cryptographically verify that an ISP is allowed to advertise a route, preventing fat-finger errors from breaking the internet.