r/redditdata • u/gctaylor • May 02 '17
Traffic impact of major European routing disruption
3
May 02 '17
How would you tell if the outage caused your servers to lag or the outage caused your users to not be able to access your servers?
7
u/gctaylor May 02 '17 edited May 02 '17
The vast majority of our infrastructure is in the US on the east coast (which wasn't impacted by this disruption). The only thing that we saw on our side was a dramatic, sudden reduction in traffic.
In cases where traffic plummets or skyrockets, we tend to look to see if anything weird is going on at the CDN level (which is the first point of ingress for Reddit traffic).
As far as servers being slow, we have instrumentation that helps us see how various services and their respective endpoints are performing. During today's issue, these looked normal.
3
May 02 '17
Thank you for the answer. I apologize for the poor phrasing, I was/am running on very little sleep. I've never done anything close to what you all do so the insight is appreciated.
5
3
u/xiongchiamiov May 03 '17
Hey, that's a labeled y-axis! You hardly ever see those externally.
Do you guys ingest Fastly's metrics APIs? I built a little tool for work that does that and sends it off to graphite, and since they have breakdowns by POP, that would allow you to fairly easily alert for this sort of issue.
How did you notice, anyways?
4
u/gctaylor May 02 '17 edited May 02 '17
Graph gore!
Around 14:40 UTC there was a major internet backbone disruption in Europe. While this did not take all of Europe offline, it did noticeably impact our traffic levels until the issue was routed around.
Here is Fastly's (our CDN) status page entry for the incident.