Dyn DNS Outage: What happened and how to reduce your risk

By now you’ve likely read about the Distributed Denial of Service (DDoS) attack that took down a healthy swath of the Internet last Friday, October 21st. This attack has been reported as the largest DDoS attack on record, fitting in with the worrisome trend that attacks of this type are becoming more prevalent and disruptive.

What is a DDoS?

A Distributed Denial of Service, or DDoS, attack is one in which a large number of devices spread across the Internet (“Distributed”) attack and overwhelm a target, rendering it unable to provide services to its normal users (the “Denial of Service” part). Attacks of this sort aren’t new, but they’re growing more massive and more distributed (which suggests they are also becoming more sophisticated), resulting in an increase in both the number and average duration of attacks year over year. Imagine millions of bees emerging from the corners of a large field, and forming a swarm before attacking its victim. The specific attack vector in a DDoS can vary greatly. In this recent attack, the vector was a swarm of DNS queries (small packets of digital information sent via the Internet to DNS servers) sent to the Dyn company’s DNS servers. The flood of traffic (packets) to the servers overwhelms the servers and prevents them from responding to legitimate requests.

What was affected by this DDoS attack?

Dyn has a large, globally distributed network, so the early attacks affected a minor portion (the East coast) of their systems. Later sorties spread the outage across the country, with subsequent attempts thwarted by Dyn’s vigilant defenses. This attack affected the Domain Name System (DNS) services for Dyn’s customers, which includes several high-profile websites including the New York Times, Spotify, and Reddit. These attacks aren’t new for Dyn, and in fact they have teams focused on mitigating and addressing them. However, the unprecedented scope and scale of the attack was atypical and went well beyond what standard defenses can protect against.

DNS is the “address book” for the Internet -- it translates the human-friendly domain names we’ve all become so reliant on (i.e. “nytimes.com”) to the IP addresses that systems need to transfer data to and from web servers (i.e. -  151.101.21.164, or 2607:f8b0:4000:802::2004). When those address book lookup requests fail, your browser doesn’t know how to get to the server it's looking for. The servers and datacenters at the New York Times remained unaffected and continued to hum along just fine; yet the ability of client devices (your browser) to lookup the addresses of those servers was compromised. This is why many in the technical community are so deeply concerned -- these attacks underscore a fundamental failure of one of the foundational components of the Internet, exposing a flaw in its genome.

Where did it come from?

News reports suggest the source of this DDoS was a “botnet” called “Mirai”. A botnet is a swarm of systems, most often workstations, that have been infected with malware. Malware, like viruses, come in different flavors, but in the case of a botnet, they lay dormant on a system until a remote command is received from some centralized command-and-control system. When that command is received, the malware springs to life and attacks the target (in this case, likely sending continuous DNS query packets to Dyn’s DNS servers). The good thing about those typical botnet sources is that eventually the systems’ users (or their network administrators) are able to notice the issue or impact on network performance, and respond by running an antivirus program and/or patching the system. The Mirai botnet is reportedly infecting quite a few non-traditional devices, such as Internet-connected cameras, DVRs, and other Internet of Things (IoT) devices. This makes the botnet doubly dangerous, as those devices tend to be monitored and updated less frequently than a user’s workstation.

How can I protect my website?

Unfortunately, you can’t completely insulate any service on the Internet from an intentional DDoS attack, especially one as widespread as last week’s. Anybody suggesting otherwise is not telling the whole story for technical reasons that we won’t go into here. In many ways, the very design that makes the Internet resilient prevents the control of such attacks beyond a certain threshold.

While you can’t completely eliminate the chance of attack, you can take many steps to mitigate the risk.

First, ensure that you have multiple, geographically disparate, and independent DNS servers to provide the DNS services for your domain names and online properties. If you have two DNS servers (a primary and secondary) and they’re both on the same network, on the same server, or even in the same datacenter, then you’re doing it wrong. Most larger organizations utilize an outsourced DNS provider (i.e.- Dyn, UltraDNS, Amazon’s Route 53, or CloudFlare) that handle much of this for them. We leverage these services on our clients’ behalfs whenever practical. But what happens when your DNS provider goes down as Dyn did last week, or as UltraDNS did back in 2015? You need another layer of protection to prevent against a larger-scale DNS provider outage. 

The next level of risk mitigation involves utilizing multiple, independent, hosted DNS providers. This adds a bit more to the management layer for your network staff (you’ll likely need to replicate your updates across each provider’s management systems whenever an update is required to your network infrastructure). However, the added resilience would help ensure that one of your providers can always respond to DNS queries should the other provider (say, Dyn) go down. Depending on your firm’s services and profile, the extra effort may be well worth the reduction in risk.

If you’re not sure how your site is configured, and you’re concerned about the risk of DoS and DDoS attacks, talk to your IT team and pass on the suggestions above. If we are your technology partner, reach out and let us know your concerns and we’ll work with you to provide an appropriate security upgrade to your web/application presence. Beyond DNS, many of our clients have put website resilience and security at the top of their lists, and we have a variety of solutions, both proactive and reactive, to assist with these challenges.