Protocol DDoS Attacks Guide

Protocol DDoS attacks are denial-of-service attacks that abuse how network protocols such as TCP, UDP and ICMP work at Layers 3 and 4 of the OSI model. Instead of simply filling a link with traffic, they fill connection state tables, SYN backlogs and packet-processing queues on servers, firewalls and load balancers until legitimate connections are dropped. SYN floods, Ping of Death and Smurf attacks are the classic examples. Below you will see how each attack works, how to spot one in your telemetry, and which defenses hold up under real load. The surprising part is that most of these attacks are tiny, and that is exactly why they slip past bandwidth alarms.

Key Takeaways

• Protocol DDoS attacks target Layer 3 (IP, ICMP) and Layer 4 (TCP, UDP) and are measured in packets per second, not bits per second.
• They exhaust finite resources such as firewall state tables, the TCP SYN backlog and CPU cycles spent per packet.
• SYN floods, ACK floods, UDP floods, ICMP floods, Ping of Death, Smurf attacks and fragmentation floods are the main protocol attack types.
• Cloudflare’s latest half-year threat report found 96.62% of network-layer attacks stayed under 500 Mbps and 90.60% ended in under 10 minutes.
• Stateful firewalls and load balancers are often the first devices to fail, because every spoofed packet consumes an entry they must track.
• Reliable mitigation layers anti-spoofing, SYN cookies, upstream filtering and anycast scrubbing so no single device carries the full load.

What Are Protocol DDoS Attacks?

A protocol DDoS attack is a distributed attack that exploits the rules of network and transport protocols to exhaust the resources a target needs to process connections. The target does not run out of bandwidth first. It runs out of memory for connection state, backlog slots or CPU time.

Every TCP handshake, every UDP datagram and every ICMP echo request asks a device to do a small amount of work. A server allocates a slot for a half-open connection. A stateful firewall creates an entry in its session table. A router’s control plane generates an ICMP reply. Protocol attacks multiply that small cost by millions of packets per second, which is why effective DDoS protection for this class of threat has to inspect and filter packets before they reach any stateful device.

The U.S. Cybersecurity and Infrastructure Security Agency (CISA) uses the same classification in its joint guidance with the FBI and MS-ISAC:

“Protocol-based DDoS attacks typically target Layers 3 (network layer) and 4 (transport layer) of the Open Systems Interconnection (OSI) model.”

Two real examples show how common this attack class is. The leaked Mirai botnet source code ships with dedicated SYN, ACK and GRE flood modules alongside its HTTP flood, so protocol attacks are built into the most copied IoT botnet in circulation. And Microsoft reported that the record Azure attack from the Aisuru botnet was built on extremely high-rate UDP floods, a Layer 4 technique, rather than application-layer requests.

Knowing the layer is only half the picture. The next question is how protocol attacks differ from the other two DDoS families, because the defense you buy depends on it.

Protocol Attacks vs Volumetric and Application-Layer DDoS

Protocol attacks sit between volumetric attacks, which saturate bandwidth, and application-layer attacks, which exhaust web server logic. They are best measured in packets per second (pps) because their damage comes from the number of packets a device must track, not the size of those packets.

Security teams often lump DDoS protocol attacks together with the other two families, and that creates blind spots. If you already understand the difference between DoS and DDoS attacks, this three-way split is the next layer of detail your runbooks need.

Yes, the categories are clean on paper, but real attacks blur them. The Azure attack that Microsoft mitigated peaked at 15.72 Tbps and nearly 3.64 billion packets per second. At that scale a UDP flood is volumetric and protocol-based at once, because both the links and every per-packet process are under pressure. Most multi-vector campaigns mix a bandwidth component with a state-exhaustion component to see which one breaks first.

So why does a protocol flood hurt more than its size suggests? The answer is in the resources it consumes.

How Protocol-Based DDoS Attacks Exhaust Network Resources

Protocol-based DDoS attacks win by exhausting a fixed-size resource that a device allocates for each connection or packet. Once that resource is full, the device drops new legitimate traffic even if the network link is mostly idle.

Think of a restaurant that holds a table for every phone reservation. If thousands of fake reservations arrive and nobody shows up, the dining room is empty but every real guest is turned away at the door. A SYN flood does the same thing to a server’s backlog of half-open connections.

• State tables: stateful firewalls, NAT gateways and load balancers keep one entry per flow. Spoofed packets from random source addresses create entries that never complete.
• SYN backlog: each half-open TCP connection holds a slot. On Linux, the default tcp_synack_retries value of 5 means a passive connection is only timed out after 63 seconds, so each fake SYN ties up a slot for about a minute.
• CPU per packet: routers and hosts spend CPU on every packet they must parse, answer or reassemble. ICMP replies and fragment reassembly are expensive when multiplied by millions of packets.

This is why architecture matters as much as filtering. A TCP and UDP reverse proxy such as a Layer 4 Gateway terminates sessions on distributed edge infrastructure, so half-open connections pile up on hardware built to absorb them instead of on your origin firewall.

Each protocol attack type targets these resources in its own way. Here is how the main ones work in practice.

Types of Protocol DDoS Attacks You Will Actually See

The network protocol attacks that appear in real incident data fall into five groups. TCP DDoS attacks (SYN floods and other flag floods) and UDP floods work at Layer 4, while ICMP-based attacks and fragmentation or GRE floods are Layer 3 DDoS attacks. Each one abuses a different protocol rule.

SYN Flood Attacks: Filling the TCP Backlog

A SYN flood attack sends a stream of TCP SYN packets, usually with spoofed source addresses. The server answers each with a SYN-ACK and waits for a final ACK that never arrives. The IETF’s RFC 4987 describes the outcome in one line:

“By keeping the backlog full of bogus half-opened connections, legitimate requests will be rejected.”

One of the first widely reported SYN floods took New York ISP Panix offline for days, and the same RFC records that the attack “caused well-publicized outages” of an ISP’s mail servers. The technique is still current, and SYN floods remain one of the standard modes in the Mirai source code and its many forks.

ACK, RST and Other TCP Flag Floods

ACK and RST floods send packets that claim to belong to connections that do not exist. A server simply discards them, but a stateful firewall must look up every packet in its session table before deciding, and that lookup is the bottleneck. Mirai ships a dedicated “ack” mode for this purpose, and SYN cookies do nothing against it because no handshake is involved.

Researchers from the University of Maryland also showed at USENIX Security that misconfigured censorship middleboxes will reflect and amplify TCP traffic, which turned TCP into a reflection vector that many teams assumed only UDP could provide.

UDP DDoS Attacks and Port Floods

A UDP flood attack sends large numbers of UDP datagrams to random or specific ports. UDP is connectionless, so there is no handshake to validate the sender. For each closed port, the host generates an ICMP Destination Unreachable reply, which doubles the processing cost.

The largest cloud attack on record was this type. Microsoft reported that the Aisuru botnet hit a single public IP address in Australia with UDP floods from more than 500,000 source IPs, using random source ports and minimal spoofing. At 3.64 billion packets per second, the packet rate alone would overwhelm any single appliance.

ICMP DDoS Attacks: Ping Floods, Ping of Death and Smurf

ICMP attacks abuse the protocol that networks use for diagnostics and error reporting. A ping flood simply sends echo requests faster than the target can answer. Two older variants are more interesting because they keep coming back.

• Ping of Death: the attacker sends fragments that reassemble into an IP packet larger than the 65,535-byte maximum the IPv4 header allows, overflowing the receiving buffer. The Ping of Death idea returned in CVE-2020-16898 (“Bad Neighbor”), a Windows TCP/IP flaw in ICMPv6 Router Advertisement handling rated CVSS 9.8.
• Smurf attack: the attacker sends ICMP echo requests to a network’s directed broadcast address with the victim’s IP as the spoofed source, so every host on that network replies to the victim. CERT Advisory CA-98.01 documented the technique, and RFC 2644 later changed router defaults to drop directed broadcasts. A Smurf attack still works wherever that default has been overridden.

IP Fragmentation and GRE Floods

Fragmentation floods send IP fragments that the target must hold in memory while waiting for the rest of the packet, which often never arrives. GRE floods wrap junk traffic in Generic Routing Encapsulation headers so that devices must decapsulate each packet. Mirai includes two GRE flood modes (greip and greeth), and GRE traffic was a major part of the Mirai attack that security journalist Brian Krebs reported at about 620 Gbps against KrebsOnSecurity.

Here is how the main attack types compare side by side:

Knowing the types is useful. Knowing how big and how long real attacks are is what shapes your alerting thresholds.

What Real Attack Data Says About Layer 3 and Layer 4 DDoS Attacks

Layer 3 and Layer 4 DDoS attacks are extremely frequent, usually small and usually short, with a growing minority reaching multi-terabit scale. Planning only for the headline records misses most of the attacks that actually happen.

Cloudflare’s most recent half-year DDoS threat report counted 23.2 million network-layer DDoS attacks, which works out to roughly 128,000 attacks per day. At the top end, 935 network-layer attacks exceeded 1 Tbps, and 805 of those landed in the second quarter alone. DNS-based floods made up 34.3% of network-layer attacks, and CLDAP floods rose 580% quarter over quarter.

Here is what most people miss: the same report found that 96.62% of network-layer attacks stayed under 500 Mbps and 90.60% ended in under 10 minutes. A 300 Mbps SYN flood will not trip a bandwidth alarm on a 10 Gbps link, yet it can fill a firewall’s session table in seconds. By the time a human opens a ticket, the attack is often over and the outage already happened.

If attacks are that short, detection has to be fast and automatic. The next section covers the signals worth wiring into your monitoring.

How to Detect Protocol DDoS Attacks Early

You detect protocol DDoS attacks by watching packet rates, connection states and protocol ratios rather than bandwidth graphs. A healthy baseline for each of these makes the anomaly obvious within seconds.

Many of the early signs of a DDoS attack show up in kernel counters long before users complain. These are the signals worth tracking:

• A spike in connections stuck in the SYN_RECV state, or a jump in the kernel’s SyncookiesSent counter.
• A rising ratio of inbound SYN packets to completed three-way handshakes.
• Firewall or load balancer session tables approaching their configured maximum.
• Unusual volumes of ICMP echo requests, ICMP unreachable messages or IP fragments.
• Traffic from very large numbers of source IPs with random source ports, the pattern of botnet DDoS attacks built from compromised routers and cameras.

On a Linux host, a few commands give you a quick read during an incident:

The TcpExtListenOverflows and TcpExtListenDrops counters are especially useful because they only rise when the accept queue is already full, which means real users are being turned away. Once you can see the attack, the question becomes where to stop it.

Protocol DDoS Attack Mitigation That Holds Up Under Load

Effective DDoS attack mitigation for protocol floods stops malicious packets as far upstream as possible and keeps stateful devices out of the blast radius. No single control is enough, so the defense is built in layers.

Harden Hosts and the Network Edge

1. Validate source addresses at the edge. BCP 38 (RFC 2827) describes ingress filtering, where a network forwards only packets whose source address belongs to its own advertised prefixes and denies everything else. When ISPs and hosting providers apply it, spoofed SYN floods and the forged ICMP broadcast requests behind Smurf attacks cannot leave the attacker’s network. On your own edge, drop inbound packets that claim to come from your own prefixes or from private and unallocated ranges. The RFC is clear that this does not stop floods sent from valid, non-spoofed addresses, which is why the next steps still matter.
2. Turn on SYN cookies. They let a server answer SYNs without allocating state until the handshake completes.
3. Shorten half-open timeouts. Lowering tcp_synack_retries from 5 to 2 frees backlog slots far sooner.
4. Rate-limit ICMP instead of blocking it. Echo and unreachable messages need limits, but path MTU discovery depends on ICMP working.
5. Drop directed broadcasts and malformed fragments on routers and hosts.

Push Filtering Upstream Before Traffic Reaches You

Host tuning fails once packet rates exceed what your uplink or firewall can process. At that point filtering has to happen on your provider’s routers. BGP FlowSpec (RFC 8955) lets you push precise match rules, such as UDP to port 0 or fragments to a specific prefix, to upstream routers in seconds. Remotely triggered blackholing is the blunter option: it drops all traffic to the victim IP, which stops the attack but also completes it.

Absorb Floods on a Distributed Anycast Edge

For attacks measured in hundreds of millions or billions of packets per second, capacity has to be distributed. An anycast network announces the same IP space from many locations, so each point of presence absorbs only the share of the botnet closest to it. Scrubbing systems at each location drop spoofed SYNs, invalid flags and malformed fragments, then forward clean traffic to the origin. Combined with the DDoS mitigation techniques above, this is the model that survived the multi-terabit Aisuru floods.

Even well-built layers fail when teams make a few predictable mistakes. Those are worth checking before the next attack does it for you.

Common Mistakes in Layer 3 and Layer 4 DDoS Defense

The most common mistakes are trusting stateful devices to protect themselves, blocking ICMP entirely and sizing defenses by bandwidth alone. Each one leaves a gap that protocol attacks are designed to find.

You might be thinking your next-generation firewall already handles this. It does handle a lot, but a stateful firewall is also one of the most common targets. Every spoofed SYN or ACK creates or checks a session entry, so the device that is supposed to filter the attack becomes the bottleneck. Filtering has to happen in front of it.

You might also be thinking the safest move is to drop all ICMP. That breaks path MTU discovery and, on IPv6, neighbor discovery and router advertisements. RFC 4890 lists which ICMPv6 messages must never be filtered for exactly this reason. Rate limits and patched IP stacks are the better answer.

• Sizing protection by Gbps only, when protocol floods are defined by packets per second.
• Relying on manual response for attacks that end in under 10 minutes.
• Leaving default half-open timeouts and backlog sizes in place on internet-facing hosts.
• Protecting HTTP services through a CDN while leaving SSH, game, VoIP or database ports exposed on the origin IP.
• Never testing failover to upstream scrubbing, so the first real test is a live attack.

Avoiding these mistakes narrows your options to a few realistic protection models. Here is how they compare.

How to Choose DDoS Protection for Protocol Attacks

Choose DDoS protection for protocol attacks based on packet-rate capacity, where filtering happens relative to your stateful devices, and whether it covers every TCP and UDP port you expose, not only web traffic.

Most organizations end up with a hybrid: host hardening and anti-spoofing everywhere, FlowSpec or provider filtering for known patterns, and always-on cloud scrubbing in front of anything that must stay reachable. The right mix depends on how many non-HTTP services you expose and how much downtime a single attack would cost.

Final Thought on Protocol DDoS Attacks

Protocol DDoS attacks succeed because they target the finite state that every network device keeps, not the size of your pipe. That single principle explains why small, short floods take services offline, why stateful firewalls fail first and why bandwidth-based monitoring misses so much.

The practical takeaway is to measure and defend in packets per second. Harden hosts with SYN cookies and sensible timeouts, stop spoofed traffic at the edge, push precise filters upstream, and keep a distributed scrubbing layer in front of every exposed TCP and UDP service. Teams that plan this way turn most protocol attacks into a line in a log file rather than an outage.

Common Questions About Protocol DDoS Attacks