Part 1: Link Layer Attacks
Part 2: Network Layer Attacks
Part 3: Transport Layer Attacks
Part 4: DNS Attacks
Part 5: DDoS Attacks

Transport Layer: Managing Connections

The Transport layer is responsible for moving data between applications on different hosts. Two protocols dominate this layer. TCP (Transmission Control Protocol) provides reliable, ordered delivery and is used for most applications that need correctness, such as web browsing or file transfer. UDP (User Datagram Protocol) provides best-effort delivery without reliability or ordering and is used where low latency is more important than guaranteed delivery, such as gaming, voice, time synchronization, and DNS.

Both protocols have their own security weaknesses. Most of them relate to the assumptions the designers made about how networks would behave and how endpoints would interact.

TCP: The Connection-Oriented Protocol

TCP gives applications the illusion of a dependable byte stream. To do this, it maintains connection state, tracks sequence numbers, resends lost packets, and ensures that the receiving application sees data in exactly the order it was sent. These features are implemented through mechanisms that can be exploited when they are pushed outside their normal operating conditions.

The Three-Way Handshake

Before any useful data is exchanged, TCP must establish a connection. This happens through a three-step handshake.

SYN: The client sends a SYN (synchronize) packet to the server, including an Initial Sequence Number (ISN) that will be used to track data in this connection
SYN-ACK: The server responds with a SYN-ACK packet, acknowledging the client's ISN and providing its own ISN
ACK: The client finishes the handshake by sending an ACK packet back to the server, acknowledging the server's ISN.

Once the server receives that final ACK, both sides know where to begin counting bytes and can send data. This handshake is simple but creates two important weaknesses: sequence numbers that can be predicted and connection state that can be abused.

Predictable Sequence Numbers and Session Hijacking

In TCP’s early days, Initial Sequence Numbers often followed simple patterns. Some systems incremented the number for each new connection, while others derived it from a timer.

Predictable patterns allowed attackers to guess the sequence number that a client would use. With a correct guess, an attacker could forge packets that the server would accept as belonging to the connection. This form of attack was known as TCP session hijacking because the attacker injected data into an existing connection without ever seeing the legitimate traffic.

Modern systems fix this by generating random initial sequence numbers as described in RFC 6528. Blind guessing attacks are no longer feasible. However, an attacker who can observe traffic on the network can still read the real sequence numbers and inject packets. In those cases, encryption such as TLS prevents injected packets from being interpreted as meaningful data.

SYN Flooding: Overwhelming the Handshake

The handshake also creates an imbalance in resource consumption. When a server receives a SYN, it allocates kernel memory to keep track of the new connection and sends a SYN-ACK. The server then waits for the final ACK.

If that ACK never arrives, the server keeps the allocated memory for a timeout period (typically 30–60 seconds) before freeing it.

Attackers exploit this imbalance by sending many SYN packets and never completing the handshake. Each incomplete handshake forces the server to hold state for several seconds. A sustained flood of these partial connections quickly exhausts the server’s backlog of pending connections.

This technique is known as SYN flooding. Once the backlog is full, the server cannot accept legitimate connections, and clients experience the service as unavailable.

Defenses Against SYN Flooding: SYN Cookies

SYN cookies are an effective defense against SYN flooding. Instead of allocating memory when the SYN arrives, the server encodes the connection information into the sequence number of the SYN-ACK message.

The initial sequence number is not random; it is a short cryptographic hash of the client address, client port, server address, server port, a timer, and a secret key. When the final ACK arrives, the server recomputes the value and verifies it. Only then does it allocate memory for the connection.

SYN cookies eliminate the resource asymmetry that enables SYN flooding. The tradeoff is that some optional TCP features cannot be negotiated during attack conditions. Most systems enable SYN cookies automatically when they detect that their connection backlog is filling.

TCP Reset Attacks: Forcibly Closing Connections

A TCP connection can also be attacked with a forged RST (reset) packet. The RST flag tells the receiver to stop using the connection immediately. This feature exists to allow systems to recover quickly after crashes or software failures.

TCP reset attacks exploit this mechanism to forcibly terminate legitimate connections.

If an attacker can see a connection’s packets, the attacker can forge a reset that contains the correct source addresses, ports, and sequence numbers. The connection closes instantly, making this a way to terminate downloads, SSH sessions, video calls, or any application that relies on a steady stream of data.

Reset injection has been used by censorship systems. China’s Great Firewall has historically looked for certain keywords or destinations and injected RST packets to break those connections. To the user, the connection simply seemed unstable.

Defenses Against Reset Attacks

Modern TCP implementations perform tighter validation of RST packets as defined in RFC 5961. The receiver now accepts a reset only if the sequence number is extremely close to the number it expects next, which makes blind guessing much harder. These stricter checks significantly reduce the chance that an attacker who cannot see the traffic can force a reset. TLS also helps, not by preventing resets, but by making it harder for attackers to know when to send a reset or what sequence number to use.

TCP also supports MD5 signatures for critical connections such as BGP sessions. These signatures prove that packets come from the real peer. They are not used for normal client-server traffic because they require pre-shared keys and add operational overhead.

Note that, while it's called a signature, the MD5 signature is not a digital signature in the cryptographic sense. It is simply a keyed MD5 hash, functionally similar to an HMAC but implemented in a nonstandard way. It is computed by hashing certain fields in the TCP header, the payload, and a secret key using the MD5 hashing algorithm. MD5 is an older hash function that is now considered insecure for any cryptographic use.

UDP: The Stateless Protocol

UDP is fundamentally different from TCP. It provides no connections, no reliability, no ordering: just simple message delivery. It sends a datagram and does not track whether it arrived. There are no sequence numbers and no initialization handshake. This makes UDP useful for real-time communication, where resending old data is pointless, and for simple query-response protocols such as DNS. DNS avoids TCP because creating and maintaining a connection adds unnecessary delay and memory overhead.

UDP Address Spoofing

The stateless nature of UDP creates opportunities for abuse. Because there is no handshake to flood, no sequence numbers to predict, and no connection state to hijack, an attacker can send a UDP packet that claims to come from any source address. The receiver has no way to verify this.

Forged UDP packets allow attackers to impersonate trusted systems. If a service expects messages from a particular host, an attacker can send packets with that source address, and the service will accept them as genuine. Whether this is harmful depends on the application protocol, but many early UDP-based protocols assumed local trust.

Reflection and Amplification

Spoofed UDP packets enable a more serious threat: reflection. If the attacker sends a request to a server and uses the victim’s IP address as the source, the server sends its reply to the victim. If the attacker sends spoofed requests to many different servers, or if each response is much larger than the request, the victim can become overwhelmed with traffic. This is an amplification attack. DNS and several other UDP-based protocols have been abused this way. We will examine these attacks more closely in the DDoS lecture.

Defenses

Because UDP itself has no authentication, the most important defenses occur in the network:

ISPs can block packets that have source addresses that do not belong to the customer’s network. This is known as anti-spoofing or ingress filtering (ingress = entry).
Servers can rate limit responses to avoid becoming reflection points.
Some applications add their own challenge-response or token-based authentication to verify that a request is coming from a legitimate source.

Transport Layer Security Principles

Transport layer attacks share several patterns. Many exploit asymmetry: it is cheap for attackers to create half-open connections or spoof packets, but expensive for servers to handle them. Others exploit protocol assumptions. TCP assumes packets belong to legitimate connections. UDP assumes networks will prevent spoofing. When these assumptions fail, the protocols behave poorly.

Another issue is visibility. Floods, resets, and spoofed packets often look similar to normal traffic. Servers may struggle to distinguish attack traffic from genuine surges in usage.

The most effective approach is defense-in-depth. SYN cookies, stricter validation rules, encryption, anti-spoofing, and traffic monitoring all contribute. No single mechanism can address every attack, but together they raise the difficulty for attackers and improve resilience.

Network Security