The Network Security Problem

The Internet's core protocols were designed without security. IP, TCP, UDP, and routing protocols have no built-in authentication, integrity checks, or encryption. This creates serious vulnerabilities:

• No authentication: IP packets can be forged with fake source addresses

• No integrity protection: Data can be modified in transit without detection

• No confidentiality: Anyone along the path can read packet contents

• Vulnerable routing: BGP and other routing protocols can be manipulated

• Spoofable protocols: ARP, DHCP, and DNS can be spoofed to redirect traffic

We need security mechanisms that work despite these vulnerabilities, providing confidentiality, integrity, and authentication even when the underlying network is completely untrusted.

Transport Layer Security (TLS)

What TLS Provides

TLS creates a secure channel between two applications. After the initial handshake completes, applications communicate as if using regular TCP sockets, but with three critical guarantees:

• Authentication: A client can verify it's talking to the correct server (the server can optionally verify the client)

• Confidentiality: Communication is encrypted, preventing eavesdropping

• Integrity: Any tampering with the data stream will be detected

TLS (Brief Review)

The TLS 1.3 handshake:

• Negotiates acceptable cryptographic algorithms

• Uses Diffie–Hellman to derive a shared secret

• Has the server prove possession of a private key associated with a CA-signed certificate

• Derives symmetric encryption keys for application data

TLS 1.3 then uses AEAD encryption such as AES-GCM or ChaCha20-Poly1305.

AEAD (Authenticated Encryption with Associated Data) combines encryption and authentication in a single operation. Think of it as generating a keyed MAC concurrently with encryption, but more efficient than doing encrypt-then-MAC separately. Common algorithms are AES-GCM and ChaCha20-Poly1305.

Mutual authentication is possible but rarely used because deploying client certificates at scale is impractical; most applications authenticate clients with passwords or MFA inside the TLS channel.

Client Authentication

TLS supports mutual authentication, but client certificates are rarely used in practice:

• Setting up certificates for users is complicated

• Moving private keys between devices is difficult

• Any website could request your certificate (destroying anonymity)

• Public computers create security risks for private keys

Common practice: Authenticate clients after establishing the TLS connection using passwords, multi-factor authentication, or other mechanisms. TLS protects the confidentiality of these credentials.

TLS Limitations

While TLS solves many problems, it has important limitations:

• It's a transport-layer solution; applications must explicitly use it

• It only protects communication between two specific applications

• A TLS connection doesn't guarantee that the server itself is trustworthy

• It adds some latency (though TLS 1.3 minimized this)

Virtual Private Networks (VPNs)

As we saw, the Internet's core protocols were designed without security. TLS solves this at the transport layer, but each application must implement it separately.

Virtual Private Networks (VPNs) take a different approach by operating at the network layer, protecting all traffic between networks or hosts automatically. Once a VPN tunnel is established, every application benefits from its security without any changes.

Tunneling

Tunneling is the foundation of VPNs. A tunnel encapsulates an entire IP packet as the payload of another IP packet. This allows private addresses within a local area network (like 192.168.x.x) to communicate across the public Internet.

A gateway router on one network takes an outgoing packet destined for a private address on another network, wraps it inside a new packet addressed to the remote gateway's public address, and sends it across the Internet. The remote gateway extracts the original packet and delivers it to its destination.

Basic tunneling provides no security. The encapsulated data travels in unencrypted, endpoints are not authenticated, and there is no integrity protection.

What makes a VPN?

A VPN combines tunneling with three security properties:

Encryption ensures outsiders cannot read the encapsulated data, even if they capture packets on the public Internet.

Integrity protection through message authentication codes ensures outsiders cannot modify data without detection. Tampered packets are discarded.

Authentication ensures you are connected to the legitimate gateway, not an imposter. This typically uses certificates, pre-shared keys, or public key cryptography.

The formula: VPN = tunnel + encryption + integrity + authentication.

VPN Deployment Models

VPNs serve three primary purposes, each with different security implications.

Site-to-Site (Network-to-Network)

This was the original VPN use case. Organizations connect geographically separated networks through VPN tunnels between gateway routers. Computers on either network communicate as if on a single unified network, unaware that traffic is encrypted and tunneled.

Remote Access (Host-to-Network)

Individual computers connect to corporate networks from remote locations. The remote computer runs VPN client software that establishes a tunnel to the corporate VPN gateway. Once connected, the computer can access internal resources as if physically in the office.

Privacy VPNs (Host-to-Provider)

Commercial services like ExpressVPN, NordVPN, and ProtonVPN allow users to establish VPN connections to the provider's network, which then acts as a gateway to the Internet. Traffic appears to originate from the provider rather than the user's actual location.

These services protect against local eavesdropping (such as on public Wi-Fi networks) and can bypass geographic content restrictions. However, the VPN provider can see all your traffic. You are shifting trust from your ISP to the VPN provider, not eliminating surveillance. Whether this improves privacy depends entirely on whether you trust the provider more than your ISP.

VPN Protocols

VPN protocols have evolved over time, each with different design philosophies. Three protocols dominate VPN deployments, each with different design philosophies.

IPsec (earliest: 1990s)

IPsec was developed as the first standardized approach to network-layer security. It operates at Layer 3 (i.e., it does not use TCP or UDP) and is typically implemented in the operating system kernel. IPsec is a separate protocol from TCP and UDP, using unique protocol numbers in the IP header to identify the encapsulated content as being IPsec data.

IPsec consists of two separate protocols (use one or the other):

1. The Authentication Header (AH) protocol provides authentication and integrity but not encryption.

2. The Encapsulating Security Payload (ESP) provides authentication, integrity, and encryption. ESP is the standard choice today since it does everything AH does plus encryption.

IPsec operates in two modes:

1. Tunnel mode encapsulates the entire original IP packet for network-to-network or host-to-network communication.

2. Transport mode protects only the payload for direct host-to-host communication.

IPsec Cryptography: Both sides negotiate on the algorithms to use when the connection is set up. IPsec uses the IKE (Internet Key Exchange) protocol to negotiate keys and algorithms, which in turn uses Diffie–Hellman key exchange. AES-CBC or AES-GCM are used for confidentiality, and HMAC-SHA1/SHA2 for integrity.

Advantages of IPsec:

• Standardized and widely supported

• Efficient kernel implementation

• Transparent to applications

Disadvantages:

• Complex standard

• Complicated configuration

• Problems with NAT (network address translation gateways)

OpenVPN (early 2000s)

OpenVPN emerged as the first widely-adopted open-source VPN protocol. Unlike IPsec, it runs in user space (not the kernel) and uses TLS for the control channel.

OpenVPN communicates via an operating system's TUN (TUNnel) virtual network interface, which the operating system treats as a regular interface. Traffic destined for the VPN is routed to this interface, encrypted by the OpenVPN process, and sent as regular UDP or TCP packets.

OpenVPN separates communication into two channels:

• The control channel uses TLS for authentication and key exchange.

• The data channel carries encrypted tunnel traffic using symmetric encryption (typically AES or ChaCha20) with keys derived from the TLS handshake. Like IPsec, OpenVPN negotiates on a specific set of algorithms to use when setting up a connection.

Advantages of OpenVPN:

• Open source

• Can run over TCP or UDP (TCP makes it easier to bypass firewalls)

• Supports pre-shared keys or certificate-based authentication

• Highly portable across operating systems

Disadvantages:

• Performance overhead from running in user space

WireGuard (newest: 2016)

WireGuard takes a minimalist approach. Its entire codebase is approximately 4,000 lines (compared to hundreds of thousands for IPsec or OpenVPN), enabling formal verification of its cryptographic properties. It was incorporated into the Linux kernel in version 5.6 (March 2020).

WireGuard Cryptography

WireGuard makes opinionated choices rather than offering configuration options. There is no cipher negotiation; it uses exactly one set of modern algorithms: Elliptic Curve Diffie-Hellman for key exchange, ChaCha20 for encryption, Poly1305 for message authentication, and BLAKE2s for hashing (used for deriving keys). This eliminates vulnerabilities related to negotiation and downgrade attacks.

Each peer is identified by its public key rather than a certificate. Configuration requires generating key pairs, exchanging public keys out-of-band, and specifying which IP addresses should route through the tunnel.

Advantages of WireGuard:

• Extremely small codebase, formally verified

• Uses only modern cryptographic primitives (ChaCha20, Poly1305, Curve25519)

• Runs in kernel space, communicates via UDP

• No complexity of issuing and managing certificates

• No protocol negotiation; uses best current practices

• Extremely efficient (rapid setup, fast algorithms, kernel operation)

• Rotates session keys regularly (~every 2 minutes of active traffic)

• Simple configuration: exchange public keys, specify IP addresses

Disadvantages:

• Newer (but verified)

• Not supported by some enterprise VPN products (but that's changing)

• Lacks cipher negotiation (which is good but an issue if more desirable ciphers emerge)

• Requires a separate key distribution infrastructure for large deployments

Comparing the Protocols

The three protocols reflect different eras and design philosophies:

IPsec offers broad compatibility with enterprise equipment but has a complex configuration. It operates at the network layer in the kernel.

OpenVPN offers portability and the ability to bypass firewalls, but with some performance overhead. It runs in user space using TLS.

WireGuard offers simplicity and high performance with modern cryptography. It runs in kernel space with a minimal, formally-verified codebase.

Security Limitations

VPNs are not a complete security solution.

• They protect data in transit through the tunnel but do not protect compromised endpoints. If your computer has malware, the attacker can access data before it enters the tunnel.

• VPNs do not guarantee anonymity. The VPN provider or corporate network can see your traffic. Websites can still identify you through cookies, browser fingerprinting, and login credentials.

• Using a VPN can create a false sense of security. VPNs protect against some threats (eavesdropping on local networks) but not others (phishing, malware, compromised websites). Security requires defense in depth.

VPN Performance Considerations

VPNs come with performance overhead, though not all types are equally significant:

• Encryption overhead: Minimal with modern hardware (CPUs have hardware acceleration for AES)

• Encapsulation overhead: Adds 20-60 bytes of headers; generally not significant

• Routing overhead: Usually the bigger issue; traffic routes through distant servers, adding latency

For corporate VPNs, server/gateway capacity can also become a bottleneck.

Terms you should know