August 26, 2026

·Quantum TLS

Beyond TLS: Post-Quantum VPNs (WireGuard and IPsec)

TL;DR

TLS 1.3's PQC migration (hybrid X25519MLKEM768) rides on TCP, which handles arbitrarily large handshake messages transparently via segmentation. Network-layer VPN protocols do not have this luxury: WireGuard's Noise-based handshake and IKEv2's IKE_SA_INIT exchange are single UDP datagrams by design, and inflating them with kilobyte-scale ML-KEM public keys/ciphertexts risks IP-layer fragmentation — which is unreliable across the real internet due to middlebox fragment-dropping and stateless-firewall behavior. Two production-relevant approaches exist: Rosenpass, a companion protocol that runs a separate post-quantum handshake and injects the result as WireGuard's existing pre-shared key (leaving WireGuard's wire format untouched), and IKEv2 Multi-KE (RFC 9370), which splits multiple key-exchange methods across sequential IKE_INTERMEDIATE round trips instead of cramming them into one oversized IKE_SA_INIT packet. Both strategies solve the same underlying problem — keep every individual UDP datagram under path MTU — through different means: PSK injection avoids touching the data-plane format at all, while Multi-KE trades additional round-trip latency for smaller per-packet size.

TLS's Migration Path Doesn't Generalize to the Network Layer

TLS 1.3's hybrid key exchange (X25519MLKEM768, standardized via the IETF draft-ietf-tls-hybrid-design lineage) adds roughly 1,216 bytes to the ClientHello/ServerHello key_share extensions relative to classical X25519 alone. This is a non-issue for TLS because:

• TLS runs over TCP, which segments the handshake across as many TCP segments as needed and reassembles them transparently at the transport layer.
• TCP has built-in retransmission and ordered delivery — losing one segment triggers a retransmission of that segment, not silent loss of the entire handshake message.
• Path MTU is a TCP-stack concern (MSS clamping, PMTUD), invisible to the application protocol.

Network-layer VPN protocols have none of these guarantees for their own control-plane traffic:

• WireGuard runs its entire Noise handshake as individual, self-contained UDP datagrams. There is no WireGuard-level segmentation or retransmission of a single oversized handshake message — a handshake initiation is exactly one UDP packet, full stop.
• IPsec/IKEv2 similarly frames IKE_SA_INIT as one UDP datagram by default. IKE fragmentation (RFC 7383) exists but is not universally implemented or enabled, and was designed for occasional oversized certificate payloads, not as the default path for every handshake.

This is the structural reason a PQC migration at the network layer is a materially different engineering problem than the TLS case, even though the underlying KEM math (ML-KEM) is identical.

WireGuard's Handshake Budget

WireGuard's handshake initiation message, per the WireGuard whitepaper, is a fixed 148-byte UDP payload: message type (4 B), sender index (4 B), ephemeral public key (32 B), encrypted static public key (32 B + 16 B AEAD tag), encrypted timestamp (12 B + 16 B tag), and two MACs (16 B each). This fits comfortably inside any realistic path MTU with enormous headroom — the entire design assumes Curve25519's 32-byte keys.

Replacing or augmenting the ephemeral/static Diffie-Hellman exchange with a hybrid classical+ML-KEM construction breaks that assumption directly:

For ML-KEM-768 (1,184 B public key, 1,088 B ciphertext), a naive substitution of one leg of the handshake alone pushes a single direction of the exchange past 1,200 bytes — before accounting for the fact that WireGuard's Noise IK pattern needs this material flowing in both directions across the two-message handshake. This is comfortably under a 1,500-byte Ethernet MTU in isolation, but leaves dangerously little headroom once any realistic path adds its own encapsulation overhead (PPPoE: MTU 1492; an outer VPN tunnel; IPv6-in-IPv4; or the WireGuard tunnel's own packets being re-encapsulated by a corporate proxy).

Rosenpass: Post-Quantum Security Without Touching the Data Plane

Rosenpass is the most production-relevant approach to post-quantum WireGuard today, and it deliberately sidesteps the handshake-datagram-size problem rather than solving it head-on. Its design:

1. Runs as a separate protocol on its own UDP port, independent of WireGuard's handshake and data-plane wire format.
2. Performs a hybrid post-quantum handshake (classical X25519 combined with an ML-KEM-class KEM) between the same two endpoints, using its own fragmentation-tolerant framing built for exactly this purpose, since it has no legacy 148-byte budget to preserve.
3. Derives a shared secret from that exchange and feeds it into WireGuard as the pre-shared key (PSK) — a feature Noise IKpsk2 (WireGuard's actual handshake pattern) already supports natively for mixing in out-of-band symmetric material.
4. Re-runs periodically to rotate the PSK, bounding the window during which a future quantum adversary's compromise of the classical Curve25519 leg alone would suffice.

The key architectural insight: WireGuard's data-plane packet format never changes. Every WireGuard data packet after the handshake remains the same highly-optimized, hardware-friendly ChaCha20Poly1305-over-UDP format it always was. Only a companion, independently-fragmentable control channel gets the large PQC payloads, and that channel is free to design its own framing without inheriting WireGuard's historical 148-byte assumption.

# /etc/wireguard/wg0.conf — WireGuard interface using a Rosenpass-managed PSK.
# The psk file is written and rotated by the rosenpass daemon, not by wg-quick.
[Interface]
PrivateKey = <classical-static-private-key>
ListenPort = 51820

[Peer]
PublicKey = <peer-classical-static-public-key>
PresharedKey = /var/run/rosenpass/wg0.psk   # rotated periodically by rosenpassd
Endpoint = peer.example.net:51820
AllowedIPs = 10.10.0.2/32

# rosenpass.toml — companion PQ handshake daemon config
public_key = "rp-public-key.bin"
secret_key = "rp-secret-key.bin"
listen = ["0.0.0.0:9999"]

[[peers]]
public_key = "peer-rp-public-key.bin"
endpoint = "peer.example.net:9999"
key_out = "/var/run/rosenpass/wg0.psk"       # written on every successful PQ handshake
wireguard_pubkey = "peer-classical-static-public-key"

Native Hybrid-KEM WireGuard Forks

A separate line of experimental work modifies WireGuard's Noise pattern directly — substituting or augmenting the DH() operations in Noise_IKpsk2_25519_ChaChaPoly_BLAKE2s with a hybrid KEM combiner, producing a pattern informally analogous to Noise_IKpsk2_25519+MLKEM768_ChaChaPoly_BLAKE2s. This requires:

• A KEM combiner in the Noise MixKey() chain: concatenating (or KDF-mixing) the X25519 shared secret and the ML-KEM shared secret before feeding the result into the symmetric key ratchet, so that breaking either primitive alone does not break the session key.
• Reworking the handshake message format to carry the ML-KEM public key and ciphertext, which — unlike Rosenpass's separate-port design — directly inherits the fragmentation risk discussed below, since it stays inside WireGuard's own single-datagram handshake messages.
• Careful attention to WireGuard's anti-DoS cookie mechanism (MAC2), which assumes small, cheap-to-verify handshake initiations; a scheme that lets an unauthenticated peer trigger expensive ML-KEM decapsulation before cookie validation reopens a resource-exhaustion vector WireGuard's original design specifically closed.

This path is standardization-relevant (it is the direction a future "WireGuard2" data-plane-format change would take) but is materially more invasive than the Rosenpass PSK-injection approach, which is why Rosenpass is the more widely deployed option today.

IKEv2 Multi-KE: RFC 9370

IPsec's key-exchange protocol, IKEv2, addresses the same problem through a formal extension: RFC 9370 (Multiple Key Exchanges in IKEv2). Rather than negotiating one Diffie-Hellman group and stuffing every KE payload into a single IKE_SA_INIT exchange, Multi-KE:

1. Negotiates an ordered list of additional key-exchange methods (e.g., classical ECP-384 first, then ML-KEM-768, then optionally ML-KEM-1024) via the ADDITIONAL_KEY_EXCHANGE transform type.
2. Carries the first key exchange (typically the classical one, for backward compatibility with peers that don't support Multi-KE) in the standard IKE_SA_INIT exchange, unchanged in size from today's deployments.
3. Carries each subsequent KEM's public key/ciphertext in its own IKE_INTERMEDIATE exchange — a separate request/response round trip introduced specifically to host exactly this kind of large, optional payload.
4. Combines all resulting shared secrets via SP 800-56C-style KDF chaining (each new secret is mixed into the running SKEYSEED derivation) before deriving the IKE SA's traffic keys.

The structural payoff is that no single UDP datagram needs to carry more than one KEM's worth of key material. An ML-KEM-1024 exchange (1,568 B public key, 1,568 B ciphertext) fits inside its own IKE_INTERMEDIATE round trip without ever forcing the classical IKE_SA_INIT — the message every IKEv2 implementation on the internet already knows how to size correctly — to grow at all. The cost is latency: each additional KE method adds one full round trip before IKE_AUTH can proceed, which matters for connection setup time on high-latency links (satellite, congested mobile networks) but not for steady-state tunnel throughput.

# strongSwan ipsec.conf — IKEv2 with classical ECP-384 + ML-KEM-768 hybrid
# via RFC 9370 Multi-KE, and IKE fragmentation explicitly enabled.
conn pq-hybrid-tunnel
    keyexchange=ikev2
    ike=aes256gcm16-prfsha384-ecp384-ke1_ml_kem_768!
    esp=aes256gcm16-ke1_ml_kem_768!
    fragmentation=yes
    left=%defaultroute
    leftauth=pubkey
    right=vpn.example.net
    rightauth=pubkey
    auto=start

The ke1_ml_kem_768 suffix requests ML-KEM-768 as the first additional key-exchange method layered onto the base ecp384 group — the ke1/ke2/… numbering directly corresponds to the sequence of IKE_INTERMEDIATE round trips Multi-KE performs. fragmentation=yes enables RFC 7383 IKE fragmentation as a safety net for any payload (certificate chains, additional KE material) that still exceeds MTU within a single exchange despite the Multi-KE split.

Avoiding IP-Layer Fragmentation of PQC Handshake Datagrams

This is the section that matters most operationally: IP fragmentation is not a safe fallback for oversized VPN handshake packets on the real internet, for three concrete, independent reasons.

1. UDP Has No Partial-Loss Recovery

A fragmented IP datagram reassembles only if every fragment arrives. UDP itself has no retransmission — if one fragment out of, say, four is dropped, the entire reassembled datagram (the entire handshake message) is silently discarded by the receiving IP stack, and the application layer only learns about it after a full handshake timeout expires. For a WireGuard handshake initiation or an IKE IKE_SA_INIT, this converts a low-probability single-packet loss into a full handshake retry with multi-second timeout penalties, directly and disproportionately hurting connection setup latency exactly when payloads are largest (initial PQC handshake).

2. Middleboxes Drop Non-Initial Fragments

Stateful firewalls and NAT devices key their connection tracking off L4 (UDP/TCP) port numbers, which live only in the first fragment of a fragmented IP datagram — every subsequent fragment carries no port information at all. A meaningful fraction of internet middleboxes drop these non-initial fragments outright as a matter of policy (they can't be classified against a stateful ruleset), independent of any firewall misconfiguration. This is a well-documented, long-standing operational reality — it is a primary reason QUIC and WireGuard were both designed from the outset to be MTU-aware and avoid relying on IP fragmentation for anything on their critical path.

3. Reassembly Buffers Are an Attack Surface

An unauthenticated peer that can induce a target to allocate IP reassembly buffers for a claimed multi-fragment datagram — without ever completing it — has a resource-exhaustion primitive. This risk is specifically unwelcome inside a security handshake whose entire purpose is authenticating an otherwise-untrusted peer; degrading gracefully to protocol-level fragmentation (which is authenticated and bounded, as in RFC 7383) rather than raw IP fragmentation (which is neither) is the safer default.

Concrete Mitigations

The unifying principle across every one of these mitigations is the same: treat "stay under one path-MTU-sized UDP datagram per message" as a hard protocol-design constraint for PQC network-layer handshakes, and either split the handshake across multiple round trips (Multi-KE, IKE fragmentation) or move the oversized material to a channel that was never bound by the original protocol's size assumptions (Rosenpass) — rather than falling back on IP fragmentation, which the operational internet does not reliably support.

Comparative Summary

Summary of Engineering Constraints

• TLS's PQC migration is transport-layer-agnostic to fragmentation because TCP absorbs arbitrary message sizes; VPN control-plane protocols over UDP do not get this for free and must design around it explicitly.
• WireGuard's 148-byte handshake budget cannot absorb ML-KEM public keys/ciphertexts without either changing the data-plane format (native PQ forks) or moving PQC entirely to a companion protocol (Rosenpass) — the latter is the more conservative, more widely deployed choice today.
• IKEv2 Multi-KE (RFC 9370) solves the same problem structurally by spreading key-exchange methods across sequential IKE_INTERMEDIATE round trips, trading round-trip latency for guaranteed per-datagram size bounds.
• IP-layer fragmentation is not a safe fallback for oversized PQC handshake datagrams on the real internet: partial-fragment loss silently kills the entire message, middleboxes routinely drop non-initial fragments, and reassembly buffers are themselves a pre-authentication attack surface.
• Prefer protocol-level fragmentation (RFC 7383) or round-trip splitting (RFC 9370) over raw IP fragmentation, and set conservative tunnel MTUs to leave explicit headroom for PQC-inflated handshake payloads.