In the previous blog post we have explored the mechanisms QUIC uses to reject packets.

Now we want to see how this works in practice.

A test handler

We start with an iroh endpoint that serves a simple echo protocol and counts completed requests.

This endpoint runs as a standalone binary. You can configure the endpoint id in the usual way with an IROH_SECRET environment variable. The binary has a cli argument to shut down after either n total or n completed requests.

It will also measure its own CPU time on shutdown, using the cpu-time crate.

Using this we can measure not just the wallclock execution time but also how much load the server was under.

Reference case: just valid requests

To establish a baseline, we just do a number of echo requests in parallel and look at the CPU usage and wallclock time:

echo: 100 accepted,   0 rejected,   0 closed  (client: 42.18ms | server: 127.98ms cpu,      781 ops/s, 45.80ms wall)

So each cpu core can handle 781 complete new handshakes and echo requests per second in terms of raw CPU time.

Note that these are fresh connection attempts. Also this is with 100 clients on running on the same machine, so the absolute number isn't that meaningful. We mostly care about relative numbers.

Now let's imagine our poor echo service is under load or under a denial of service attack. We will see at which places we can reject incoming connection attempts, and how many we can reject.

Rejection hooks

Direct requests

The first incoming datagram will contain an initial packet without retry token. At this point we don't know if the sender address is valid. This is the first opportunity to reject.

We could reject here based on the unverified sender address, or just to shed load.

The options here are ignore (silently dropping the connection attempt), retry (instruct the sender to retry with a token), and reject.

ignore addr:   0 accepted, 100 rejected,   0 closed  (client:   5.00s | server:  3.65ms cpu,    27427 ops/s, 16.07ms wall)
reject addr:   0 accepted, 100 rejected,   0 closed  (client: 11.36ms | server:  9.64ms cpu,    10370 ops/s, 14.32ms wall)
retry + reject:   0 accepted, 100 rejected,   0 closed  (client: 33.83ms | server:  9.58ms cpu,    10434 ops/s, 36.34ms wall)

Unsurprisingly, ignore is fastest at 27427 ops/s per core. But also not very useful except for unconditional load shedding.

Reject is a bit more polite, but for spoofed sender addresses it might not even reach the sender.

Retry+reject is the first rejection mode that is genuinely useful for selective filtering or rate limiting. After a retry we know the source address is correct, so we can e.g. rate limit by ip address or even filter by region.

Even retry+reject is much faster than going through with the request at ~10434 ops/s per core.

The last opportunity to reject direct requests is to reject based on the proposed ALPNs. This is useful not so much in a denial of service scenario, but if you serve different ALPNs and want to prioritize one in times of high load.

reject alpn:   0 accepted, 100 rejected,   0 closed  (client: 10.92ms | server:  8.28ms cpu,    12085 ops/s, 14.78ms wall)

Since the proposed ALPNs are available in the Initial packet, this rejection is almost as cheap as the other early filters.

Requests via relays

For requests via relays we don't have a source address that we can filter or rate limit on. What we have instead is the endpoint id as reported by the relay.

This id can be relied on only if the relay isn't lying to us, so it is a bit similar to an unverified source ip address. But it is much harder for an attacker to arrange for a compromised relay than to just forge the sender in UDP datagrams, so it is a bit more useful for filtering.

ignore eid:   0 accepted, 100 rejected,   0 closed  (client:   5.01s | server:  6.64ms cpu,    15072 ops/s, 34.91ms wall)
reject eid:   0 accepted, 100 rejected,   0 closed  (client: 17.54ms | server:  8.20ms cpu,    12194 ops/s, 21.11ms wall)
retry + reject eid:   0 accepted, 100 rejected,   0 closed  (client: 38.09ms | server: 17.51ms cpu,     5711 ops/s, 44.23ms wall)

We can also filter on the proposed ALPNs, which are available in the Initial packet just like for direct connections.

reject alpn:   0 accepted, 100 rejected,   0 closed  (client: 18.38ms | server:  9.97ms cpu,    10028 ops/s, 22.80ms wall)

Again, this is cheap and comparable to the other early filters.

Combining valid requests with spam

The last step is to try to communicate with our echo service while it is being subjected to a denial of service attack.

  echo: 100/117 accepted, 17 rejected
  server wall time:   2.64s
  server cpu time:  397.85ms
  spam packets sent: 11700
  server incoming:   2543
  server filtered:   2543
  server completed:  100
  effective spam/request: 117:1

In this test we spam the echo service binary with 100 fake ClientHello per valid echo request. Each ClientHello is valid, so it will cause the endpoint to respond with a ServerHello that will be ignored.

Not all spam packets actually arrive at the server, several are dropped by the operating system (it is allowed to do this for UDP datagrams).

But even so, we got 25 times as many fake requests arriving at the endpoint than real requests, and nevertheless all valid echo requests are getting through in a reasonable time (4.0ms of server core time per request).

Adding a firewall rule

Now let's assume we have an endpoint exposed to the internet that has long lived active connections.

We want to minimize the impact of possibly malicious incoming connections while leaving existing connections mostly unaffected.

On linux this can be done with a simple nftables rule:

nft add rule inet filter input udp dport 12345 @th,64,1 == 1 limit rate over 1000/second drop

@th,64,1 inspects 1 bit at offset 64 from the start of the transport (UDP) header. Since the UDP header is 8 bytes, this is the very first bit of the UDP payload — which is the QUIC form bit. A value of 1 means it is a long header packet, i.e. a new connection attempt.

So this rule rate limits new QUIC connections to 1000 per second on port 12345, dropping the excess. Short header packets (form bit 0) used by established connections are not affected at all.