Potential bug in GRO code for Windows?

[thomaseizinger]

We are seeing an issue on a Windows machine where quinn-udp gives us packets larger than the configured MTU (2540 and 3810). Here is how we split the packets: https://github.com/firezone/firezone/blob/5437c3e2df001c8c491749c7bff0c063ccdf39e1/rust/socket-factory/src/lib.rs#L210-L255

This code has been working on Linux for several months and we have also not had reports that it doesn't work on Windows. Recently however a customer filed a bug report and looking through the logs, I discovered that we receive packets (i.e. DatagramIn in the above snippet) that are much larger than expected.

Going through quinn-udp's source code, I developed two theories and would be interested in hearing your thoughts on it:

1. stride is being read incorrectly (endianess-issue?) and instead of being smaller than len, it is bigger than len, meaning we never end up splitting the packets despite it being coalesced.

   quinn/quinn-udp/src/windows.rs

   Line 241 in d23e4e4

   stride = unsafe { cmsg::decode::<u32, WinSock::CMSGHDR>(cmsg) };

2. The buffer for control messages is too small, thus Windows fails to fit the UDP_COALESCED_INFO message into it and can't tell us that the buffer is coalesced even though it is.

Unfortunately, I've not been able to reproduce this on my own hardware :(

[stormshield-damiend]

Does the windows run on intel CPU or on ARM CPU ?

[mxinden]

We see a similar issue on Windows on ARM, see summary in last comment on this thread.

While I myself don't have access to a Windows on ARM machine, someone at Mozilla does and is able to reproduce the issue. We have not been able to reproduce this on a Windows on ARM VM.

The buffer for control messages is too small, thus Windows fails to fit the UDP_COALESCED_INFO message into it and can't tell us that the buffer is coalesced even though it is.

Interesting. I will try reproducing the issue with a larger buffer. For what it is worth, here is MSQUIC's buffer length.

[thomaseizinger]

Does the windows run on intel CPU or on ARM CPU ?

We see this issue on an x64_86 CPU.

[thomaseizinger]

The buffer for control messages is too small, thus Windows fails to fit the UDP_COALESCED_INFO message into it and can't tell us that the buffer is coalesced even though it is.

Interesting. I will try reproducing the issue with a larger buffer. For what it is worth, here is MSQUIC's buffer length.

I thought about this a bit more and I am no longer sure that is a viable possibility unless we receive a lot of other control messages that we don't match on.

[stormshield-damiend]

Could the receiver interface be a dual stack (IPv4 and IPv6) one ? I had no chance to test that case when writing the code, maybe we receive ipv4 + ipv6 control messages ?

[thomaseizinger]

We see a similar issue on Windows on ARM, see summary in last comment on this thread.

Interesting, according to https://bugzilla.mozilla.org/show_bug.cgi?id=1916558#c43, the UDP_COALESCED_INFO never gets received. We only get IP_PKTINFO and IP_ECN.

[thomaseizinger]

In https://learn.microsoft.com/en-us/windows-hardware/drivers/network/udp-rsc-offload, it is mentioned that this only works on Windows 11, version 24H2 and later. I'll have to investigate what version our customer is using.

[thomaseizinger]

Could the receiver interface be a dual stack (IPv4 and IPv6) one ? I had no chance to test that case when writing the code, maybe we receive ipv4 + ipv6 control messages ?

I can see in our logs that this customer did have dual-stack but I don't think it matters here because the way we do dual-stack is by having two sockets, one for IPv4 and one for IPv6.

[stormshield-damiend]

This code work fine on our side on Windows 10 and 11.

[stormshield-damiend]

I do not see any special case related to this in msquic C code https://github.com/microsoft/msquic/blob/main/src/platform/datapath_winuser.c#L3216

[thomaseizinger]

@mxinden Do you know by any chance if the affected user is using a dock? We have reports that this only happens when the user is connected via a Dell dock WD19S. It works fine over WiFi and via a separate Ethernet dongle.

[stormshield-damiend]

@mxinden could your colleague print the stride value (at L210) before the loop ? it is init with the length of the received packet to handle the case were received packet is not coalesced ?

[saschanaz]

The colleague here 👋

I'm not using any dock but only the builtin WiFi, and the stride value is 1280.

[mxinden]

and the stride value is 1280.

Just to avoid any confusion, the stride value is 1280 AND the datagram length value is 1280. This is with the unit test I provided in this commit, see also the logs here. That said, I have the suspicion that the unit test simply doesn't force datagram coalescing in the kernel, see this bugzilla comment.

Looking at pcaps with fosstodon.org that @saschanaz provided, we see >1_500k UDP datagrams over an internet path. >1_500k over an internet path is very unlikely. My suspicion is, that the Windows kernel correctly coalesces the UDP datagrams (< 1_500k) into one >1_500k datagram, but then either does not provide the stride length, or we fail to read the stride length.

[stormshield-damiend]

I think you wanted to link this commit mxinden@7ce09c3