Paul's Site of Stuff ^..^

Ever since I [finally] got an IPv6 range for my home/business internet connection a few years back I have struggled to be able to access Microsoft services - either at all, or heavily delayed.

Every other online service I access works fine over IPv6. Everything. There is no problem with anything else. I cannot stress this enough. Everything else is fine. The only service with problems with my IPv6 is Microsoft services. Everything else is fine. Seriously.

Anyway...

Connection

I use Zen as my ISP. I've used them on and off over the last decade plus, but ultimately I have settled on them because they are one of the new providers who easily offer IPv6 address ranges.
Shockingly, it's still common in the UK for providers to completely ignore IPv6. I know! Mad!

Gateway / Router / Firewall

I was using an OPNSense appliance which is great firewall router but, due to reasons I'll get to in another thing I'm planning to write in the future, I have switched to Mikrotik devices. Still great, just less overhead for me basically.

Anyway, suffice to say I experienced the same issue on both of these gateways. That's enough for me to say "This is not a problem my side". Could I check with the ISP provided router? Probably, but no, my stuff is better, I don't want to check a cheapo free router.

Problem

The experience of the problem is that when trying to access any Microsoft service via a web browser the page takes ages to load, or more regularly, just completely fails and you're sat there watching a spinning wheel.

Looking at the web browser console I can see a lot of redirects to various other Microsoft services. Ultimately what's happening here is Microsoft are checking whether you either need to authenticate, or if you are already authenticated. It's in these places the connection stalls.

As I don't really access Microsoft services that often, I decided to not really look into it too much. Though, it kept niggling at me ("my network isn't perfect!") so I've investigated it from time to time.

Troubleshooting

As the first time I noticed this problem IPv6 was new on the network I performed a very rigorous and involved "turn IPv6 off" diagnosis. LibreWolf has a very nice "Enable IPv6" toggle in the settings, so that's what I did.
The site works over IPv4. OK, this is an IPv6 issue.

Onwards with further troubleshooting!

Do other sites work over IPv6?

Yes, all fine. OK, so it's not my IPv6, it's Microsoft not liking my IPv6 for some reason. Bastards.

Does it work on other internet supplies?

Yes. If I tether to mobile/cell the MS sites load fine over IPv6.

Can you ping6 the server in question?

I can ping things like teams[dot]microsoft[dot]com and admin[dot]microsoft[dot]com and so on, but if I extract the server name from the web console where things seems to be stalling, no I cannot. But they also could just be blocking pings.

Traceroute/mtr?

traceroute is sloooooow, mtr provides decent output - a few slow nodes, but typical "routers don't prioritise ICMP" knowledge applies here. Destination response time in approx 12ms. Decent.

OK, so I can "touch" the server, but still there's an issue...

Getting more detail

A little bit of web searching returns a few other people experiencing the same problem, but no real solutions. Future spoiler: If I'd have searched some better keywords I may have got my answer quicker!
Allegedly a solution from Microsoft directly, is to disable IPv6... HA! No thank you.

Verbose curl tests

curl -6v https://{ms-auth-server-in-question}

6 means use IPv6 only
v means output verbosely

When attempting this I see it stall precisesly after the line: * TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):. Could be a network issue, but running the same command numerous times at different points in the day over a few days/weeks/months (I said it wasn't urgent for me to solve this) yields the same results.

OK, so Microsoft don't respond after the cipher change. Interesting.

tcpdump

Lets run the same command while tcpdump is running then.

tcpdump -w output.pcap ipv6 and net {server IPv6 address}/128
I'm not going to pretend I can easily read tcpdump output, so I normally output (-w) to a pcap file and read it in wireshark or tshark with -r output.pcap, or you could just run: tshark ipv6 and net {server IPv6 address}/128 but remember...

Anyway, immediately after the Change Cipher line we can see the Microsoft are responding saying the previous segment was not captured.

10   0.047610 {MS IPv6} → {My IPv6} TLSv1.3 1326 [TCP Previous segment not captured] , Continuation Data

Somehow the data my computer is sending is not reaching Microsoft in a complete state.

Solution: MTU and PMTUD

A little bit more web searching for "[TCP Previous segment not captured]" eventually led me to this post which seems to be suffering mostly the same problem, just with Disney (which I have absolutely no intention of using!). Zen, IPv6, even Mikrotik - all the same as me.

The "problem" is MTU or "Maximum Transmission Unit" - in effect, it's a setting to declare the maximum size a data unit can be in a single network transaction.
Back in the day of ADSL connections in rural Devon, we had to tweak this setting quite a lot to make sure a farm at the end of a 5 mile copper cable to the nearest telecoms exchange could actually send some data, but I haven't dealt with it in some time now since infrastructure improved a little.

Anyway, the standard internet MTU is 1500. With PPPoE this typically goes down to 1492 to allow for the 8 byte PPPoE header. However, Zen do seem to follow RFC 4658 that allows for MTUs of 1508 (1500 + the 8 byte PPPoE header) so it Just Works™ for most websites.

PMTUD, or Path MTU Discovery, is a technique to determine the MTU size between two hosts. In summary, it adjusts the MTU to a value acceptable between the two hosts. Most operating systems support this technique and it should be operational.

My Mikrotik firewall does not have a MTU value set, so typically it would be 1500 with the allowance that PMTUD would adjust as necessary. This is a key point, with PPPoE my actual MTU would be 1508, unless PMTUD adjusted otherwise.

And there's the catch: Either Microsoft don't follow RFC4658 or PMTUD is broken when trying to access Microsoft services over IPv6. I don't know which, and I don't have the energy to find out, but long story short if I set my IPv6 MTU to 1492 (through IPv6 > ND > interface > MTU) Microsoft services web start working again. Set it to 1500, and they break. Back to 1492, working.
I'll leave it at 1492.

I'll fix it for me, but it's a Microsoft problem

I still believe this to be a Microsoft problem, even though only a handful of people seem to be experiencing this problem from what I can find on the internet would suggest otherwise ("if most people don't experience it, how can it be Microsoft!?").
I think the real kicker here is most people don't experience this issue because they're probably using their ISP provided router. That router probably has an MTU the ISP wishes to receive pre-set, which for PPPoE connections would be 1492, so it's never a problem. Myself, nucco, and likely others having this same issue, are using alternative routers. Due to this, there's some adjustment we need to make to suit Microsoft or Disney, even though every other website works fine. Still, setting the MTU to 1492 for IPv6 connections isn't the worst thing. I'm seeing it as more of a "Use PMTUD, but if it fails use 1492" which is sort of OK really.

I mentioned OPNSense earlier and the same thing applies here - it's an advanced router, so advanced configuration is required.

Just one of those things we need to do to keep control of our stuff I guess!