resolve PMTU discovery issues on IPv6 #752
Comments
|
Disabling IPv6 is not an option.
… On Mar 22, 2024, at 13:34, David Lang ***@***.***> wrote:
Description
Google, linkedin posting, and other sites were inacceassable during scale 21x via IPv6
Owen identified this as a PMTU discovery incompatibility between Google and the HE tunnel
Acceptance Criteria
the HE IPv6 tunnel works with PMTU, including Google's implementation, or we find a way to override PMTU or we disable IPv6
—
Reply to this email directly, view it on GitHub <#752>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAK6GTVR3ZYZHBZBM6HVL5LYZSIVPAVCNFSM6AAAAABFD6NLRWVHI2DSMVQWIX3LMV43ASLTON2WKOZSGIYDGMRWG42TCNI>.
You are receiving this because you are subscribed to this thread.
|
|
It will be if a solution is not found by next year.
I don't think you understand the impact it has to Android devices, which is a popular subset of the Linux population, at a.. Linux conference.
________________________________
From: Owen DeLong ***@***.***>
Sent: Monday, March 25, 2024 10:39:15 AM
To: socallinuxexpo/scale-network ***@***.***>
Cc: Subscribed ***@***.***>
Subject: Re: [socallinuxexpo/scale-network] resolve PMTU discovery issues on IPv6 (Issue #752)
Caution: This is an external email and may be malicious. Please take care when clicking links or opening attachments.
Disabling IPv6 is not an option.
On Mar 22, 2024, at 13:34, David Lang ***@***.***> wrote:
Description
Google, linkedin posting, and other sites were inacceassable during scale 21x via IPv6
Owen identified this as a PMTU discovery incompatibility between Google and the HE tunnel
Acceptance Criteria
the HE IPv6 tunnel works with PMTU, including Google's implementation, or we find a way to override PMTU or we disable IPv6
—
Reply to this email directly, view it on GitHub <#752>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAK6GTVR3ZYZHBZBM6HVL5LYZSIVPAVCNFSM6AAAAABFD6NLRWVHI2DSMVQWIX3LMV43ASLTON2WKOZSGIYDGMRWG42TCNI>.
You are receiving this because you are subscribed to this thread.
—
Reply to this email directly, view it on GitHub<#752 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AACLU665JXQZSWYPZ4EXRU3Y2BOMHAVCNFSM6AAAAABFD6NLRWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMJYGU2DSNRQGA>.
You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>
|
|
Guys guys, we have a year. Let's use that time to pacify both the android
camp and the IPv6 camp.
On Mon, Mar 25, 2024 at 11:44 AM Ryan Hamel ***@***.***>
wrote:
… It will be if a solution is not found by next year.
I don't think you understand the impact it has to Android devices, which
is a popular subset of the Linux population, at a.. Linux conference.
________________________________
From: Owen DeLong ***@***.***>
Sent: Monday, March 25, 2024 10:39:15 AM
To: socallinuxexpo/scale-network ***@***.***>
Cc: Subscribed ***@***.***>
Subject: Re: [socallinuxexpo/scale-network] resolve PMTU discovery issues
on IPv6 (Issue #752)
Caution: This is an external email and may be malicious. Please take care
when clicking links or opening attachments.
Disabling IPv6 is not an option.
> On Mar 22, 2024, at 13:34, David Lang ***@***.***> wrote:
>
>
> Description
>
> Google, linkedin posting, and other sites were inacceassable during
scale 21x via IPv6
>
> Owen identified this as a PMTU discovery incompatibility between Google
and the HE tunnel
>
> Acceptance Criteria
>
> the HE IPv6 tunnel works with PMTU, including Google's implementation,
or we find a way to override PMTU or we disable IPv6
>
> —
> Reply to this email directly, view it on GitHub <
#752>, or
unsubscribe <
https://github.com/notifications/unsubscribe-auth/AAK6GTVR3ZYZHBZBM6HVL5LYZSIVPAVCNFSM6AAAAABFD6NLRWVHI2DSMVQWIX3LMV43ASLTON2WKOZSGIYDGMRWG42TCNI>.
> You are receiving this because you are subscribed to this thread.
>
—
Reply to this email directly, view it on GitHub<
#752 (comment)>,
or unsubscribe<
https://github.com/notifications/unsubscribe-auth/AACLU665JXQZSWYPZ4EXRU3Y2BOMHAVCNFSM6AAAAABFD6NLRWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMJYGU2DSNRQGA>.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
—
Reply to this email directly, view it on GitHub
<#752 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAFDVQDTJRGUCVEJJEG4NATY2BO6FAVCNFSM6AAAAABFD6NLRWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMJYGU2TONJVGU>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
|
|
If we continue to prioritize broken IPv6 over usable internet, all we are doing is reinforcing attendee perception that IPv6 isn't ready for prime time and that the first thing one should do when they notice network issues is: disable ipv6. If we want to drive ipv6 adoption and education through SCALE, we have to make sure ipv6 connectivity actually works and offers an equivalent or better experience to ipv4 only. if we can't then we just need to disable it. id hate to see that be the outcome, but with the current implementation we aren't meeting our attendees, speakers or sponsors needs. |
|
So, there's a helpful theory on this :
https://twitter.com/MyzaRed/status/1671835975801253889
It's been happening to people for over 10 years now and it appears to be
isolated to one of HE's range being blocked at Google's end. Either we work
with someone other than HE, or we get a different block from them.
H.
…On Tue, Mar 26, 2024 at 4:45 PM Ilan Rabinovitch ***@***.***> wrote:
If we continue to prioritize broken IPv6 over usable internet, all we are
doing is reinforcing attendee perception that IPv6 isn't ready for prime
time and that the first thing one should do when they notice network issues
is: disable ipv6.
If we want to drive ipv6 adoption and education through SCALE, we have to
make sure ipv6 connectivity actually works and offers an equivalent or
better experience to ipv4 only. if we can't then we just need to disable
it. id hate to see that be the outcome, but with the current implementation
we aren't meeting our attendees, speakers or sponsors needs.
—
Reply to this email directly, view it on GitHub
<#752 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAFDVQEOUZPT765XBZI66XLY2HT53AVCNFSM6AAAAABFD6NLRWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRRGU4TGOBXHE>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
|
This will probably solve it. Performance hit to everyone that can do proper PMTU-D, but hey, by all means, let's cater to Google our corporate overlords above all else: |
|
I don't like the solution either, but it is what it is, Owen. My phone did not function on the Wi-Fi at all this year, compared to previous, and including the Hilton. Add on many attendees and expo hall vendor phones, tablets, and various other gadgets. The experience needs to be good for all OS vendors.
This change needs to be tested before next year, or we'll be back to square one resulting in disabling V6 or using another provider like me.
Ryan Hamel
…________________________________
From: Owen DeLong ***@***.***>
Sent: Thursday, March 28, 2024 7:05 PM
To: socallinuxexpo/scale-network ***@***.***>
Cc: Ryan Hamel ***@***.***>; Comment ***@***.***>
Subject: Re: [socallinuxexpo/scale-network] resolve PMTU discovery issues on IPv6 (Issue #752)
Caution: This is an external email and may be malicious. Please take care when clicking links or opening attachments.
This will probably solve it. Performance hit to everyone that can do proper PMTU-D, but hey, by all means, let's cater to Google our corporate overlords above all else:
https://supportportal.juniper.net/s/article/Configuring-TCP-MSS-clamping-on-SRX-devices-to-avoid-unnecessary-fragmentation?language=en_US
—
Reply to this email directly, view it on GitHub<#752 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AACLU6ZOUOGOAAXXALXE2ETY2TD7PAVCNFSM6AAAAABFD6NLRWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRWGQ3TGMJQGA>.
You are receiving this because you commented.Message ID: ***@***.***>
|
|
If the problem was all apple devices not working would you be just as willing to
break them?
It's not that we are big fans of Google or that they control us, it's that a
large percentage of our users are dependent on them.
David Lang
…On Thu, 28 Mar 2024, Owen DeLong wrote:
This will probably solve it. Performance hit to everyone that can do proper PMTU-D, but hey, by all means, let's cater to Google our corporate overlords above all else:
https://supportportal.juniper.net/s/article/Configuring-TCP-MSS-clamping-on-SRX-devices-to-avoid-unnecessary-fragmentation?language=en_US
|
|
since we will have the tunnel up during the year, we can test it.
David Lang
…On Thu, 28 Mar 2024, Owen DeLong wrote:
This will probably solve it. Performance hit to everyone that can do proper PMTU-D, but hey, by all means, let's cater to Google our corporate overlords above all else:
https://supportportal.juniper.net/s/article/Configuring-TCP-MSS-clamping-on-SRX-devices-to-avoid-unnecessary-fragmentation?language=en_US
|
|
Yes, I would, as a matter of fact, but it turns out that Apple does PMTU-D correctly. Further, Ryan, if your phone wasn't working on the WiFi at the Hilton, this had NOTHING to do with IPv6 or problems on our network. We don't extend our network to the Hilton and the Hilton has ZERO IPv6 capability. Perhaps your phone just suffers from Android. Another provider won't help because we still won't be able to get a 1500 octet MTU through you, GRE is GRE and 6in4 is 6in4 and both have a certain amount of overhead that you can't get around. The MTU on the ethernet interface facing the convention center is limited to 1500 octets. They won't do jumbo frames (not like I didn't ask, but the response was something between a blank stare and "what's an MTU" or "what's a frame", or "jumbo what?"). This is not a surprise given the level of training I've observed among their on-site people. They're nice, they try to be helpful, but they really have very minimal training and understanding of networking. I'm actually less concerned about Android than I am about non-Android users trying to get to Google services from Linux devices, which was the problem we were able to observe and trace in the NOC. Android would be even harder to troubleshoot since it has a complete lack of troubleshooting tools (e.g. tcpdump or any other libpcap based capture tool) last I heard. If we want to test it, we'll need to add some equipment behind the tunnel and get a little creative. Doable, but not currently deployed. Right now, the tunnel is just idling on an interface on one of my MX-240s just to keep HE from deleting it. It's not actually moving real traffic or anything and I don't have an easy way to so without adding hardware. I can probably pull a spare SRX I have here into service rather than needing someone to ship our SRX devices. I have the replacement ex4200-48px from Hula already (same day replacement, no questions asked). It's probably a good idea to deploy that and get it tested anyway. I don't have anything that pretends to be Android, but I can probably through a pi at it and we can at least do some testing with that. |
|
On another note, I have good paths into Apple for getting bugs this serious resolved. Google, OTOH, is a black hole of uselessness when it comes to this sort of issue. |
|
Owen,
Further, Ryan, if your phone wasn't working on the WiFi at the Hilton, this had NOTHING to do with IPv6 or problems on our network. We don't extend our network to the Hilton and the Hilton has ZERO IPv6 capability. Perhaps your phone just suffers from Android.
That's not what I said at all... I said that my phone did not like the network this year, compared to the previous years which worked just fine (including the Hilton).
Anyway, if someone could meet me at the storage facility, then tell me what cart and bin where the firewall is, I can bring it to the colo in DTLA, and get it operational temporarily. When your tweaks are done, I can test with a USB Ethernet adapter connected to my phone, if that works for you. Please let me know your thoughts.
…________________________________
From: Owen DeLong ***@***.***>
Sent: Saturday, March 30, 2024 11:13:43 PM
To: socallinuxexpo/scale-network ***@***.***>
Cc: Ryan Hamel ***@***.***>; Comment ***@***.***>
Subject: Re: [socallinuxexpo/scale-network] resolve PMTU discovery issues on IPv6 (Issue #752)
Caution: This is an external email and may be malicious. Please take care when clicking links or opening attachments.
On another note, I have good paths into Apple for getting bugs this serious resolved. Google, OTOH, is a black hole of uselessness when it comes to this sort of issue.
—
Reply to this email directly, view it on GitHub<#752 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AACLU63CMR7ELV3FIFTRHILY26SRPAVCNFSM6AAAAABFD6NLRWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRYGU3TCMBSHE>.
You are receiving this because you commented.Message ID: ***@***.***>
|
|
If you weren’t having problems in years previous, including the Hilton, it wasn’t a change we made this year, because we didn’t make any IPv6 changes that would have affected that. We did have some problems with Google users in years past (though it wasn’t android phones so much as people trying to use Google apps on other systems).
… On Mar 31, 2024, at 05:11, Ryan Hamel ***@***.***> wrote:
Owen,
> Further, Ryan, if your phone wasn't working on the WiFi at the Hilton, this had NOTHING to do with IPv6 or problems on our network. We don't extend our network to the Hilton and the Hilton has ZERO IPv6 capability. Perhaps your phone just suffers from Android.
That's not what I said at all... I said that my phone did not like the network this year, compared to the previous years which worked just fine (including the Hilton).
Anyway, if someone could meet me at the storage facility, then tell me what cart and bin where the firewall is, I can bring it to the colo in DTLA, and get it operational temporarily. When your tweaks are done, I can test with a USB Ethernet adapter connected to my phone, if that works for you. Please let me know your thoughts.
________________________________
From: Owen DeLong ***@***.***>
Sent: Saturday, March 30, 2024 11:13:43 PM
To: socallinuxexpo/scale-network ***@***.***>
Cc: Ryan Hamel ***@***.***>; Comment ***@***.***>
Subject: Re: [socallinuxexpo/scale-network] resolve PMTU discovery issues on IPv6 (Issue #752)
Caution: This is an external email and may be malicious. Please take care when clicking links or opening attachments.
On another note, I have good paths into Apple for getting bugs this serious resolved. Google, OTOH, is a black hole of uselessness when it comes to this sort of issue.
—
Reply to this email directly, view it on GitHub<#752 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AACLU63CMR7ELV3FIFTRHILY26SRPAVCNFSM6AAAAABFD6NLRWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRYGU3TCMBSHE>.
You are receiving this because you commented.Message ID: ***@***.***>
—
Reply to this email directly, view it on GitHub <#752 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAK6GTVI4NHL77TTAQ3NTEDY274QHAVCNFSM6AAAAABFD6NLRWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRYGY4DKOJWGI>.
You are receiving this because you commented.
|
|
Well, when someone opens Chrome (default browser on non-modified Android installs), then wants to search a topic from an ongoing talk and stares at a loading bar, it doesn't look good either way.
________________________________
From: Owen DeLong ***@***.***>
Sent: Monday, April 1, 2024 2:23:14 PM
To: socallinuxexpo/scale-network ***@***.***>
Cc: Ryan Hamel ***@***.***>; Comment ***@***.***>
Subject: Re: [socallinuxexpo/scale-network] resolve PMTU discovery issues on IPv6 (Issue #752)
Caution: This is an external email and may be malicious. Please take care when clicking links or opening attachments.
If you weren’t having problems in years previous, including the Hilton, it wasn’t a change we made this year, because we didn’t make any IPv6 changes that would have affected that. We did have some problems with Google users in years past (though it wasn’t android phones so much as people trying to use Google apps on other systems).
On Mar 31, 2024, at 05:11, Ryan Hamel ***@***.***> wrote:
Owen,
> Further, Ryan, if your phone wasn't working on the WiFi at the Hilton, this had NOTHING to do with IPv6 or problems on our network. We don't extend our network to the Hilton and the Hilton has ZERO IPv6 capability. Perhaps your phone just suffers from Android.
That's not what I said at all... I said that my phone did not like the network this year, compared to the previous years which worked just fine (including the Hilton).
Anyway, if someone could meet me at the storage facility, then tell me what cart and bin where the firewall is, I can bring it to the colo in DTLA, and get it operational temporarily. When your tweaks are done, I can test with a USB Ethernet adapter connected to my phone, if that works for you. Please let me know your thoughts.
________________________________
From: Owen DeLong ***@***.***>
Sent: Saturday, March 30, 2024 11:13:43 PM
To: socallinuxexpo/scale-network ***@***.***>
Cc: Ryan Hamel ***@***.***>; Comment ***@***.***>
Subject: Re: [socallinuxexpo/scale-network] resolve PMTU discovery issues on IPv6 (Issue #752)
Caution: This is an external email and may be malicious. Please take care when clicking links or opening attachments.
On another note, I have good paths into Apple for getting bugs this serious resolved. Google, OTOH, is a black hole of uselessness when it comes to this sort of issue.
—
Reply to this email directly, view it on GitHub<#752 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AACLU63CMR7ELV3FIFTRHILY26SRPAVCNFSM6AAAAABFD6NLRWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRYGU3TCMBSHE>.
You are receiving this because you commented.Message ID: ***@***.***>
—
Reply to this email directly, view it on GitHub <#752 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAK6GTVI4NHL77TTAQ3NTEDY274QHAVCNFSM6AAAAABFD6NLRWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRYGY4DKOJWGI>.
You are receiving this because you commented.
—
Reply to this email directly, view it on GitHub<#752 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AACLU66QTKDHKTS4C7E456LY3HF4FAVCNFSM6AAAAABFD6NLRWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZQGU4DAMJSGQ>.
You are receiving this because you commented.Message ID: ***@***.***>
|
|
That won’t usually be a result of the PMTU problem we’ve observed. A partial page of results until the window size expands is far more likely. We mostly saw the problem in past years with apps (docs, sheets, etc. )If it’s more widespread this year, that’s interesting data, but not the result of any changes, since the only change we made this year was to reduce interface MTUs. On Apr 1, 2024, at 14:40, Ryan Hamel ***@***.***> wrote:
Well, when someone opens Chrome (default browser on non-modified Android installs), then wants to search a topic from an ongoing talk and stares at a loading bar, it doesn't look good either way.
________________________________
From: Owen DeLong ***@***.***>
Sent: Monday, April 1, 2024 2:23:14 PM
To: socallinuxexpo/scale-network ***@***.***>
Cc: Ryan Hamel ***@***.***>; Comment ***@***.***>
Subject: Re: [socallinuxexpo/scale-network] resolve PMTU discovery issues on IPv6 (Issue #752)
Caution: This is an external email and may be malicious. Please take care when clicking links or opening attachments.
If you weren’t having problems in years previous, including the Hilton, it wasn’t a change we made this year, because we didn’t make any IPv6 changes that would have affected that. We did have some problems with Google users in years past (though it wasn’t android phones so much as people trying to use Google apps on other systems).
On Mar 31, 2024, at 05:11, Ryan Hamel ***@***.***> wrote:
Owen,
> Further, Ryan, if your phone wasn't working on the WiFi at the Hilton, this had NOTHING to do with IPv6 or problems on our network. We don't extend our network to the Hilton and the Hilton has ZERO IPv6 capability. Perhaps your phone just suffers from Android.
That's not what I said at all... I said that my phone did not like the network this year, compared to the previous years which worked just fine (including the Hilton).
Anyway, if someone could meet me at the storage facility, then tell me what cart and bin where the firewall is, I can bring it to the colo in DTLA, and get it operational temporarily. When your tweaks are done, I can test with a USB Ethernet adapter connected to my phone, if that works for you. Please let me know your thoughts.
________________________________
From: Owen DeLong ***@***.***>
Sent: Saturday, March 30, 2024 11:13:43 PM
To: socallinuxexpo/scale-network ***@***.***>
Cc: Ryan Hamel ***@***.***>; Comment ***@***.***>
Subject: Re: [socallinuxexpo/scale-network] resolve PMTU discovery issues on IPv6 (Issue #752)
Caution: This is an external email and may be malicious. Please take care when clicking links or opening attachments.
On another note, I have good paths into Apple for getting bugs this serious resolved. Google, OTOH, is a black hole of uselessness when it comes to this sort of issue.
—
Reply to this email directly, view it on GitHub<#752 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AACLU63CMR7ELV3FIFTRHILY26SRPAVCNFSM6AAAAABFD6NLRWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRYGU3TCMBSHE>.
You are receiving this because you commented.Message ID: ***@***.***>
—
Reply to this email directly, view it on GitHub <#752 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAK6GTVI4NHL77TTAQ3NTEDY274QHAVCNFSM6AAAAABFD6NLRWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRYGY4DKOJWGI>.
You are receiving this because you commented.
—
Reply to this email directly, view it on GitHub<#752 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AACLU66QTKDHKTS4C7E456LY3HF4FAVCNFSM6AAAAABFD6NLRWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZQGU4DAMJSGQ>.
You are receiving this because you commented.Message ID: ***@***.***>
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
Google, linkedin posting, and other sites were inacceassable during scale 21x via IPv6
Owen identified this as a PMTU discovery incompatibility between Google and the HE tunnel
Acceptance Criteria
the HE IPv6 tunnel works with PMTU, including Google's implementation, or we find a way to override PMTU or we disable IPv6
The text was updated successfully, but these errors were encountered: