Skip to content

net/netcheck,wgengine/magicsock: don't rebind when IPv4 was never probed - #20722

Open
BastienClement wants to merge 1 commit into
tailscale:mainfrom
BastienClement:magicsock-nov4send-rebind
Open

net/netcheck,wgengine/magicsock: don't rebind when IPv4 was never probed#20722
BastienClement wants to merge 1 commit into
tailscale:mainfrom
BastienClement:magicsock-nov4send-rebind

Conversation

@BastienClement

Copy link
Copy Markdown

What this does

Adds netcheck.Report.IPv4SendAttempted, recording whether an IPv4 STUN send was actually tried, and changes magicsock to set noV4Send only when a send was tried and failed.

Fixes #16755
Updates #19791

Why

On a host with no usable IPv4 address, makeProbePlan gates IPv4 probes on ifState.HaveV4 and plans none. Report.IPv4CanSend is written true in exactly one place, in runProbe after a send, so it can never become true on such a host. magicsock stored noV4Send = !report.IPv4CanSend and updateEndpoints read that as "the IPv4 socket cannot transmit", so every netcheck called Rebind(). From an IPv6-only EKS pod on #16755:

magicsock: last netcheck reported send error. Rebinding.
Rebind; defIf="v4if0", ips=[169.254.172.3/22 fe80::683a:f1ff:fec8:1aa5/64]
magicsock: closing connection to derp-12 (rebind-default-route-change), age 23s

Periodic re-STUN is suppressed once the node has been idle past sessionActiveTimeout, so the loop runs at 20-26s intervals precisely while there is traffic. Where the default route sits on an interface that does not hold the socket's address, each cycle also closes every DERP connection and new connections stall while the loop runs.

Reported on IPv6-only VMs (#16755), Google Cloud Run (#19791, #20330), and IPv6-only EKS pods where aws-vpc-cni provides IPv4 egress through a link-local address. The thread on #16755 has a measurement of 12 rebinds in 5 minutes against 0 once the link is made to report IPv4.

The rebind exists for #2994, where a socket is left bound to a departed interface after a network reconfiguration. That case is unaffected: the link still has a usable IPv4 address, so probes are planned, the send is attempted and fails, and noV4Send is still set.

Tests

Two new tests, both mutation-checked with go test -overlay (working tree left untouched):

  • TestIPv4SendAttempted (net/netcheck) covers send-ok, send-failed, EPERM and no-IPv4-node. Dropping the runProbe change fails the first three; dropping the TreatAsLostUDP tolerance fails the EPERM case.
  • TestUpdateNetInfoNoV4Send (wgengine/magicsock) asserts noV4Send end to end through updateNetInfo. Restoring noV4Send = !report.IPv4CanSend fails no_v4_probe with noV4Send = true; want false.

netmon.Monitor exposes no way to inject interface state, so the new tests reach the empty IPv4 probe set through a DERP node without an IPv4 address rather than through HaveV4. The existing TestMakeProbePlan/only_v6_initial covers the other half, that HaveV4 == false yields a plan with no IPv4 probes.

Also run locally: go test -race -count=2 over net/netcheck and wgengine/magicsock, go test -race over cmd/tailscale/cli and ipn/ipnlocal (the other netcheck.Report consumers), staticcheck for darwin/arm64, windows/amd64 and linux/amd64, make depaware, go mod tidy with no diff, and cross-builds for plan9, solaris, illumos, aix, js/wasm, windows amd64 and arm64, darwin, freebsd, openbsd and linux/arm.

On a host with no usable IPv4 address, makeProbePlan plans no IPv4 STUN
probes, so Report.IPv4CanSend can never become true. magicsock read that
as "the IPv4 socket cannot transmit" and rebound every 20-26s for the
life of the process.

Record whether an IPv4 send was attempted at all, and treat only a send
that was tried and failed as a broken socket. The rebind still fires in
the case it was added for: a socket left bound to a departed interface
still has IPv4 probes planned, so the send is attempted and fails.

RELNOTE: stop magicsock from rebinding every ~20s on hosts with no
usable IPv4 address

Fixes tailscale#16755
Updates tailscale#19791

Change-Id: I93359937485d475b4197278e778123b4e0435117
Signed-off-by: Bastien Clément <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Perpertual "last netcheck reported send error. Rebinding." on IPv6 only machine

1 participant