wgengine/router/osrouter: SNAT to the egress interface address, not (self) - #20738
Draft
mminkus wants to merge 2 commits into
Draft
wgengine/router/osrouter: SNAT to the egress interface address, not (self)#20738mminkus wants to merge 2 commits into
mminkus wants to merge 2 commits into
Conversation
…self) The FreeBSD subnet-router NAT rule translated with "-> (self)": nat on ! tailscale0 inet from 100.64.0.0/10 to any -> (self) In pf, "(self)" is a round-robin pool of every address on the machine, including tailscale0's own address and loopback, and pf deals each new state the next address in the pool. Only flows that happen to draw the egress interface's address work; a flow translated to any other address gets replies the far end cannot route, and hangs at SYN. With N usable addresses on the box, roughly (N-1)/N of connections through the subnet router silently fail. Observed in a natlab vmtest against a FreeBSD 15.0 subnet router with four addresses (WAN, LAN, QEMU debug NIC, tailscale0): exactly half of 8 HTTP requests hung, alternating, and the pf state table showed the failed flows translated to the debug NIC's address and to tailscale0's own address: 10.0.0.102:51100 (100.64.0.1:35132) -> 10.0.0.103:8080 ESTABLISHED 10.0.2.15:56553 (100.64.0.1:35148) -> 10.0.0.103:8080 SYN_SENT:CLOSED 100.64.0.2:52655 (100.64.0.1:54106) -> 10.0.0.103:8080 SYN_SENT:CLOSED On a production FreeBSD firewall running this branch, the equivalent IPv6 rule shows 55 state creations totalling 117 packets (about two packets per state): SYNs whose replies never came back. Emit one rule per up, non-loopback, non-Tailscale interface instead, translating to that interface's own address, per address family only where the interface holds a usable address of that family: nat on vtnet0 inet from 100.64.0.0/10 to any -> (vtnet0) nat on vtnet1 inet from 100.64.0.0/10 to any -> (vtnet1) which is also the rule form FreeBSD firewall operators write by hand. With this, the same 8-request test passes 8/8, and the LAN interface's rule shows 8 states with healthy packet counts (56 packets total). The interface set is sampled when SNAT is enabled; interfaces added later are not covered until SNAT is toggled or tailscaled restarts. Updates tailscale#5573 Change-Id: Ife3367124737ce5c8785ca7f920eafca593ec705 Signed-off-by: Martin Minkus <[email protected]>
The existing TestSubnetRouterFreeBSD makes a single HTTP request, which
cannot catch a NAT rule whose translation address pool is wrong on
average but right occasionally: with the previous "-> (self)" rule, pf
round-robins new states across every address on the machine, so one
request has decent odds of drawing the working address while most flows
hang. Add a test that opens several fresh flows and requires all of
them to work and to arrive at the backend from the router's LAN
address.
To let the test interrogate PF state on the router (rule counters and
the state table showing what each flow was translated to), add:
- a /exec handler to TTA that runs a shell command on the node, for
test diagnostics on nodes the harness has no SSH access to, and an
Env.Exec helper that calls it;
- Env.HTTPGetErr, a variant of HTTPGet that reports failure to the
caller instead of failing the test, so a test can treat a hung
request as data.
Fails 4/8 requests against the previous "-> (self)" NAT rule, with the
PF state table showing the failed flows translated to the QEMU debug
NIC's address and tailscale0's own address. Passes 8/8 with per-egress-
interface NAT rules.
Updates tailscale#5573
Change-Id: Ica94fba152a07705d47923ecedf50e1fafb857e2
Signed-off-by: Martin Minkus <[email protected]>
This was referenced Aug 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The FreeBSD subnet-router NAT rule translates with
-> (self):In pf,
(self)is a round-robin pool of every address on the machine -- including tailscale0's own address and loopback -- and pf deals each new state the next address in the pool. Only flows that draw the egress interface's address work. A flow translated to any other address gets replies the far end cannot route, and hangs at SYN. With N usable addresses on the box, roughly (N-1)/N of connections through the subnet router silently fail.Evidence
vmtest, FreeBSD 15.0 router with 4 addresses (WAN, LAN, QEMU debug NIC, tailscale0), 8 sequential HTTP requests: exactly half hang, alternating, and the pf state table names the wrong address each dead flow was translated to:
(10.0.0.102 = router LAN IP, works; 10.0.2.15 = QEMU debug NIC; 100.64.0.2 = the router's own tailscale0 address, despite the rule matching
on ! tailscale0-- the interface match constrains where the rule applies, not the address pool.)On one of our production FreeBSD firewalls running this branch, the IPv6 twin of this rule shows 55 state creations totalling 117 packets -- about two packets per state, i.e. SYNs whose replies never came back, in production, today.
This also explains why
TestSubnetRouterFreeBSDhas been maddeningly flaky: a single-request test is a per-run coin flip on which pool address the one flow draws.Fix
One rule per up, non-loopback, non-Tailscale interface, translating to that interface's own address (the form FreeBSD firewall operators write by hand), emitted per address family only where the interface holds a usable address of that family:
Same 8-request test: 8/8, backend sees only the router's LAN address, and the LAN interface's rule shows 8 states / 56 packets (healthy flows).
Known limitation, noted in a comment: the interface set is sampled when SNAT is enabled; hot-added interfaces aren't covered until SNAT toggles or tailscaled restarts.
Second commit: the test
TestSubnetRouterFreeBSDManyFlowsopens several fresh flows and requires all of them to work. Support pieces: a/exechandler in TTA plusEnv.Exec(run diagnostics on nodes the harness cannot SSH into -- the FreeBSD router here), andEnv.HTTPGetErr(treat a hung request as data instead of failing the test).Caveat: this branch carries an older snapshot of the vmtest package, from before main's qemu accel fallback / QMP retry / SSH key generation landed, so vmtests here flake under TCG (gokrazy VMs wedge at 100% CPU). The 4/8-vs-8/8 runs above were done with those harness fixes applied locally; merging current main into this branch brings them all in properly.
Split out per @bradfitz's request in #19312.