In T394402#10830072, @Tgr wrote:

No, that's the logging happenin in the logPersistenceChange() method.

In T394402#10827022, @Tgr wrote:

The Persisting... stuff was added for {T309943} & co, I don't think we need it anymore

I've added a few more sub-tasks which are about removing some of these logs:

• T379505: "Expected key {key} not found due to timeout" CentralAuth error is spammy
• T391646: Analyze "temporary logging for T372702" data and then stop logging it

In T393899#10814386, @Paladox wrote:

I've done: https://gerrit-review.googlesource.com/c/gerrit/+/474382

Visiting https://office.wikimedia.org/wiki/Homepage (as an anonymous user) and (hovering or) clicking on "Mobile view" at the bottom of the page (to switch to mobile view), results in me landing on a Special:Badtitle page.

In T390751#10814781, @matmarex wrote:

Sorry it took us a while to get to this. The fix would be deployed next week per the usual schedule. I wonder if we should backport it earlier?

In T393880#10810880, @Lucas_Werkmeister_WMDE wrote:

I assume this is due to MediaWikiEntryPoint: stop emitting legacy statsd metrics (T228380) which was just merged on the MediaWiki core master branch.

Can confirm, if I git revert b8146163eb locally then the test passes again.

One question I have is which login page are we supposed to show to the user in the SUL3 case? I think it's going to be the login form on the shared domain right? Not the local one I assume?

I was able to reproduce the /wiki/Special:UserLogin?usesul3=1 part (which doesn't present the login form but just re-triggers a login chain) but couldn't reproduce /wiki/Special:UserLogin?usesul3=0. The later will present to me a login form that I can use locally to authenticate as a different user.

Just checked this today and we're having over 18K entries on logstash and over 1M entries in the last 30 days. Seems to be very frequent recently.

I think this started off as part of T380500: CentralAuthUser returning outdated data after user creation when we were investigating the frequent logout issue. @Tgr already worked on fixing one leg of it and there are other things to fix but I doubt that it's related to the logging here.

Thanks for filing @thcipriani.

Let's reopen this if things go up on logstash. This is a warning and we've got 1 instance in the last 1 month.

In T390784#10728004, @Tgr wrote:

Microstash is cross-DC memcached, right? So in theory not the most reliable - could be evicting data because the slab is full. It does seem unlikely though.

In T375796#10704803, @Tgr wrote:

We might also want to swap the central autologin lookup order of loginwiki and auth.wikimedia.org at some point.

But we few more days is a better cadence for conclusion so let's wait till end of week and if we no longer see them, we can resolve the task. I'll check again tomorrow evening :)

Looking at https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/1078400/89/includes/Hooks/Handlers/PageDisplayHookHandler.php, we were already using /start here for non-JS clients.

@Tgr wrote:
Maybe the same as T380500: CentralAuthUser returning outdated data after user creation?

In T388177#10706509, @matmarex wrote:

We should wait a few days to see if that really resolves the errors: https://logstash.wikimedia.org/goto/69ae850bf878fc526f85273c8547494f

On enwiki (in prod), I can reproduce this by checking the dev tools and saw an anon session created but can't reproduce this locally.

Anything left to do here? Or can this be marked as resolved?

Reading the class level docblock of the ChronologyProtector, it states the following (extract):

See (logs snippet) of the Special:UserLogin/return&wpLoginToken... request which is what triggers continue authentication on the local wiki - from debug log

In T388177#10694196, @Krinkle wrote:

In theory that can't happen because of ChronologyProtector, which ensures that the replicas we use have caught up to our own previous writes, given that it will be the same browser that created the account and the same browser getting redirected to read the data.

If this turns out incorrect, we need to file an Rdbms bug and confirm whether there are any missing CP-cookies or CP-query params in this journey.

In T388177#10639731, @Tgr wrote:

This always seems to be happening on the local continueAuthentication() call, presumably right after account creation.

Hm! What should be the expected behavior here, I have some 3 ideas that come to mind:

This seems really weird, vaguely following the Special:ConfirmEmail code, I see that it sometimes call Special:UserLogin as a GET request. I think maybe this is when we some weird behavior when cookies on shared domain are set but not on local domain (when continueAuthentication is called) - maybe some timeout(?). Then that throws the logic error but I wonder why/when that should ever happen.

Thanks @Reedy, we've added some logging, let's see if we can get more info about this as the patch goes out.

In T388252#10613965, @Reedy wrote:

rECAU9f8ff1156fed: Special: Add support for central autologin in SUL3 mode made assertLocalWikiIsValid accept a nullable string, but WikiMap::getWiki isn't defined to take a null(able string)...

We had some issues today, filed T388222: Remove(?) Office wiki from DAL as it's not an SUL wiki

In T388172#10612015, @Tgr wrote:

Seems like a SessionManager bug - those sessions clearly don't have the same priority:

[b2fa3a34-b31a-4584-9900-06adcb3bde76] /w/rest.php/wikibase/v0/entities/lexemes/L1041834/senses   MediaWiki\Session\SessionOverflowException: Multiple sessions for this request tied for top priority: [100]MediaWiki\Extension\OAuth\SessionProvider<anon>h5cshtuck442udqpv2lsupnhp2u8buao, [50]CentralAuthSessionProvider<anon>rspba02td80u2kaju3teghvqm0tf11qi

But also, the priority of CentralAuthSessionProvider should be 11, not 50 (it's hardcoded). What's going on?

In T388177#10612820, @Tgr wrote:

Maybe one of the ids is zero and this is some version of T380500: CentralAuthUser returning outdated data after user creation.

In T388177#10611985, @Tgr wrote:

Only a single instance (logstash). Do we seriously not log the user name/ID when this happens? Ugh.
It's very much not supposed to happen, but without any specifics, can't really investigate.

@Tgr, do you want to sign this off or is there something else you want me to have a look?

In T387789#10600098, @Stashbot wrote:

Mentioned in SAL (#wikimedia-operations) [2025-03-04T10:35:39Z] <xSavitar> T387789 Ran mwscript-k8s --comment="T387789" -f -- extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=enwiki --logwiki=metawiki 'JamesVilla44' 'DartsF4' --ignorestatus

In T384007#10585015, @Tgr wrote:

In one of my tests I ended up logged out after a successful registration (but then top-level autologin worked). Not sure if that's a SUL3 bug that happens infrequently, or just generic authentication fragility. I thought I saw a log message about session metadata conflict, but now I can't find it.