How does this related to existing acl*release_security_pre_announce?

Done, and documented at https://wikitech.wikimedia.org/wiki/Help:Email_in_Cloud_VPS.

No, not really. Some of that data comes from etcd, but there are also files like https://config-master.wikimedia.org/known_hosts that are generated from PuppetDB and quite useful to have on deployment-prep as well. I believe this was previously applied to the puppetmaster VM, but that was lost in the Puppet 5 -> 7 migration. On the wikiprod side this service was split to a tiny separate VM and that's a route deployment-prep could use as well.

This is fixed, and I moved all the existing rules to the correct OU.

This is due to a hardware issue with one of the hosts involved in the replication chain to the wiki replicas: T394624: db1155 HW memory errors

Yes please.

Thanks for the poke. This host seems to have accidentally gotten rebooted back to the broken kernel from T393366: Regression in RAID10 software RAID with 6.1.135 and was in a locked up state. I've rebooted it and upgraded it to a working kernel which should resolve those backup issues.

In T382171#10834959, @isarantopoulos wrote:

checking on quarry.wmcloud.org I see that the table isn't there

In T376400#10834856, @Andrew wrote:

Can you point me to some specific examples? My half-baked spot checks (e.g. http://ec2-54-81-201-239.compute-1.amazonaws.com/wiki/Eqiad_data_center.html#/media/File:Eqiad_logical.png) seem to be hosted locally (or I am misunderstanding something fundamental).

In T332478#10747989, @dcaro wrote:

@taavi should I close this as duplicate? Or do you want to refresh/extend the oauth+dedicated auth server specific proposal?

In T394035#10823194, @fnegri wrote:

Option Purple is my favourite so far, but I'm still a bit confused about how the new service would look like. Apologies if I'm fixating on this, we can also split this topic into a separate Decision Request or Design Document.

Huh. In that case probably worth noting that I'm actually not sure if that particular account is set to English, I just saw a message with mixed languages and figured that is a bug.

Hmm, are you intentionally using a new developer account for this instead of the existing 'mbinder' one that your Phabricator shell access is already associated with?

Re-opening, because that doesn't really solve the problem. We do not want to send Puppet failure emails to these service accounts in the first place.

This extension is lacking a formal steward on https://www.mediawiki.org/wiki/Developers/Maintainers, I do not believe it is a good idea to deploy it more widely unless that is addressed.

In T290147#10751338, @JoelyRooke-WMDE wrote:

@taavi As the Wikidata Integrations in Wikimedia Projects team, we are happy to review any new changes on the topic (although I can't +2 config changes). I can see you have an open mediawiki-config change, but it's difficult to review without knowing what the next steps will be. I can see you've written some changes are pending answers from T171143, what questions are those, and what impact may the answers make?

The site at http://ec2-54-81-201-239.compute-1.amazonaws.com/ seems to embed images from upload.wikimedia.org, for pages like network diagrams those should also be hosted off-cluster.

Not really, since the train moving forward has made backporting to a wmf branch more or less moot at this point.

In T394035#10817585, @fnegri wrote:

it's better to make a Toolforge-specific service than to make something that's in theory generic but in practice only used by us

Agreed! But we should consider carefully the requirements for this service and its interface. I think I would like this service to be as "thin" as possible, does it need to contain Toolforge-specific logic? Could it be a generic LDAP adapter, with some minimal logic to restrict the damage you can do through its API?

In T394035#10820913, @fnegri wrote:

Could it be a generic LDAP adapter, with some minimal logic to restrict the damage you can do through its API?

This would increase the security issues with that service, as would expand the possible actions (just saying, that might make it harder to harden).

Yes, that's fair. What are the write actions that the API needs to perform against LDAP? From a quick search in the Striker codebase, I can only see adding and deleting SSH keys. I imagine it also needs to modify LDAP group membership, but I can't find that in the Striker codebase.

This is worked around for now by pinning a commit hash from https://github.com/supertassu/django-ratelimit-backend until T359554: Use IDP for authentication in Striker happens.

In T394035#10817110, @fnegri wrote:

In option Purple, would it be possible to deploy the new "LDAP API" service to wikiprod hardware

Yes.

In T394035#10816904, @aborrero wrote:

I can definitely see the benefit of the HTTP API, way easier to use than the LDAP interface.

But also, I'm trying to think, how is exposing the REST API different than exposing the LDAP URI for RW directly? It may be the same level of security/auth/authn complications, etc. Or maybe better, because there is usually better tooling, middleware and such, for HTTP.

I'm not seeing any opposition, so went ahead and implemented this. And wikitech-l post for the records.

taavi@tools-bastion-12:~ $ kubectl sudo delete cm -n tool-inat2wiki maintain-kubeusers-inat2wiki
configmap "maintain-kubeusers-inat2wiki" deleted

The system will now provision new files within a few minutes.

The Keystone update done in https://gerrit.wikimedia.org/r/c/labs/striker/+/1145140 fixes the issue at least for me.