Upgrade of CRI-O from 1.34 to 1.35 breaks pod running systemd with `hostUsers: false`

### What happened?

In https://github.com/freeipa/freeipa-container we run systemd in container. We test on numerous setups and runtimes, one of them being CRI-O. When I tried to update the test to use CRI-O 1.35 instead of 1.34, the tests started to fail.

I tried to minimize the reproducer of the issue on a fresh Ubuntu 24.04 VM, and upgraded to latest versions of all packages, and found the problem deterministic with CRI-O 1.35 and Kubernetes 1.34 and 1.35, both a fresh installation of CRI-O 1.35, and upgrade of CRI-O from 1.34 to 1.35. I saw the problem both with Kubernetes installed using `kubeadm init` which is what we will show bellow, and with RKE2 and K0s.

### What did you expect to happen?

I expected my pod
```yaml
apiVersion: v1
kind: Pod
metadata:
  name: test-systemd
  labels:
    app: test-systemd
spec:
  restartPolicy: Always
  hostUsers: false
  containers:
  - name: test-systemd
    image: registry.access.redhat.com/ubi10/ubi-init
    imagePullPolicy: IfNotPresent
    securityContext:
      readOnlyRootFilesystem: true
    tty: true
```
to run on a Kubernetes node with CRI-O 1.35 just as well as it did with 1.34.

### How can we reproduce it (as minimally and precisely as possible)?

Have a fresh Ubuntu 24.04 VM with packages upgraded to latest versions:
```
sudo apt update
sudo apt upgrade -y
sudo reboot
```

We will install CRI-O 1.34:
```
export CRIO_VERSION=v1.34
curl -fsSL https://download.opensuse.org/repositories/isv:/cri-o:/stable:/$CRIO_VERSION/deb/Release.key \
        | gpg --dearmor | sudo tee /etc/apt/keyrings/cri-o-apt-keyring.gpg > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/cri-o-apt-keyring.gpg] https://download.opensuse.org/repositories/isv:/cri-o:/stable:/$CRIO_VERSION/deb/ /" \
        | sudo tee /etc/apt/sources.list.d/cri-o.list

sudo apt update
sudo apt install -y cri-o

sudo cp /etc/cni/net.d/10-crio-bridge.conflist.disabled /etc/cni/net.d/10-crio-bridge.conflist
sudo systemctl start crio.service
```

We will install Kubernetes 1.34:
```
export KUBERNETES_VERSION=v1.34
curl -fsSL https://pkgs.k8s.io/core:/stable:/$KUBERNETES_VERSION/deb/Release.key \
        | gpg --dearmor | sudo tee /etc/apt/keyrings/kubernetes-apt-keyring.gpg > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/$KUBERNETES_VERSION/deb/ /" \
        | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt update
sudo apt install -y kubelet kubeadm kubectl kubernetes-cni

sudo modprobe br_netfilter
sudo sysctl -w net.ipv4.ip_forward=1
sudo iptables -A FORWARD -o cni0 -j ACCEPT

cat <<'EOF' > k8s-initconfiguration-crio.yaml
---
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
networking:
  serviceSubnet: 10.43.0.0/16
---
apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
nodeRegistration:
  criSocket: unix:///var/run/crio/crio.sock
EOF
sudo kubeadm init --config k8s-initconfiguration-crio.yaml

mkdir ~/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $( id -u ):$( id -g ) ~/.kube/config

kubectl taint nodes $( hostname ) node-role.kubernetes.io/control-plane:NoSchedule-
```

We will create a Pod running systemd:
```
cat <<'EOF' > test-systemd-k8s.yaml
apiVersion: v1
kind: Pod
metadata:
  name: test-systemd
  labels:
    app: test-systemd
spec:
  restartPolicy: Always
  hostUsers: false
  containers:
  - name: test-systemd
    image: registry.access.redhat.com/ubi10/ubi-init
    imagePullPolicy: IfNotPresent
    securityContext:
      readOnlyRootFilesystem: true
    tty: true
EOF
kubectl apply -f test-systemd-k8s.yaml
kubectl get pod/test-systemd
kubectl logs pod/test-systemd
```

We will observe that the systemd is running fine:
```
systemd 257-13.el10-g833aa52 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +IPE +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT -QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +BTF +XKBCOMMON +UTMP +SYSVINIT +LIBARCHIVE)
Detected virtualization container-other.
Detected architecture x86-64.

Welcome to Red Hat Enterprise Linux 10.1 (Coughlan)!

Initializing machine ID from random generator.
Failed to mount /run/machine-id (type n/a) on /etc/machine-id (MS_BIND ""): Permission denied
bpf-restrict-fs: BPF LSM hook not enabled in the kernel, BPF LSM not supported.
Queued start job for default target graphical.target.
[  OK  ] Created slice system-modprobe.slice - Slice /system/modprobe.
[  OK  ] Started systemd-ask-password-conso&hellip;equests to Console Directory Watch.
[  OK  ] Started systemd-ask-password-wall.&hellip;d Requests to Wall Directory Watch.
[  OK  ] Reached target local-fs.target - Local File Systems.
[  OK  ] Reached target network-online.target - Network is Online.
[  OK  ] Reached target paths.target - Path Units.
[  OK  ] Reached target remote-fs.target - Remote File Systems.
[  OK  ] Reached target slices.target - Slice Units.
[  OK  ] Reached target swap.target - Swaps.
[  OK  ] Listening on systemd-creds.socket - Credential Encryption/Decryption.
[...]
```

We will upgrade CRI-O to 1.35:
```
export CRIO_VERSION=v1.35
curl -fsSL https://download.opensuse.org/repositories/isv:/cri-o:/stable:/$CRIO_VERSION/deb/Release.key \
        | gpg --dearmor | sudo tee /etc/apt/keyrings/cri-o-apt-keyring.gpg > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/cri-o-apt-keyring.gpg] https://download.opensuse.org/repositories/isv:/cri-o:/stable:/$CRIO_VERSION/deb/ /" \
        | sudo tee /etc/apt/sources.list.d/cri-o.list

sudo apt update
sudo apt upgrade -y cri-o
```

We will check the systemd Pod:
```
$ kubectl get pod/test-systemd
NAME           READY   STATUS   RESTARTS     AGE
test-systemd   0/1     Error    2 (2s ago)   117s
$ kubectl logs pod/test-systemd
systemd 257-13.el10-g833aa52 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +IPE +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT -QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +BTF +XKBCOMMON +UTMP +SYSVINIT +LIBARCHIVE)
Detected virtualization container-other.
Detected architecture x86-64.

Welcome to Red Hat Enterprise Linux 10.1 (Coughlan)!

Initializing machine ID from random generator.
Failed to mount /run/machine-id (type n/a) on /etc/machine-id (MS_BIND ""): Permission denied
Failed to create /init.scope control group: Permission denied
Failed to allocate manager object: Permission denied
[!!!!!!] Failed to allocate manager object.
Exiting PID 1...
```

### Anything else we need to know?

When I check `journalctl -l`, I don't see anything suspicious.

Diffing the `Current CRI-O configuration:` lines for CRI-O, version: 1.34.4, git: cc8e8609ed352d8e34b12e8a659fe4dd3bbb7892(dirty) where things work vs. CRI-O, version: 1.35.0, git: 92c18a2e2673764cd10f89b1a5061e2b26f44209(dirty), after unpacking the newlines, gives me just
```diff
@@ -80,6 +80,7 @@
         monitor_cgroup = \"system.slice\"
         container_min_memory = \"12MiB\"
         no_sync_log = false
+        container_create_timeout = 240
       [crio.runtime.runtimes.runc]
         runtime_config_path = \"\"
         runtime_path = \"/usr/libexec/crio/runc\"
@@ -89,6 +90,7 @@
         monitor_cgroup = \"system.slice\"
         container_min_memory = \"12MiB\"
         no_sync_log = false
+        container_create_timeout = 240
   [crio.image]
     default_transport = \"docker://\"
     global_auth_file = \"\"
```

Note to self, I used
```
$ sudo journalctl -l | grep -E '(Starting|Current) CRI-O' | sed '/version: 1.35.0/,$d' | grep 'Current CRI-O configuration:' | tail -1 | sed 's/\\n/\n/g' > /tmp/cri-o-1.34.config
$ sudo journalctl -l | grep -E '(Starting|Current) CRI-O' | sed '1,/version: 1.35.0/d' | grep 'Current CRI-O configuration:' | head -1 | sed 's/\\n/\n/g' > /tmp/cri-o-1.35.config
```
to get the last 1.34 and first 1.35 configuration information.

### CRI-O and Kubernetes version

<details>

```console
$ crio --version
crio version 1.35.0
   GitCommit:      92c18a2e2673764cd10f89b1a5061e2b26f44209
   GitCommitDate:  2025-12-22T19:51:19Z
   GitTreeState:   dirty
   BuildDate:      1970-01-01T00:00:00Z
   GoVersion:      go1.25.0
   Compiler:       gc
   Platform:       linux/amd64
   Linkmode:       static
   BuildTags:
     static
     netgo
     osusergo
     exclude_graphdriver_btrfs
     seccomp
     apparmor
     selinux
   LDFlags:          unknown
   SeccompEnabled:   true
   AppArmorEnabled:  false
```

```console
$ kubectl version --output=json
{
  "clientVersion": {
    "major": "1",
    "minor": "34",
    "gitVersion": "v1.34.3",
    "gitCommit": "df11db1c0f08fab3c0baee1e5ce6efbf816af7f1",
    "gitTreeState": "clean",
    "buildDate": "2025-12-09T15:06:39Z",
    "goVersion": "go1.24.11",
    "compiler": "gc",
    "platform": "linux/amd64"
  },
  "kustomizeVersion": "v5.7.1",
  "serverVersion": {
    "major": "1",
    "minor": "34",
    "emulationMajor": "1",
    "emulationMinor": "34",
    "minCompatibilityMajor": "1",
    "minCompatibilityMinor": "33",
    "gitVersion": "v1.34.3",
    "gitCommit": "df11db1c0f08fab3c0baee1e5ce6efbf816af7f1",
    "gitTreeState": "clean",
    "buildDate": "2025-12-09T14:59:13Z",
    "goVersion": "go1.24.11",
    "compiler": "gc",
    "platform": "linux/amd64"
  }
}
```

</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.3 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo
$ uname -a
Linux crio-135-4-ubuntu-24-04.redacted.com 6.8.0-90-generic #91-Ubuntu SMP PREEMPT_DYNAMIC Tue Nov 18 14:14:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
```

</details>


### Additional environment details (AWS, VirtualBox, physical, etc.)

<details>
A libvirtd VM.
</details>


Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade of CRI-O from 1.34 to 1.35 breaks pod running systemd with `hostUsers: false` #9705

What happened?

What did you expect to happen?

How can we reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

CRI-O and Kubernetes version

OS version

Additional environment details (AWS, VirtualBox, physical, etc.)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Upgrade of CRI-O from 1.34 to 1.35 breaks pod running systemd with hostUsers: false #9705

Description

What happened?

What did you expect to happen?

How can we reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

CRI-O and Kubernetes version

OS version

Additional environment details (AWS, VirtualBox, physical, etc.)

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Upgrade of CRI-O from 1.34 to 1.35 breaks pod running systemd with `hostUsers: false` #9705