Got a message from a #freifunk colleague that users are unable to change their password on our mailserver. They just get bounced back into the login form of our PostfixAdmin after submitting it.
Quick check: Yes, I have the same problem. Even the admin login is broken.
No idea when it broke.
#NixOS allows me to quickly activate an old configuration and software by executing a script (/nix/var/nix/profiles/system-476-link/bin/switch-to-configuration test), so I went back 15 days. That old generation worked. First success.
Switching only takes a couple of seconds unless you care about kernel, etc. which would require a reboot.
So finding the exact generation where it broke only took me about 5 minutes.
But what causes it? I already had a guess, as I saw which services changed, but I wanted to be sure:
nix-store -qR /nix/var/nix/profiles/system-476-link | sort -t- -k2 gives me the complete list of all included files and software in that configuration.
So I dumped the known-good and known-bad lists and diff'ed them.
/nix/store/...-dovecot-2.3.10.1 vs. /nix/store/...-dovecot-2.3.11.3
and a couple of unrelated libraries.
PostfixAdmin or PHP did not change. But PostfixAdmin uses Dovecot to check passwords, e.g. during login.
PostfixAdmin uses a simple command defined in the configuration file, so it should be easy to verify.
Of course it works as root, but as the user that PostfixAdmin is actually running:
[pfa@mail:~]$ /nix/store/...-dovecot-2.3.10.1/bin/doveadm pw -r 12
doveadm(pfadmin): Error: net_connect_unix(/var/run/dovecot/stats-writer) failed: Permission denied
Enter new password:
Retype new password:
{CRYPT}$2y$12$...
[pfa@mail:~]$ /nix/store/...-dovecot-2.3.11.3/bin/doveadm pw -r 12
doveconf: Fatal: Error in configuration file /etc/dovecot/dovecot.conf line 7: ssl_cert: Can't open file /var/lib/acme/mail.example.org/fullchain.pem: Permission denied
There's our culprit, Dovecot's new version breaks because it's unable to read a private key, which it doesn't even need for its current job.
Apparently it's a known issue in Dovecot, as it has been reported on the Dovecot mailinglist about a week ago: https://dovecot.org/pipermail/dovecot/2020-August/119642.html
There's even a workaround. Instead of specifying the ssl certificate in the config file, you move that part into a new config file that's only readable by root and use !include_try to include that file.
Easy, right? Well, NixOS requires all config files to be world-readable (for users on that system). So I modified the dovecot service to create that root-only config file before starting.
And PostfixAdmin is happy again and allows users to login and change their password.