Random texts

nixos

Got a message from a #freifunk colleague that users are unable to change their password on our mailserver. They just get bounced back into the login form of our PostfixAdmin after submitting it. Quick check: Yes, I have the same problem. Even the admin login is broken. No idea when it broke. #NixOS allows me to quickly activate an old configuration and software by executing a script (/nix/var/nix/profiles/system-476-link/bin/switch-to-configuration test), so I went back 15 days. That old generation worked. First success. Switching only takes a couple of seconds unless you care about kernel, etc. which would require a reboot. So finding the exact generation where it broke only took me about 5 minutes. But what causes it? I already had a guess, as I saw which services changed, but I wanted to be sure: nix-store -qR /nix/var/nix/profiles/system-476-link | sort -t- -k2 gives me the complete list of all included files and software in that configuration. So I dumped the known-good and known-bad lists and diff'ed them. /nix/store/...-dovecot-2.3.10.1 vs. /nix/store/...-dovecot-2.3.11.3 and a couple of unrelated libraries. PostfixAdmin or PHP did not change. But PostfixAdmin uses Dovecot to check passwords, e.g. during login. PostfixAdmin uses a simple command defined in the configuration file, so it should be easy to verify. Of course it works as root, but as the user that PostfixAdmin is actually running:

[pfa@mail:~]$ /nix/store/...-dovecot-2.3.10.1/bin/doveadm pw -r 12
doveadm(pfadmin): Error: net_connect_unix(/var/run/dovecot/stats-writer) failed: Permission denied
Enter new password:
Retype new password:
{CRYPT}$2y$12$...

[pfa@mail:~]$ /nix/store/...-dovecot-2.3.11.3/bin/doveadm pw -r 12
doveconf: Fatal: Error in configuration file /etc/dovecot/dovecot.conf line 7: ssl_cert: Can't open file /var/lib/acme/mail.example.org/fullchain.pem: Permission denied

There's our culprit, Dovecot's new version breaks because it's unable to read a private key, which it doesn't even need for its current job. Apparently it's a known issue in Dovecot, as it has been reported on the Dovecot mailinglist about a week ago: https://dovecot.org/pipermail/dovecot/2020-August/119642.html There's even a workaround. Instead of specifying the ssl certificate in the config file, you move that part into a new config file that's only readable by root and use !include_try to include that file. Easy, right? Well, NixOS requires all config files to be world-readable (for users on that system). So I modified the dovecot service to create that root-only config file before starting. And PostfixAdmin is happy again and allows users to login and change their password.

nixos-shell is a small shell script written by Jörg Thalheim for Nix and #NixOS. What it does is very simple: It takes a machine configuration from your file, builds a VM with it and runs the VM directly in your terminal with sensible defaults.

$ cat vm.nix
{}

$ nixos-shell vm.nix
/nix/store/6cxb591jsb9n8vwqk9zia1a7l3fji1x6-nixos-vm
Formatting '/home/user/nix/vms/nixos.qcow2', fmt=qcow2 size=536870912 cluster_size=65536 lazy_refcounts=off refcount_bits=16
SeaBIOS (version rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org)



iPXE (http://ipxe.org) 00:03.0 C980 PCI2.10 PnP PMM+1F3910F0+1F2F10F0 C980
Press Ctrl-B to configure iPXE (PCI 00:03.0)...
                                                                              

Booting from ROM...
Probing EDD (edd=off to disable)... k

<<< NixOS Stage 1 >>>

loading module virtio_balloon...
loading module virtio_console...
loading module virtio_rng...
loading module dm_mod...
running udev...
kbd_mode: KDSKBMODE: Inappropriate ioctl for device
starting device mapper and LVM...
mke2fs 1.44.4 (18-Aug-2018)
Creating filesystem with 131072 4k blocks and 32768 inodes
Filesystem UUID: a3b328ea-6db1-410e-bae1-88865281022d
Superblock backups stored on blocks: 
	32768, 98304

Allocating group tables: 0/4done                            
Writing inode tables: 0/4done                            
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: 0/4done

checking /dev/vda...
fsck (busybox 1.29.3)
[fsck.ext4 (1) -- /mnt-root/] fsck.ext4 -a /dev/vda
/dev/vda: clean, 11/32768 files, 6353/131072 blocks
mounting /dev/vda on /...
mounting store on /nix/.ro-store...
mounting tmpfs on /nix/.rw-store...
mounting shared on /tmp/shared...
mounting xchg on /tmp/xchg...
mounting overlay filesystem on /nix/store...
/nix/store/a9i0a06gcs8w9fj9nghsl0b6vvqpzpi4-bash-4.4-p23/bin/bash: line 3: mounts: bad array subscript
mount: mounting /dev/vda on /mnt-root/ failed: Device or resource busy

<<< NixOS Stage 2 >>>

running activation script...
setting up /etc...
Initializing machine ID from random generator.
starting systemd...

Welcome to NixOS 18.09.2030.06808d4a140 (Jellyfish)!

[  OK  ] Created slice system-getty.slice.
[  OK  ] Reached target Remote File Systems.
[  OK  ] Started Forward Password Requests to Wall Directory Watch.
[  OK  ] Listening on udev Control Socket.
[  OK  ] Created slice system-serial\x2dgetty.slice.
[  OK  ] Created slice User and Session Slice.
[  OK  ] Reached target Slices.
[  OK  ] Listening on Journal Socket.
         Mounting POSIX Message Queue File System...
         Mounting Huge Pages File System...
         Starting Remount Root and Kernel File Systems...
         Mounting Kernel Debug File System...
         Starting Create list of required st…ce nodes for the current kernel...
         Starting Load Kernel Modules...
[  OK  ] Listening on Journal Socket (/dev/log).
[  OK  ] Started Dispatch Password Requests to Console Directory Watch.
[  OK  ] Reached target Paths.
[  OK  ] Reached target Swap.
[  OK  ] Listening on udev Kernel Socket.
[  OK  ] Listening on initctl Compatibility Named Pipe.
[  OK  ] Reached target All Network Interfaces (deprecated).
         Starting udev Coldplug all Devices...
[  OK  ] Listening on Journal Audit Socket.
         Starting Journal Service...
[  OK  ] Mounted POSIX Message Queue File System.
[  OK  ] Mounted Huge Pages File System.
[  OK  ] Started Remount Root and Kernel File Systems.
[  OK  ] Mounted Kernel Debug File System.
[  OK  ] Started Create list of required sta…vice nodes for the current kernel.
[  OK  ] Started Load Kernel Modules.
         Starting Apply Kernel Variables...
         Starting Create Static Device Nodes in /dev...
         Starting Update UTMP about System Boot/Shutdown...
         Starting Load/Save Random Seed...
[  OK  ] Started Apply Kernel Variables.
[  OK  ] Started Create Static Device Nodes in /dev.
[  OK  ] Started Load/Save Random Seed.
         Starting udev Kernel Device Manager...
[  OK  ] Reached target Local File Systems (Pre).
[  OK  ] Reached target Local File Systems.
         Starting Rebuild Journal Catalog...
[  OK  ] Started Update UTMP about System Boot/Shutdown.
[  OK  ] Started Rebuild Journal Catalog.
         Starting Update is Completed...
[  OK  ] Started Update is Completed.
[  OK  ] Started udev Kernel Device Manager.
         Starting Networking Setup...
[  OK  ] Started Journal Service.
         Starting Flush Journal to Persistent Storage...
[  OK  ] Started Flush Journal to Persistent Storage.
         Starting Create Volatile Files and Directories...
[  OK  ] Started udev Coldplug all Devices.
         Starting udev Wait for Complete Device Initialization...
[  OK  ] Started Create Volatile Files and Directories.
[  OK  ] Started Networking Setup.
[  OK  ] Started udev Wait for Complete Device Initialization.
[  OK  ] Reached target System Initialization.
[  OK  ] Listening on Nix Daemon Socket.
[  OK  ] Started Daily Cleanup of Temporary Directories.
[  OK  ] Reached target Timers.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Reached target Sockets.
[  OK  ] Reached target Basic System.
         Starting Name Service Cache Daemon...
         Starting Extra networking commands....
[  OK  ] Started serial-getty@ttyS0.service.
         Starting Kernel Auditing...
         Starting DHCP Client...
[  OK  ] Started Extra networking commands..
[  OK  ] Started Kernel Auditing.
[  OK  ] Started D-Bus System Message Bus.
[  OK  ] Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch.
[  OK  ] Started Name Service Cache Daemon.
[  OK  ] Reached target Host and Network Name Lookups.
[  OK  ] Reached target User and Group Name Lookups.
         Starting Login Service...
[  OK  ] Started Login Service.
         Stopping Name Service Cache Daemon...
[  OK  ] Stopped Name Service Cache Daemon.
         Starting Name Service Cache Daemon...
[  OK  ] Started Name Service Cache Daemon.
[  OK  ] Started DHCP Client.
[  OK  ] Reached target Network.
[  OK  ] Reached target Network is Online.
         Starting Permit User Sessions...
[  OK  ] Started Permit User Sessions.
[  OK  ] Started Getty on tty1.
[  OK  ] Reached target Login Prompts.
[  OK  ] Reached target Multi-User System.



<<< Welcome to NixOS 18.09.2030.06808d4a140 (x86_64) - ttyS0 >>>

Run `nixos-help` for the NixOS manual.
Log in as "root" with an empty password.


nixos login: root

[root@nixos:~/nix/vms]# ls -l
total 26124
-rw-r--r-- 1 root root 26804224 Jan 29 11:58 nixos.qcow2
-rw-r--r-- 1 root root        3 Jan 29 11:57 vm.nix

[root@nixos:~/nix/vms]# 
QEMU: Terminated

I've got several systems to administrate at work. It's common that I have to access them through ssh or copy a file from one system to another. I love the Solaris automounter that's configured on /net by default, meaning that if you access /net/server1/nfsshare2/path/to/file you get exactly what you expect: the file. Now the automounter isn't exactly rocket science and it's easy to setup on a linux system, but I don't want to access everything through NFS and all the security issues that come with that. Luckily, #SSHFS is part of pretty much every linux distribution and the server just requires sftp, which is default on pretty much every system, as it's just a subsystem of the ssh daemon and the ssh daemon handles authentication. So all you need on the server side is an ssh daemon and you need access to login through ssh, preferably through public keys. On the client side all you need is sshfs and the matching ssh client.

$ sshfs myserver:/ /tmp/tmp.FFqyLYuk1X

$ ls -l /tmp/tmp.FFqyLYuk1X/etc/passwd
-rw-r--r-- 1 root root 3679  5. Dez 03:00 /tmp/tmp.FFqyLYuk1X/etc/passwd

This way I can easily access files on the server with local commands on my system.

Having to manually mount every sshfs manually gets tiresome pretty fast. Remember /net on Solaris? The system can easily figure out which server I want to access, so why should I have to mount everything myself?

The automounter is a pretty obvious service that could do that. There's one problem though: automountd runs as root, while the mounts have to run as my own user to be able to access my ssh-agent. I'm sure there are some possible tricks, but hardcoding a root daemon to remotely access the ssh-agent of a user just sounds wrong. Also sshfs isn't a kernel filesystem, it's a filesystem in Userspace (FUSE) and that doesn't seem to work with the kernel automounter. Luckily there's #afuse, that runs as a user and can mount FUSE filesystems.

I would have liked to have this as a systemd user service, but I couldn't figure out how to get sshfs to use my ssh-agent, meaning that all connections would fail. If you have any idea of how to do that... please contact me.

Failing a decent user service managed by systemd, I wrote a simple wrapper, that takes care of running afuse with the necessary options, so my environment.systemPackages in configuration.nix for #NixOS looks like this:

environment.systemPackages = with pkgs; [
	( writeShellScriptBin "afuse-sshfs" ''
		mkdir -p $HOME/sshfs
		exec ${afuse}/bin/afuse -o mount_template='${sshfsFuse}/bin/sshfs %r:/ %m' -o unmount_template='fusermount -u -z %m' $HOME/sshfs
	'' )
] ;

Note: I've discovered that $HOME/sshfs is probably not the best directory for this, you may want to change that e.g. to /sshfs on a single user system or whatever else you fancy.

So now I just have to run afuse-sshfs after login, which I have delegated to the XFCE startup procedure.

The result:

$ ls -l sshfs
total 0

$ ls -l sshfs/myserver/etc/passwd
-rw-r--r-- 1 root root 3679  Dec  5 03:00 sshfs/myserver/etc/passwd

$ df -h | tail -1
myserver:/    125G     11G  108G    9% /tmp/afuse-MHdCcY/myserver

The only issue I have with this solution is that it doesn't seem to automatically unmount the filesystems after some idle time, but as I regularly shut down my system after each work day, that doesn't bother me too much. One more nice feature: Instead of accessing e.g. sshfs/myserver/etc/passwd, I can access sshfs/root@myserver/etc/passwd to force sshfs to login as root. Basically the directory name accepts everything that a simple sftp would accept as well and thus aliases I've added in ~/.ssh/config work just fine.

So... time to finally add some nix content. This one is probably more along the lines of “did you know ... ?”

Just as a heads up, I'll only be using configuration.nix. I don't like the imparative way of installing packages through nix-env.

When I need to add a package on a single system, that's actually very easy, but it depends a bit on how you want to use it. Let's assume that you just want to add it to environment.systemPackages. That's the setting that will take care of adding binaries to /run/current-system/sw/bin (the main binary path on #NixOS) and also results in *.desktop files being parsed as well for desktop environments.

Here I'm adding the package stored in the file ./pkgs/mbuffer/default.nix to the environment. Note that the path is relative to the nix file you put this code in. So if you add it in /etc/nixos/configuration.nix it refers to /etc/nixos/pkgs/mbuffer/default.nix. If you add it to /etc/nixos/imports/mypkgs.nix instead, and you import that file from configuration.nix, it will refer to /etc/nixos/import/pkgs/mbuffer/default.nix.

environment.systemPackages = with pkgs; [
	vlc
	(callPackage ./pkgs/mbuffer/default.nix {})
];

I like to do it this way, because the pkgs/mbuffer/default.nix has the exact same format as all the other package files found in nixpkgs, so if you want to know the exact format of the file, have a look at this file.

This means that you can easily integrate new packages or changed packages into your own system.

Important: It does not work if the package is used in a service, unless that service gives you the option to explicitly overwrite it, for example like services.nginx. You could overwrite the nginx package by specifiying something like the following in your configuration.nix: services.nginx.package = (callPackage ./pkgs/newnginx/default.nix {})

NOTE: This text describes how I set up #Hubzilla on #NixOS. As I'm no longer using Hubzilla, I've just copied this text over for archival purposes.

PHP webapplications usually ignore all sensible conventions that exist in the unix world. Your typical php webservice needs to be put directly into the document root or a subdirectory of the webserver. That's not a big deal if it is completely static. But usually it is not, it typically needs a somewhat static config.php containing a database password, it may need a temporary directory, a log directory and maybe another directory for long-term storage. This is bad, because you can't just change a link to point to a different version. This is bad in terms of security because the program must reside in a writeable directory. This is bad because config.php with your precious database password is in a directory that's readable by the webserver. Typically you have another problem: all php webservices share a common user that they run as.

NixOS and its package manager Nix completely clashes with that. All derivations are put into /nix/store world-readable and immutable. There's no place for a config.php with a database password in /nix/store. Even root cannot write to /nix/store. The distinction between application and data is enforced by NixOS. You could put the webapplication into /srv/www or a similar directory, but you would lose all of the features that make Nix so good. Instead there's no other sensible option than to split the webapplication into the program and data part. The trick is to set symlinks during the build. I'm going to use hubzilla as an obvious example here.

Building Hubzilla with Nix

Nix first needs some generic information about how and where to download Hubzilla:

{ stdenv, lib, fetchgit, php, dataDir ? null }:
stdenv.mkDerivation rec {
  name = "hubzilla-${version}";
  version = "3.6.1";
  rev = "${version}";

  src = fetchgit {
    inherit rev;
    url = "https://framagit.org/hubzilla/core.git";
    sha256 = "1zaczw4mxxbv7p6xmmf8wpy54jmnf980yd21c4kfncmh3ri0mrf6";
  };

  nativeBuildInputs = [ php ];

  phases = [ "unpackPhase" "installPhase" ];
  installPhase = ''
    cp -Rp ./ $out/
    cd "$out"
    echo Building documentation...
    TEMP=$(mktemp -d)
    ln -s $TEMP/store $out/store
    mkdir -p "$TEMP/store/[data]/smarty3"
    php util/importdoc
    rm -rf "$out/store" "$TEMP/store"
    ${lib.optionalString (dataDir != null) ''
      ln -s ${dataDir}/htconfig.php $out/.htconfig.php
      ln -s ${dataDir}/addon $out/addon
      ln -s ${dataDir}/extend $out/extend
      ln -s ${dataDir}/store $out/store
      mv $out/view/theme $out/view/theme.dist
      ln -s ${dataDir}/view/theme $out/view/theme
      ln -s ${dataDir}/widget $out/widget
    ''}
  '';
}

If you store the above code in default.nix and build it with nix-build -E 'with import <nixpkgs> { }; callPackage ./default.nix { }', you already get the hubzilla source in a /nix/store and even updated documentation in there. Not special so far. Try to build it with nix-build -E 'with import <nixpkgs> { }; callPackage ./default.nix { dataDir = "/var/lib/hubzilla" }'. Now you get a special version that expects its writeable directories and config file in /var/lib/hubzilla. Everytime you change that directory you obviously get a new derivation in /nix/store. The nice thing about this is that a version upgrade is just a change of the version number in the file and thus rollbacks should work – as long as the database is not upgraded. I also like that I can give the dataDir permissions that forbid the webserver any access. Only the php processes can access dataDir.

I haven't noticed any downsides yet, but I haven't delved into themes or addons yet, so there may be some issues later.

#NixOS is a linux distribution that has a very different approach compared to other distributions.

You do not change configuration files of applications. You just change the build instructions that the package manager Nix uses to build the system. On NixOS the full system is rebuilt everytime you want to change even a minor detail. A rebuild means that the Nix package manager reads in NixPKGs, then reads in your local build instructions (e.g. “services.openssh.enable = true”) to complement them and then builds all derivations that are required for this specific configuration. A derivation is roughly similar to a package in other distributions, but in NixOS even /etc/ssh/sshd_config has its own derivation, meaning that it gets built automatically. Every derivation that gets built, ends up in a directory in /nix/store, indexed with a checksum over all its version information, build instruction and all its dependencies. That means when you change the build instructions (even if you just insert an irrelevant space in a field somewhere), change a dependency or update to a newer version, Nix will put it into a different directory in /nix/store. That in turn means that Nix can easily determine if it has already built a specific configuration or program with its specific dependencies. If the path in /nix/store exist, it has already been built. Oh, and /nix/store is immutable. You are not supposed to change any files in there and there are many good reasons to just accept that and not even try it. If you want to make a change, edit the build instructions or the system configuration and let Nix rebuild the system. By following that, you gain the ability to just rollback to older versions (called generations) of your system configuration. Made a change and noticed that it doesn't work well? sudo nixos-rebuild --rollback And you're good. The system doesn't boot anymore because of a software problem? Hit space in the boot manager and boot an older generation. Your harddisk failed and you have to reinstall your system? With NixOS you still have to partition your drive manually from the installation CD/DVD, that's a bit annoying. But then you just copy /etc/nixos from your backup, tell Nix to install the system (completely non-interactive). Finally restore your /home from the latest backup and your system will be in exactly the same state as before the disk failure. Well ok: it will be in the same state as your last backup. But there is no question about the packages you had installed or their versions. Also the system configuration is completely included in /etc/nixos, so just backing up /etc/nixos and /home is sufficient for a desktop or laptop. For servers you obviously may need other directories as well, but that depends on the applications running on them.

On my systems /etc/nixos is a git repository. That's enough to allow me to share this system configuration on multiple systems. I'm using NixOS since 2015 now and I really don't want to go back. Ansible and Puppet are just workarounds for the package and /etc hell.