NixOS. Tricks & Troubleshooting

It always seems impossible until its done, Theodor Seuss Geisel

Tricks

1. Use packages without installing them globally, e.g., nix-shell -p unzip. nix-shell opens a new shell with an environment configured by Nix. It allows you to use packages without installing them globally. The -p flag specifies a package to be included in the environment.

   # In this case, it is unzip, a utility for extracting files from ZIP archives.
   nix-shell -p unzip
   
   unzip /home/nmaximo7/Downloads/Send\ Anywhere.zip -d /home/nmaximo7/Downloads/
   

Killing processes

Sometimes, applications freeze or become unresponsive. NixOS, like other Linux distributions, offers several tools to handle such situations:

1. Install xkill and killall.

   environment.systemPackages = with pkgs; [
     psmisc  # This package includes killall
     xorg.xkill  # This package includes xkill
   ];
   

   After adding these lines in /etc/nixos/configuration.nix, run sudo nixos-rebuild switch to apply changes.

2. Type “xkill” in a terminal and press enter. This will transform your mouse cursor into an “x” or into a scull. Now you could simply left-click whatever window is not responding and it’ll be gone. killall command allows you to kill a process by name, e.g., sudo killall sshd, killall htop, etc.

3. Use the i3, Hyprland, etc. force kill shortcut, e.g., i3, Mod + Shift + Q where Mod is usually the Windows key or Alt, bindsym $mod+Shift+q kill or Hyprland, Mod + Q, bind = ${modifier},Q,killactive.

4. Find the PID (PID stands for process identification name) with ps aux | grep [application_name] and kill the process from the terminal: kill -9 [PID].

5. If your system is completely frozen, you may need to switch to a virtual terminal or console, e.g., Ctrl+Alt+F2 and kill the stubborn process from there or even reboot.

Repairing a Broken NixOS Boot

NixOS boot failures can be frustrating, but they are not cause for despair or panic. Remain Calm: It’s natural to feel frustrated when your system fails to boot, but panicking won’t help. Take a deep breath and approach the problem systematically.

First, you can attempt to repair it using a NixOS Live USB and tools like chroot. The steps below will guide you through identifying your root partition, mounting your file systems, chrooting into your installed environment, and fixing the bootloader. The goal is to restore your system to a working state without having to reinstall everything from scratch and wasting your time miserably.

Step 1. Create a NixOS Live USB. Download a NixOS ISO image from the official website and create a bootable USB using tools like dd, balenaEtcher or Ventoy: sudo dd if=nixos-graphical-*.iso of=/dev/sdX bs=4M conv=fsync.

Step 2. Boot from the NixOS Live USB. Insert the USB into your machine and reboot. Access your firmware or BIOS boot menu (often by pressing a key like F12, F2, or Del/Supr) and select your USB. You will then enter the NixOS live environment.

Step 3. Identify Your Root Partition. In the NixOS live environment, open a terminal, and run the command: sudo fdisk -l. This command lists all disk partitions. You’ll need to identify the correct root partition (e.g., /dev/nvme0n1p1, /dev/sda2) where your NixOS root filesystem resides.

Step 4. Create a mount point: sudo mkdir /mnt. This command creates a directory at /mnt where you will mount your root partition.

Step 5. Mount the root partition: sudo mount /dev/nvme0n1p1 /mnt. Replace /dev/nvme0n1p1 with your actual root partition.Mounting the root partition allows you to access your system’s files and configurations.

Step 6. Mount essential virtual filesystems:

sudo mount --bind /dev /mnt/dev
sudo mount --bind /dev/pts /mnt/dev/pts
sudo mount --bind /proc /mnt/proc
sudo mount --bind /run /mnt/run
sudo mount --bind /sys /mnt/sys

These commands mount essential virtual filesystems that are necessary for the chroot environment to function correctly.

Step 7. Chroot into Your System:

sudo chroot /mnt /nix/store/*-bash-*/bin/bash --login

The `chroot` command changes the root directory to `/mnt`, effectively giving you a shell and allowing you to run commands as if you were booted into your installed system. Specifying the path to bash from the Nix store ensures you use the correct shell in the NixOS environment.

Step 8. Fix the Bootloader: nixos-rebuild boot. This command rebuilds the bootloader configuration.

Step 9. Exit the chroot environment: exit

Step 10: Unmount all filesystems you have previously mounted:

sudo umount /mnt/dev/pts
sudo umount /mnt/dev
sudo umount /mnt/proc
sudo umount /mnt/run
sudo umount /mnt/sys
sudo umount /mnt/boot
sudo umount /mnt

Step 11: Reboot the system: sudo reboot and remove the USB.

Use NixOS’ System Rollback Feature

One of NixOS’s strengths is the ability to roll back to previous generations of configurations. NixOS keeps generations of system configurations (historical versions of your system configuration) allowing you to roll back to a previous working configuration.

Step 1: Access GRUB Menu. During booting, press and hold the Shift or Esc key to access the GRUB menu, allowing you to choose between different boot options and see a list of “NisOS - Bootable generations.”

Step 2: Select a Previous Generation. Navigate to NixOS - Bootable generations and choose an earlier generation. This is useful if the current configuration is causing boot failures, sudden crashes or any other types of problems.

Step 3: Boot into the Selected Generation. Using the arrow keys, select an older generation that you know was working and press Enter to boot into it.

Step 4: Make the Previous Generation the Default. Once you have booted into a working generation:

1. List generations: sudo nix-env --list-generations --profile /nix/var/nix/profiles/system. This lists all available generations of your system configuration, allowing you to see which ones are available for rollback.
2. Switch to the desired generation: sudo nix-env --switch-generation X --profile /nix/var/nix/profiles/system. Replace X with the generation number you want to switch to. This effectively sets the chosen generation as your current configuration.
3. Rebuild the system to update the bootloader: sudo nixos-rebuild boot. After switching generations, this command updates the bootloader to reflect the new default configuration, ensuring that the system boots with the selected generation next time.

Troubleshooting nixos-rebuild hangs

When the nixos-rebuild process hangs, it can be frustrating. Here are some steps to help you diagnose and resolve the issue:

• Previous step. Ensure you have sufficient disk space, as a full disk can cause builds to hang (df -h) and test that your network connection is stable (ping -c 4 cache.nixos.org). If you recently added or modified services in your configuration, disable them and then rebuild again because they might be causing conflicts.

• Interrupt the Current Process. Press Ctrl + C to attempt to gracefully stop the rebuild process or force Termination. If the process does not respond to Ctrl + C, you can forcefully terminate it: sudo killall -9 nixos-rebuild or sudo pkill -9 nixos-rebuild.

• Check for Stuck Nix Processes.

ps aux | grep nix
# Identify any stuck and suspicious processes related to Nix.

sudo kill -9 < PID > # Forcefully terminate (kill) any suspicious processes. Replace PID with the actual process ID related to Nix.

• Remove build locks sudo rm -rf /nix/var/nix/building/ and caches sudo rm -rf /nix/var/cache/nix/. Nix uses lock files and caches to manage builds. Corrupted or leftover locks can cause rebuilds to hang.
• Collect garbage. Run garbage collection to free up space, remove unused packages, and potentially resolving issues with stuck builds: sudo nix-collect-garbage -d. The -d flag deletes all unused Nix stores path.
• Optimize store. Optimize the Nix store to improve performance: sudo nix-store ‐‐optimize. This command deduplicates identical files in the Nix store, saving disk space and potentially resolving issues related to corrupted files.
• Update Channels. Ensure all channels are up-to-date (updates the list of available Nix packages): sudo nix-channel ‐‐update.
• Clean Rebuild. Perform a clean rebuild with additional options for debugging: sudo nixos-rebuild switch ‐‐show-trace ‐‐keep-going ‐‐max-jobs 4

  ‐‐show-trace: Provides detailed error messages. ‐‐keep-going: Continues the build even if some derivations fail, allowing you to see all errors at once. ‐‐max-jobs 4: Limits the number of parallel jobs to reduce resource consumption. Another option is ‐‐fallback. If a binary cache is unavailable for a package, this option allows Nix to build it from source instead of failing.

• Check Logs. Review system logs for any errors or warnings that might indicate the cause of the hang: journalctl -xe
• Inspect Configuration Files. Verify that your configuration(s) files(s) (/etc/nixos/configuration.nix) do not contain syntax errors or invalid configurations. Besides, nixos-rebuild dry-build checks the syntax and validity of the configuration without building it;

You can also use nix-linter /etc/nixos/configuration.nix to detect common issues (you may need to install nix-linter).

• Reset the Nix Database (Last Resort): sudo rm -rf /nix/var/nix/db (it deletes the Nix database, which keeps track of installed packages), sudo nix-store ‐‐repair (it attempts to rebuild the database based on the current contents of the Nix store.)

  This step will reset the Nix database. This is a risky operation and can lead to data loss. Proceed with caution.

• Reboot. Sometimes, a simple reboot can resolve lingering issues: sudo reboot

By following these steps, you should be able to identify and resolve most issues causing the nixos-rebuild process to hang. If problems persist, consider restoring from a previous backup, reinstalling NixOS again (please backup important and sensitive data), seeking assistance from the NixOS community forums (when seeking help, include any relevant information such as logs or error messages, and steps you’ve already taken) or support channels.