Troubleshooting

A rebuild can fail in three places, and knowing which one you are in saves most of the work. Evaluation fails when Nix reads your configuration and the expressions do not make sense, before anything is compiled. A build fails when the configuration evaluated fine but a derivation did not compile or its check phase failed. Activation succeeds and the system switches, but a service you defined refuses to start once it is running. Each of these has its own error text and its own set of tools, and this page walks through reading the failure, reproducing it, and recovering from it.

Everything here assumes the Ternix layout, a flake exposing nixosConfigurations.nixos-host for x86_64-linux, living in a git repository you control (Using a config).

Read the failure before you change anything

Nix error output is long and the useful part is usually at the very end. Scroll to the last error: line first. The lines above it are the trace that led there, and they matter once you know what failed, but the final line is the failure itself. Resist the urge to read top to bottom.

When an evaluation error is hard to follow, ask for the full trace. Without it Nix collapses the chain of expressions that led to the error into a short summary, and --show-trace prints every step so you can see which option or module triggered it.

sudo nixos-rebuild switch --flake .#nixos-host --show-trace

A build failure is different. Evaluation succeeded, so the trace is not the issue. What you want is the output of the builder that failed, which Nix captured in the store as a log. Copy the .drv path or the output path from the failure line and read its log.

nix log /nix/store/abcd1234...-some-package-1.2.0.drv

nix log replays everything the builder printed, including the compiler error or failing test that actually stopped the build. If the derivation is still known to Nix you can also pass the output path or a flake reference such as nixpkgs#hello instead of the .drv.

By default Nix deletes the temporary build directory whether the build passed or failed, which throws away the half-built tree you would want to inspect. --keep-failed leaves it on disk and prints its path so you can go look at the config files, logs, and intermediate output the build left behind.

nixos-rebuild build --flake .#nixos-host --keep-failed

The kept directory lives under /tmp (or wherever TMPDIR points) and is named after the derivation. Delete it yourself when you are done, because nothing else will.

Common evaluation errors and what they mean

Evaluation errors read like a foreign language at first, but there are only a handful you will hit often, and each one points at a specific kind of mistake.

infinite recursion encountered. Nix tried to compute a value that depends on itself. The classic cause is referring to config in a way that feeds back into the option you are defining, or shadowing a name so a definition refers to its own output. A frequent trigger is using config inside a let that the module's own config block then depends on.

# infinite recursion: enabling the service is gated on the
# service's own resolved config, which is not known yet
{ config, lib, ... }:
{
  services.nginx.enable = config.services.nginx.enable or false;
}

The fix is to break the cycle by reading from a separate option or a plain value rather than the attribute you are currently defining. When the trace is too tangled to see the loop, comment out modules until the error disappears, then reintroduce the one that brings it back.

attribute 'X' missing. You asked for an attribute that does not exist on the set you indexed. This is often a typo in an option path, a package name that moved or was renamed in nixpkgs, or reaching into a pkgs set for something under a different name.

# attribute 'ripgrepp' missing
{ pkgs, ... }:
{
  environment.systemPackages = [ pkgs.ripgrepp ];
}

Check the real name with nix search nixpkgs ripgrep, or open a repl and tab-complete pkgs.rip to see what exists (Commands and scripts).

value is a Y while a Z was expected. A type mismatch. You gave a string where a list was wanted, a bool where a string was wanted, or similar. The message names both the type it found and the type it needed, so it tells you exactly what to change.

# value is a string while a list was expected
{
  environment.systemPackages = "vim";
}

environment.systemPackages takes a list, so the value must be [ pkgs.vim ]. When the path in the error is an option you did not set directly, the bad value is coming from a module, and --show-trace shows which one.

The git-tracked-files gotcha. Flakes only see files that git knows about. When you add a new module file and reference it but have not staged it, Nix evaluates the flake from the git tree and the file is invisible, so you get a confusing path does not exist or attribute missing error for code that is sitting right there on disk. Stage the file and the error vanishes.

git add hosts/nixos-host/new-module.nix

You do not need to commit, only to make the file known to git. This one catches nearly everyone at least once, so suspect it whenever a brand-new file behaves as if it is not there.

Interactive inspection with nix repl

When you want to know what an option actually resolved to, the repl beats guessing. Start it, load your flake, and the whole configuration tree is available for tab-completion and evaluation.

nix repl

Inside the repl, :lf . loads the flake in the current directory and binds its outputs as variables. From there you can read any path under a configuration to see its final value after every module has contributed.

:lf .
nixosConfigurations.nixos-host.config.networking.hostName
nixosConfigurations.nixos-host.config.services.openssh.enable
nixosConfigurations.nixos-host.config.systemd.services.nginx.serviceConfig.ExecStart

This is the fastest way to answer the question that underlies most config bugs, which is whether the option holds the value you think it does. If ExecStart is empty or points at the wrong binary, you have found your problem without booting anything. Tab-completion works at every level, so nixosConfigurations.nixos-host.config.services. followed by Tab lists every service namespace your system knows about.

To build something from inside the repl, use :b. It realises a derivation and prints its output path, which is handy for checking that a package or the system closure builds at all.

:b nixosConfigurations.nixos-host.config.system.build.toplevel

Dependency and store inspection

Sometimes the question is not whether something builds but why it is even there, or how much it is costing you. The store tooling answers both.

nix why-depends explains the chain of references that pulls one store path into another. When you find an old or unexpected package in your system closure, this tells you which option dragged it in.

nix why-depends .#nixosConfigurations.nixos-host.config.system.build.toplevel nixpkgs#openssl

nix path-info -Sh reports the closure size of a path. -S adds the total size of everything the path transitively references, and -h prints it in human units, so you can see what a result actually weighs on disk.

nix path-info -Sh ./result

For a visual breakdown, nix-tree opens an interactive view of a closure where you can walk the dependency graph and see which paths dominate the size. It is a separate tool you can run without installing it.

nix run nixpkgs#nix-tree -- ./result

Service debugging at runtime

A configuration that builds and activates can still leave a service dead. NixOS turns most services into systemd units, so systemd is where you look. Start with the unit's status, which shows whether it is running, its last exit code, and the final lines of its log.

systemctl status nginx.service

For the full log of one unit since the last boot, use journalctl scoped to the unit. -u selects the unit and -b limits it to the current boot, which keeps you from scrolling through history that no longer matters.

journalctl -u nginx.service -b

When you do not yet know which unit failed, journalctl -xe jumps to the end of the whole journal and adds explanatory text for the entries it can. It is the quickest way to catch a service that died loudly during the last boot.

journalctl -xe

To connect a running unit back to your configuration, read the unit definition straight from the evaluated config. Every systemd.services.<name> you set, and every one a module set for you, is visible there, so you can confirm the ExecStart, environment, and dependencies the running system is actually using.

:lf .
nixosConfigurations.nixos-host.config.systemd.services.nginx.serviceConfig

The config builds but the service will not start

When activation succeeds but a unit will not come up, work through the usual causes in order. Each one is quick to check and rules out a whole class of failure.

Firewall. The service is listening but the port is closed. NixOS blocks ports by default, so a service is unreachable until you open its port. Check networking.firewall.allowedTCPPorts in your config, or read it live with nixosConfigurations.nixos-host.config.networking.firewall.allowedTCPPorts in the repl.
Wrong path. ExecStart, a config file path, a working directory, or a data directory points somewhere that does not exist. The status output and the journal name the path that failed to open.
Missing dependency. The unit starts before something it needs is ready, or a binary it calls is not on its PATH. Units assembled by hand often need after and wants to order them, and a package the script shells out to must be in the service's path.
Permissions. The service runs as a user that cannot read its config or write its state directory. Check the User and Group in serviceConfig, and use StateDirectory so systemd creates and owns the directory for you.
ExecStart. The command itself is wrong, the binary is missing, or its flags changed. Read the exact ExecStart from the evaluated config, then run that same command by hand to see the error directly.

# reproduce the failure outside systemd by running ExecStart yourself
journalctl -u myservice.service -b --no-pager
sudo -u myservice /nix/store/...-myservice/bin/myservice --config /etc/myservice.toml

Running the command by hand as the service user is the single most useful step here, because it gives you the program's own error message instead of systemd's summary of it.

Recovery of last resort

When a rebuild leaves the machine unbootable or broken in a way you cannot fix from inside it, NixOS keeps the previous system one menu selection away. This is the safety net that makes risky changes survivable.

Reboot and pick an earlier generation from the bootloader menu. Every switch and boot writes its own boot entry, so the last working system is still there to select. Booting it changes nothing permanently, it just starts the older system so you can work.

Once you are booted into a generation that works, make the rollback permanent so the next normal boot uses it too.

sudo nixos-rebuild switch --rollback

This sets the previous generation as the current system and the boot default. From there you can fix the configuration that broke, rebuild, and switch forward again when it is right.

Advanced

Reproduce a problem in a throwaway VM

nixos-rebuild build-vm builds your whole configuration into a runnable QEMU image without touching the host. It is the safest way to chase a boot failure or a service problem, because everything happens inside a disposable virtual machine on its own throwaway disk.

nixos-rebuild build-vm --flake .#nixos-host
./result/bin/run-nixos-host-vm

The VM boots your real configuration, so a service that fails on the metal usually fails the same way in the VM, where you can break it, read its logs, and try fixes without risk. When you are done you delete the disk image and nothing remains.

Check the whole flake at once

nix flake check evaluates every output the flake exposes and runs its checks, which surfaces evaluation errors in configurations you are not currently building. It is a good gate before committing, because it catches a broken host or module that a single nixos-rebuild would not have touched.

nix flake check

Add --show-trace here too when a check fails with a terse message, and the same reading habit applies, which is to start at the last error: line and work back.

Using a config covers applying a configuration and the rebuild modes the errors here come from.
Updating safely shows the build, test, switch loop that keeps a bad change off the boot default.
Commands and scripts collects the rebuild, flake, repl, and store commands used on this page.
Writing packages goes deeper on derivations, which is where nix log and --keep-failed earn their place.