Remote Builds - Lix Reference Manual

## Keyboard shortcuts

Press `←` or `→` to navigate between chapters

Press `S` or `/` to search in the book

Press `?` to show this help

Press `Esc` to hide this help

# Remote Builds

Lix supports remote builds, where a local Lix installation can forward Nix builds to other machines. This allows multiple builds to be performed in parallel and allows Lix to perform multi-platform builds in a semi-transparent way. For instance, if you perform a build for a `x86_64-darwin` on an `i686-linux` machine, Lix can automatically forward the build to a `x86_64-darwin` machine, if available.

To forward a build to a remote machine, it’s required that the remote machine is accessible via SSH and that it has Nix installed. You can test whether connecting to the remote Nix instance works, e.g.

    $ nix store ping --store ssh://mac

will try to connect to the machine named `mac`. It is possible to specify an SSH identity file as part of the remote store URI, e.g.

    $ nix store ping --store ssh://mac?ssh-key=/home/alice/my-key

Since builds should be non-interactive, the key should not have a passphrase. Alternatively, you can load identities ahead of time into `ssh-agent` or `gpg-agent`.

If you get the error

    bash: nix-store: command not found
    error: cannot connect to 'mac'

then you need to ensure that the `PATH` of non-interactive login shells contains Nix.

> Warning

> If you are building via the Lix daemon (default on Linux and macOS), it is the Lix daemon user account (that is, `root`) that should have SSH access to a user (not necessarily `root`) on the remote machine.

> Furthermore, `root` needs to have the public host keys for the remote system in its `.ssh/known_hosts`. To add them to `known_hosts` for root, do `ssh-keyscan HOST | sudo tee -a ~root/.ssh/known_hosts`.

> If you can’t or don’t want to configure `root` to be able to access the remote machine, you can use a private Nix store instead by passing e.g. `--store ~/my-nix` when running a Nix command from the local machine.

## Configuration

The list of remote machines can be specified on the command line or in the Lix configuration file. The former is convenient for testing. Additionally, there are two supported formats to configure remote builders: The legacy, "space"-separated format and starting with Lix 2.95.0, a TOML.

Remote builders can also be configured in `nix.conf`, e.g.

    builders = ssh://mac x86_64-darwin ; ssh://beastie x86_64-freebsd

Finally, remote builders can be configured in a separate configuration file included in `builders` via the syntax `@file`. For example,

    builders = @/etc/nix/machines

causes the list of machines in `/etc/nix/machines` to be included. (This is the default.)

If you want the builders to use caches, you likely want to set the option `builders-use-substitutes` in your local `nix.conf`.

To build only on remote builders and disable building on the local machine, you can use the option `--max-jobs 0`.

Each machine specification consists of the following attributes. How those are combined within the configuration file differs for the formats, and will be explained further down.

1. `uri` (required) The URI of the remote store in the format `ssh[-ng]://[username@]hostname[?port=<port>]`, e.g. `ssh://nix@mac` or `ssh://mac`. If the ssh server is not listening on port 22 (e.g. port 1337 in this case) the URI would be `ssh[-ng]://nix@mac?port=1337`. The hostname may be an alias defined in your `~/.ssh/config`.

2. `system-types` (optional) A list of Nix platform type identifiers, such as `x86_64-darwin`. It is possible for a machine to support multiple platform types, e.g., `i686-linux` and `x86_64-linux`.

   Defaults to the local platform type

3. `ssh-key` (optional) The SSH identity file to be used to log in to the remote machine.

   Defaults to SSHs regular identities.

4. `jobs` (optional) The maximum number of builds that Lix will execute in parallel on the machine. Typically, this should be equal to the number of CPU cores divided by the cores within the target machines configuration, i.e. `jobs * cores ~= cpu cores`

   Defaults to 1; must be a positive integer.

5. `speed-factor` The “speed factor”, indicating the relative speed of the machine. If there are multiple machines of the right type, Lix will prefer the fastest, taking load into account.

   Defaults to 1; must be a positive float.

6. `supported-features` (optional) A list of supported features. If a derivation has the `requiredSystemFeatures` attribute, then Lix will only schedule the derivation on a machine that has the specified features. For example, the attribute

       requiredSystemFeatures = [ "kvm" ];

   will cause the build to be performed on a machine that has the `kvm` feature.

   Defaults to an empty list.

7. `mandatory-features` (optional) A list of mandatory features. A machine will only be used to build a derivation if all the machine’s mandatory features appear in the derivation’s `requiredSystemFeatures` attribute.

   Defaults to an empty list.

8. `ssh-public-host-key` (optional) The public host key of the remote machine.

   Defaults to basic ssh behavior (checking contents of the known-hosts file)

### Using a TOML configuration

Each machine is configured as an attribute within the map called `machines`. The attributes name is the machines name. Attributes can be in any order.

For example:

    version = 1

    [machines.andesite]
    uri = "ssh://lix@andesite.lix.systems" # toml also allows for comments
    system-types = ["i686-linux"]
    jobs = 8
    speed-factor = 1.0
    supported-features = ["kvm"]
    ssh-key = "/home/deepslate/.ssh/id_ed25519"

    [machines.diorite]
    uri = "ssh://lix@diorite.lix.systems"
    system-types = ["i686-linux"]
    jobs = 8
    speed-factor = 2.0
    ssh-key = "/home/deepslate/.ssh/id_ed25519"

    [machines.granite]
    uri = "ssh://lix@granite.lix.systems"
    system-types = ["i686-linux"]
    jobs = 1
    speed-factor = 2.0
    supported-features = ["kvm", "benchmark"]
    ssh-key = "/home/deepslate/.ssh/id_ed25519"

    [machines.legacy]
    uri = "ssh://nix@nix-15-11.nixos.org"
    enable = false

> Note

> If the version tag is omitted (e.g. in the CLI), it defaults to the latest version. It is strongly recommended to always provide a version tag for configuration within files to avoid breakage.

For testing purposes, one can also define a builder ad hoc on the CLI as follows: `--builders 'machines.andesite = {uri = "ssh://lix@andesite.lix.systems", jobs = 8}'`

#### Special handling of fields

- `enable` (optional) If set to false, the declared machine will not be loaded. This allows one to statically disable machines.

  Defaults to true

### Using the legacy format

> Warning

> This format is frozen and new features / configuration options will not be backported to this format.

It is possible to specify multiple builders separated by a semicolon or a newline, e.g.

      --builders 'ssh://mac x86_64-darwin ; ssh://beastie x86_64-freebsd'

Every machine specification consists of the elements listed in the section above, seperated by any amount of spaces or tabs. The Attributes need to be provided in order and without names. To leave a field at its default, set it to `-`. Lists are colon seperated, without additional spaces.

    lix@andesite.lix.systems    i686-linux      /home/deepslate/.ssh/id_ed25519        8 1 kvm
    lix@diorite.lix.systems     i686-linux      /home/deepslate/.ssh/id_ed25519        8 2
    lix@granite.lix.systems     i686-linux      /home/deepslate/.ssh/id_ed25519        1 2 kvm benchmark

#### Special handling of fields

- `uri`: Due to backward compatibility, the `ssh://` may be omitted for the store-uri.
- `ssh-public-host-key`: The key must be provided encoded in base64. Specifically calculated via `base64 -w0 /etc/ssh/ssh_host_ed25519_key.pub`

### Format detection

At first, the given configuration is being parsed syntactically as a toml. If parsing fails and the given configuration contains a `"` the error is presented to the user, as those characters are necessary for TOML, but disallowed for the legacy format. Otherwise, parsing is retried using the legacy format. If non-syntactic errors are detected within the toml, the exception will always be shown to the user directly.

## Builder selection

The configuration(s) above specify several machines that can perform `i686-linux` builds. However, `granite` will only do builds that have the attribute

    requiredSystemFeatures = [ "benchmark" ];

or

    requiredSystemFeatures = [ "benchmark" "kvm" ];

`diorite` cannot do builds that require `kvm`, but `andesite` does support such builds. For regular builds, `diorite` will be preferred over `andesite` because it has a higher speed factor.