Using the vpsAdminOS test framework in other projects

The vpsAdminOS test framework can be reused without copying its sources:

  • Nix code defines QEMU machines and turns test definitions under tests/suite/ into flake outputs.
  • The upstream Ruby runner (vpsadminos#test-runner) evaluates your project's flake outputs, boots the machines and runs Ruby testScripts against them.

The important bit: the runner always evaluates/builds tests from the current working directory.

By default it uses these flake outputs:

  • .#testsMeta.<system> (discovery: ls, tags/labels, templates, ...)
  • .#tests.<system>."<test-path>" (build JSON config for a test)

If you use test-runner --test-config, the runner re-evaluates the suite and also needs an optional flake API:

  • .#lib.testFramework.mkTests
  • .#lib.testFramework.mkTestsMeta

So to integrate the runner into another repository, your project needs:

  1. A Nix flake that exports tests and testsMeta.
  2. A tests/ tree with all-tests.nix + suite/ (and usually make-test.nix).
  3. Optional: lib.testFramework if you want test-runner --test-config.

For writing tests (machine definition, testScript, tags, multiple scripts), see Testing.

Source snapshots and OS version metadata

Each runner invocation resolves the tested repository to one immutable Nix source and keeps it GC-rooted until the command finishes. Path snapshots retain the files selected by Nix, including untracked and ignored files. The runner captures the original repository's commit once and compares that commit with the snapshot through a temporary Git index. Included extra files and changed tracked content mark the snapshot dirty. The caller's index is unchanged; later checkout or HEAD changes do not change the captured identity.

The runner scopes TEST_RUNNER_SOURCE_INFO to that invocation. Its JSON object contains sourcePath, nullable revision, shortRev and dirty. Git-free roots have unknown identity; a Git repository enclosing a different source root is not used as its provenance. The previous environment value is restored when the command returns or raises.

lib.testFramework.sourceVersionModule supplies the same source revision, dirty flag and version suffix defaults as normal flake OS construction. The framework uses captured metadata only when sourcePath equals the vpsAdminOS provider's source path. For example, a vpsAdmin test suite using a pinned vpsAdminOS input retains that input's identity, rather than the vpsAdmin caller's revision. Explicit module settings still override these defaults. Older framework providers without this optional field remain supported; existing packaged revision files remain the fallback when source identity is unknown.

Integration example (vpsAdmin)

The vpsAdmin repository reuses the framework directly via a flake input. The sections below describe the same setup in a copy/paste-friendly way.

1) Add vpsAdminOS as a flake input

In your flake.nix, add vpsAdminOS and (recommended) align nixpkgs with it:

{
  inputs = {
    vpsadminos.url = "github:vpsfreecz/vpsadminos/<ref>";
    nixpkgs.follows = "vpsadminos/nixpkgs";
  };

  outputs = { self, nixpkgs, vpsadminos, ... }:
  let
    systems = [ "x86_64-linux" ];
    forAllSystems = nixpkgs.lib.genAttrs systems;
  in
  {
    # filled in below
  };
}

Tip for local development: point the input at a local checkout: vpsadminos.url = "path:../vpsadminos";.

2) Add the tests/ tree

Create this minimal layout in your project:

tests/
  all-tests.nix
  make-test.nix
  suite/
    hello/
      boot.nix

tests/make-test.nix should delegate to the upstream helper (this is what each suite file imports):

testFn:
{ testFramework, ... }@args:
let
  upstream = testFramework.makeTest testFn;

  # Optional: pass extra args to NixOS/vpsAdminOS modules as `specialArgs`.
  # The vpsAdmin repo uses this to make the vpsAdminOS checkout available as
  # `vpsadminos` inside NixOS module evaluation.
  mergedExtraArgs = {
    vpsadminos = testFramework.sourcePath;
  } // (args.extraArgs or { });
in
upstream (args // { extraArgs = mergedExtraArgs; })

tests/all-tests.nix lists tests (and templates) and is evaluated by the flake helpers:

{
  pkgs,
  system,
  testFramework,
  suiteArgs ? { },
}:
let
  nixpkgs = import pkgs { inherit system; };
  lib = nixpkgs.lib;

  testLib = testFramework.makeTestLib {
    inherit pkgs system lib suiteArgs;
    suitePath = ./suite;
  };
in
testLib.makeTests [
  "hello/boot"
]

And a minimal suite test, tests/suite/hello/boot.nix:

import ../../make-test.nix ({ ... }: {
  name = "hello-boot";
  description = "Boot a VM and run a simple command";

  machine = {
    spin = "nixos";
    config = {
      networking.hostName = "hello";
      virtualisation.memorySize = 1024;
      virtualisation.cores = 2;
    };
  };

  testScript = ''
    machine.start
    machine.wait_for_boot
    machine.wait_for_service("test-shell")
    machine.succeeds("echo hello")
  '';
})

Note: test paths in all-tests.nix map to files under tests/suite/ without the .nix suffix, e.g. hello/boot -> tests/suite/hello/boot.nix.

3) Export flake outputs

Your flake must export tests and testsMeta so the runner can list and build tests via nix eval/nix build:

{
  outputs = { self, nixpkgs, vpsadminos, ... }:
  let
    systems = [ "x86_64-linux" ];
    forAllSystems = nixpkgs.lib.genAttrs systems;
  in
  {
    tests = forAllSystems (system:
      vpsadminos.lib.testFramework.mkTests {
        inherit system;
        testsRoot = ./tests;
      });

    testsMeta = forAllSystems (system:
      vpsadminos.lib.testFramework.mkTestsMeta {
        inherit system;
        testsRoot = ./tests;
      });
  };
}

If your project uses a different nixpkgs than vpsAdminOS, also pass pkgsPath = nixpkgs.outPath; to both mkTests and mkTestsMeta.

This is enough for normal test-runner ls/test/debug usage.

3a) Support --test-config (optional)

If you want to use test-runner --test-config, also export lib.testFramework. The runner uses it to re-evaluate the suite with the provided Nix configuration.

{
  outputs = { self, nixpkgs, vpsadminos, ... }:
  let
    withTestFrameworkDefaults = args:
      {
        pkgsPath = args.pkgsPath or nixpkgs.outPath;
      } // args;
  in
  {
    lib.testFramework = {
      mkTests = args: vpsadminos.lib.testFramework.mkTests (withTestFrameworkDefaults args);
      mkTestsMeta = args: vpsadminos.lib.testFramework.mkTestsMeta (withTestFrameworkDefaults args);
    };
  };
}

If you do not export lib.testFramework, the runner still works without --test-config, but --test-config will fail with a clear error.

4) Expose the runner and add a wrapper script

Expose the upstream runner as an app:

{
  outputs = { self, nixpkgs, vpsadminos, ... }:
  let
    systems = [ "x86_64-linux" ];
    forAllSystems = nixpkgs.lib.genAttrs systems;
  in
  {
    apps = forAllSystems (system: {
      test-runner = {
        type = "app";
        program = "${vpsadminos.packages.${system}.test-runner}/bin/test-runner";
      };
    });

    # Optional convenience
    packages = forAllSystems (system: {
      test-runner = vpsadminos.packages.${system}.test-runner;
    });
  };
}

Then add a small wrapper script in your repo root, e.g. test-runner.sh (so $PWD is correct):

#!/usr/bin/env bash
set -euo pipefail

ROOT="$(cd -- "$(dirname "$0")" && pwd)"
cd "$ROOT"

exec nix run .#test-runner -- "$@"

Make it executable:

chmod +x test-runner.sh

The runner loads tests/runner/extensions/*.rb (if present) relative to the current directory, so it is important to run it from the flake root. Generated test configs, logs and VM state are stored under the selected state directory for each run.

5) Run the suite

Invoke the wrapper with the usual runner commands:

  • ./test-runner.sh ls to list available tests.
  • ./test-runner.sh test hello/boot to run a single test.
  • ./test-runner.sh test 'hello/*' to run selected tests.
  • ./test-runner.sh test -t ci --filter 'tag=vps || tag=storage' to combine simple tag filters with a metadata expression.
  • ./test-runner.sh ls --filter 'tag=ci && runtime!=long' to list tests selected by a boolean metadata expression.
  • ./test-runner.sh debug hello/boot to open the interactive REPL.

--filter expressions can match tags through the virtual tag key and labels through their label name. Use && for AND, || for OR and parentheses for grouping, for example tag=ci && (component=web || component=api).

6) Extend the runner when needed (optional)

Custom helpers can be added under tests/runner/extensions/. vpsAdmin adds tests/runner/extensions/vpsadmin_services.rb, which: - wraps vpsadminctl to make JSON-friendly succeeds/fails helpers; - registers a custom machine class for machines tagged vpsadmin-services: ```ruby TestRunner::Hook.subscribe(:machine_class_for) do |machine_config| next unless machine_config.tags.include?('vpsadmin-services')

VpsadminServicesMachine

end `` Use the same hook points to add helpers for your own services or machine types. Two more hooks are available for post-run diagnostics: -:after_test_runreceives the test, scripts, machines and aTestRunner::TestResult. -:after_test_script_runreceives the test, machines and aTestRunner::TestScriptResult`. Example: gather logs only when a script ends with an unexpected result:

TestRunner::Hook.subscribe(:after_test_script_run) do |script_result:, machines:, **|
  next if script_result.expected_result?

  machines.each_value do |machine|
    next unless machine.can_execute?

    machine.execute("journalctl -n 200 --unit #{script_result.test_script.name}")
  end
end

Common gotchas

  • nix eval testsMeta failed: the runner must be executed from the directory that contains your flake.nix and exports testsMeta.
  • testFramework is missing: ensure tests/all-tests.nix accepts testFramework and uses testFramework.makeTestLib, and tests/make-test.nix uses testFramework.makeTest.
  • --test-config fails with a lib.testFramework error: exportlib.testFramework.mkTestsandlib.testFramework.mkTestsMetaif you want the runner to re-evaluate your suite with--test-config. Repositories that do not use--test-configonly needtestsandtestsMeta`.

Persistent machines in custom runners

OSVM reuses managed disks on startup by default, including NixOS root disks. Custom runners retain disks by stopping and cleaning up machines without calling destroy or destroy_disks. Set preserve = false on an individual disk to recreate it at each start. Explicit destruction still removes all managed disks. See Testing for test/debug modes, --fresh, and disk configuration examples.

New disk contents are prepared in a temporary file and renamed only after preparation succeeds. A failed or interrupted copy cannot become a retained root image.

A retained NixOS root must contain the system closure selected for direct boot. Copy and activate configuration changes in the running VM before stopping it and starting it with the new configuration. Disk preservation does not copy new closures into an existing root or migrate its data.