Using the vpsAdminOS test framework in other projects
The vpsAdminOS test framework can be reused without copying its sources:
- Nix code defines QEMU machines and turns test definitions under
tests/suite/into flake outputs. - The upstream Ruby runner (
vpsadminos#test-runner) evaluates your project's flake outputs, boots the machines and runs RubytestScripts against them.
The important bit: the runner always evaluates/builds tests from the current working directory.
By default it uses these flake outputs:
.#testsMeta.<system>(discovery:ls, tags/labels, templates, ...).#tests.<system>."<test-path>"(build JSON config for a test)
If you use test-runner --test-config, the runner re-evaluates the suite and
also needs an optional flake API:
.#lib.testFramework.mkTests.#lib.testFramework.mkTestsMeta
So to integrate the runner into another repository, your project needs:
- A Nix flake that exports
testsandtestsMeta. - A
tests/tree withall-tests.nix+suite/(and usuallymake-test.nix). - Optional:
lib.testFrameworkif you wanttest-runner --test-config.
For writing tests (machine definition, testScript, tags, multiple scripts),
see Testing.
Source snapshots and OS version metadata
Each runner invocation resolves the tested repository to one immutable Nix source and keeps it GC-rooted until the command finishes. Path snapshots retain the files selected by Nix, including untracked and ignored files. The runner captures the original repository's commit once and compares that commit with the snapshot through a temporary Git index. Included extra files and changed tracked content mark the snapshot dirty. The caller's index is unchanged; later checkout or HEAD changes do not change the captured identity.
The runner scopes TEST_RUNNER_SOURCE_INFO to that invocation. Its JSON object
contains sourcePath, nullable revision, shortRev and dirty. Git-free roots
have unknown identity; a Git repository enclosing a different source root is
not used as its provenance. The previous environment value is restored when the
command returns or raises.
lib.testFramework.sourceVersionModule supplies the same source revision,
dirty flag and version suffix defaults as normal flake OS construction. The
framework uses captured metadata only when sourcePath equals the vpsAdminOS
provider's source path. For example, a vpsAdmin test suite using a pinned
vpsAdminOS input retains that input's identity, rather than the vpsAdmin caller's
revision. Explicit module settings still override these defaults. Older
framework providers without this optional field remain supported; existing
packaged revision files remain the fallback when source identity is unknown.
Integration example (vpsAdmin)
The vpsAdmin repository reuses the framework directly via a flake input. The sections below describe the same setup in a copy/paste-friendly way.
Step-by-step integration (recommended, flake-based)
1) Add vpsAdminOS as a flake input
In your flake.nix, add vpsAdminOS and (recommended) align nixpkgs with it:
{
inputs = {
vpsadminos.url = "github:vpsfreecz/vpsadminos/<ref>";
nixpkgs.follows = "vpsadminos/nixpkgs";
};
outputs = { self, nixpkgs, vpsadminos, ... }:
let
systems = [ "x86_64-linux" ];
forAllSystems = nixpkgs.lib.genAttrs systems;
in
{
# filled in below
};
}
Tip for local development: point the input at a local checkout:
vpsadminos.url = "path:../vpsadminos";.
2) Add the tests/ tree
Create this minimal layout in your project:
tests/
all-tests.nix
make-test.nix
suite/
hello/
boot.nix
tests/make-test.nix should delegate to the upstream helper (this is what
each suite file imports):
testFn:
{ testFramework, ... }@args:
let
upstream = testFramework.makeTest testFn;
# Optional: pass extra args to NixOS/vpsAdminOS modules as `specialArgs`.
# The vpsAdmin repo uses this to make the vpsAdminOS checkout available as
# `vpsadminos` inside NixOS module evaluation.
mergedExtraArgs = {
vpsadminos = testFramework.sourcePath;
} // (args.extraArgs or { });
in
upstream (args // { extraArgs = mergedExtraArgs; })
tests/all-tests.nix lists tests (and templates) and is evaluated by the flake
helpers:
{
pkgs,
system,
testFramework,
suiteArgs ? { },
}:
let
nixpkgs = import pkgs { inherit system; };
lib = nixpkgs.lib;
testLib = testFramework.makeTestLib {
inherit pkgs system lib suiteArgs;
suitePath = ./suite;
};
in
testLib.makeTests [
"hello/boot"
]
And a minimal suite test, tests/suite/hello/boot.nix:
import ../../make-test.nix ({ ... }: {
name = "hello-boot";
description = "Boot a VM and run a simple command";
machine = {
spin = "nixos";
config = {
networking.hostName = "hello";
virtualisation.memorySize = 1024;
virtualisation.cores = 2;
};
};
testScript = ''
machine.start
machine.wait_for_boot
machine.wait_for_service("test-shell")
machine.succeeds("echo hello")
'';
})
Note: test paths in all-tests.nix map to files under tests/suite/ without
the .nix suffix, e.g. hello/boot -> tests/suite/hello/boot.nix.
3) Export flake outputs
Your flake must export tests and testsMeta so the runner can list and build
tests via nix eval/nix build:
{
outputs = { self, nixpkgs, vpsadminos, ... }:
let
systems = [ "x86_64-linux" ];
forAllSystems = nixpkgs.lib.genAttrs systems;
in
{
tests = forAllSystems (system:
vpsadminos.lib.testFramework.mkTests {
inherit system;
testsRoot = ./tests;
});
testsMeta = forAllSystems (system:
vpsadminos.lib.testFramework.mkTestsMeta {
inherit system;
testsRoot = ./tests;
});
};
}
If your project uses a different nixpkgs than vpsAdminOS, also pass
pkgsPath = nixpkgs.outPath; to both mkTests and mkTestsMeta.
This is enough for normal test-runner ls/test/debug usage.
3a) Support --test-config (optional)
If you want to use test-runner --test-config, also export
lib.testFramework. The runner uses it to re-evaluate the suite with the
provided Nix configuration.
{
outputs = { self, nixpkgs, vpsadminos, ... }:
let
withTestFrameworkDefaults = args:
{
pkgsPath = args.pkgsPath or nixpkgs.outPath;
} // args;
in
{
lib.testFramework = {
mkTests = args: vpsadminos.lib.testFramework.mkTests (withTestFrameworkDefaults args);
mkTestsMeta = args: vpsadminos.lib.testFramework.mkTestsMeta (withTestFrameworkDefaults args);
};
};
}
If you do not export lib.testFramework, the runner still works without
--test-config, but --test-config will fail with a clear error.
4) Expose the runner and add a wrapper script
Expose the upstream runner as an app:
{
outputs = { self, nixpkgs, vpsadminos, ... }:
let
systems = [ "x86_64-linux" ];
forAllSystems = nixpkgs.lib.genAttrs systems;
in
{
apps = forAllSystems (system: {
test-runner = {
type = "app";
program = "${vpsadminos.packages.${system}.test-runner}/bin/test-runner";
};
});
# Optional convenience
packages = forAllSystems (system: {
test-runner = vpsadminos.packages.${system}.test-runner;
});
};
}
Then add a small wrapper script in your repo root, e.g. test-runner.sh (so
$PWD is correct):
#!/usr/bin/env bash
set -euo pipefail
ROOT="$(cd -- "$(dirname "$0")" && pwd)"
cd "$ROOT"
exec nix run .#test-runner -- "$@"
Make it executable:
chmod +x test-runner.sh
The runner loads tests/runner/extensions/*.rb (if present) relative to the
current directory, so it is important to run it from the flake root. Generated
test configs, logs and VM state are stored under the selected state directory
for each run.
5) Run the suite
Invoke the wrapper with the usual runner commands:
./test-runner.sh lsto list available tests../test-runner.sh test hello/bootto run a single test../test-runner.sh test 'hello/*'to run selected tests../test-runner.sh test -t ci --filter 'tag=vps || tag=storage'to combine simple tag filters with a metadata expression../test-runner.sh ls --filter 'tag=ci && runtime!=long'to list tests selected by a boolean metadata expression../test-runner.sh debug hello/bootto open the interactive REPL.
--filter expressions can match tags through the virtual tag key and labels
through their label name. Use && for AND, || for OR and parentheses for
grouping, for example tag=ci && (component=web || component=api).
6) Extend the runner when needed (optional)
Custom helpers can be added under tests/runner/extensions/. vpsAdmin adds
tests/runner/extensions/vpsadmin_services.rb, which:
- wraps vpsadminctl to make JSON-friendly succeeds/fails helpers;
- registers a custom machine class for machines tagged vpsadmin-services:
```ruby
TestRunner::Hook.subscribe(:machine_class_for) do |machine_config|
next unless machine_config.tags.include?('vpsadmin-services')
VpsadminServicesMachine
end
``
Use the same hook points to add helpers for your own services or machine types.
Two more hooks are available for post-run diagnostics:
-:after_test_runreceives the test, scripts, machines and aTestRunner::TestResult.
-:after_test_script_runreceives the test, machines and aTestRunner::TestScriptResult`.
Example: gather logs only when a script ends with an unexpected result:
TestRunner::Hook.subscribe(:after_test_script_run) do |script_result:, machines:, **|
next if script_result.expected_result?
machines.each_value do |machine|
next unless machine.can_execute?
machine.execute("journalctl -n 200 --unit #{script_result.test_script.name}")
end
end
Common gotchas
nix eval testsMeta failed: the runner must be executed from the directory that contains yourflake.nixand exportstestsMeta.testFramework is missing: ensuretests/all-tests.nixacceptstestFrameworkand usestestFramework.makeTestLib, andtests/make-test.nixusestestFramework.makeTest.--test-configfails with alib.testFrameworkerror: exportlib.testFramework.mkTestsandlib.testFramework.mkTestsMetaif you want the runner to re-evaluate your suite with--test-config. Repositories that do not use--test-configonly needtestsandtestsMeta`.
Persistent machines in custom runners
OSVM reuses managed disks on startup by default, including NixOS root disks.
Custom runners retain disks by stopping and cleaning up machines without
calling destroy or destroy_disks. Set preserve = false on an individual
disk to recreate it at each start. Explicit destruction still removes all
managed disks. See Testing for test/debug
modes, --fresh, and disk configuration examples.
New disk contents are prepared in a temporary file and renamed only after preparation succeeds. A failed or interrupted copy cannot become a retained root image.
A retained NixOS root must contain the system closure selected for direct boot. Copy and activate configuration changes in the running VM before stopping it and starting it with the new configuration. Disk preservation does not copy new closures into an existing root or migrate its data.