Testing
vpsAdminOS has a framework for writing and running tests. It is similar to tests on NixOS, but it is a different implementation.
Tests are run on one or more virtual machines running vpsAdminOS. These machines are managed by the test framework. All tests can be found in the vpsAdminOS repository.
Writing a test
Each test is a Nix file stored in directory tests/suite. A test has a name,
one or more virtual machines to run on and a Ruby script that is run from
the host system and which can interact with the virtual machines.
import ../make-test.nix ({ pkgs }: {
name = "my-test";
description = ''
It's a great test indeed
'';
machine = import ../machines/empty.nix pkgs;
testScript = ''
machine.start
machine.succeeds("shell command that must succeed...")
'';
})
Different machines may be needed for various storage, configuration or clustering
tests. If only one machine is needed, it is simply called machine and declared
as such. More machines can be defined as:
import ../make-test.nix ({ pkgs }: {
name = "my-test";
description = ''
It's a great test indeed
'';
machines = {
first = import ../machines/empty.nix pkgs;
second = import ../machines/empty.nix pkgs;
};
testScript = ''
first.start
second.start
'';
})
Disk lifecycle
Managed file-backed disks preserve their contents on VM startup by default.
This includes NixOS root disks. A stop/start within a test reuses those disks;
set preserve = false on a disk to recreate it on each start.
vpsAdminOS VMs booted from squashfs still have a temporary root filesystem. Files in that root disappear on reboot, while data on attached disks, including ZFS pools and containers, remains. Firmware-booted installations keep their installed root on the attached disk.
| Command | Initial reset | Cleanup on exit |
|---|---|---|
test |
None | Delete managed disks |
test --fresh |
All managed disks, once per attempt | Delete managed disks |
test --no-destructive |
None | Keep disks |
test --no-destructive --fresh |
All managed disks, once per attempt | Keep disks |
debug |
None | Keep disks |
debug --fresh |
All managed disks, before the REPL opens | Keep disks |
The initial reset from -f, --fresh applies regardless of preserve. It does
not reset disks again between examples or VM restarts. A disk with
preserve = false is recreated when the VM starts, even in debug mode.
machine.destroy_disks explicitly removes managed disks regardless of their
preservation setting. Block devices and files with create = false are
externally managed and are never created, replaced, or deleted by these actions.
They cannot use preserve = false.
Test and debug use the same state directory for a given test path and
--state-dir. For example, retain a failed run and inspect it with:
./test-runner.sh test --no-destructive --fresh driver/nixos
./test-runner.sh debug driver/nixos
Each test has a lock held through VM cleanup and result publication. A second
command targeting the same state fails while that lock is held. Scripts of the
same test share its state. Use separate --state-dir values for independent
runs, including tests from different repositories.
Retaining a disk does not resize it or copy a newer source image into it.
NixOS direct boot requires the selected system closure to exist on its root
filesystem. After changing a test's NixOS configuration, use --fresh to start
from its new image, or copy and activate the closure in the running VM before
restarting it.
A managed disk can use image = /path/to/source.img instead of size. OSVM
copies the image when creating the disk. Preparation uses a temporary file next
to the destination; a failed copy or resize leaves the previous disk intact.
For a NixOS machine, rootDisk overrides the generated root disk descriptor:
machine = {
spin = "nixos";
rootDisk.preserve = false;
config.system.stateVersion = "26.05";
};
The generated root uses {machine}-root.img and is attached before additional
disks. Root and additional disks use the same device, type, create,
preserve, image, and size fields. JSON configurations using diskImage
remain readable; they cannot also specify rootDisk.
Disks can be added as:
import ../make-test.nix ({ pkgs }: {
name = "my-test";
description = ''
It's a great test indeed
'';
machine = {
# List of disk devices
disks = [
# 10 GB file sda.img will be created in the test's state directory
# and added to the virtual machine
{ type = "file"; device = "sda.img"; size = "10G"; }
# This scratch disk is blank on every VM start
{ type = "file"; device = "scratch.img"; size = "1G"; preserve = false; }
];
# Machine configuration
config = {
imports = [ ../configs/base.nix ];
boot.zfs.pools.tank = {
layout = [
{ devices = [ "sda" ]; }
];
doCreate = true;
install = true;
};
};
};
testScript = ''
machine.start
machine.wait_for_zpool("tank")
'';
})
See template machine configs in tests/machines/. vpsAdminOS configurations
used by machines for testing can be found in tests/configs and the tests
themselves in tests/suite/.
All tests have to be registered in tests/all-tests.nix, otherwise they cannot
be run.
Machine spins (vpsAdminOS vs NixOS)
Machines default to vpsAdminOS. To boot NixOS instead, set spin = "nixos" on
the machine definition and provide the NixOS config you want to test. You can
mix vpsAdminOS and NixOS machines within a single test.
Minimal NixOS example:
import ../make-test.nix ({ pkgs }: {
name = "nixos-example";
description = "NixOS machine example";
machine = {
spin = "nixos";
config = {
networking.hostName = "nixos";
virtualisation.memorySize = 2048;
virtualisation.cores = 2;
};
# optional extra modules
# modules = [ ./extra-module.nix ];
};
testScript = ''
machine.start
machine.wait_for_boot
machine.wait_for_service("test-shell")
machine.succeeds("echo hello")
'';
})
Running tests
To run the entire test suite, use:
./test-runner.sh test
Selected tests can be pattern-matched, e.g.:
./test-runner.sh test 'docker/*'
Tests can also be selected by tags, labels and metadata expressions. Tags are
listed in the test definition using tags = [ ... ];; labels are listed using
labels = { ... };. Test scripts inherit test-level tags and labels, and can
add or override their own metadata.
Existing tag and label filters are simple AND filters:
./test-runner.sh test -t ci
./test-runner.sh test -t ci -t docker
./test-runner.sh test -t ^manual
./test-runner.sh test -l runtime=long
./test-runner.sh test -l runtime!=long
For more advanced selection, use --filter with a metadata expression. The
virtual label tag matches tags; other names match labels. Expressions use
&& for AND, || for OR and parentheses for grouping:
./test-runner.sh ls --filter 'tag=ci && tag!=manual'
./test-runner.sh ls --filter 'tag=ci && runtime!=long'
./test-runner.sh test -t ci --filter 'tag=vps || tag=storage'
./test-runner.sh test --filter 'tag=ci && (tag=docker || tag=podman)'
./test-runner.sh test -t ci --filter 'runtime!=long && (tag=docker || tag=podman)'
Shell quoting is required for expressions containing &&, || or parentheses.
While developing a test, it is possible to start it with an interactive Ruby REPL:
./test-runner.sh debug my-test
The REPL can be used to issue the same commands as in the test script. The test
script itself can be run by calling method test_script. You can call method
breakpoint from inside the test to open the REPL from any point of execution.
Explicit ZFS full suite
OpenZFS full-suite execution is available as an explicit test-runner tag and is
not part of default -t ci runs. Run these commands from the repository root.
# Full profile (long run)
./run-zfs-full-suite.sh
# Quicker profiles
PROFILE=sanity ./run-zfs-full-suite.sh
PROFILE=smoke ./run-zfs-full-suite.sh
# Start multiple runs in parallel by using unique state dirs/run ids
RUN_ID=zfs-full-a ./run-zfs-full-suite.sh
RUN_ID=zfs-full-b ./run-zfs-full-suite.sh
The wrapper derives its default state directory from RUN_ID. Give each
parallel run a unique RUN_ID and do not reuse a STATE_DIR override.
Userspace upgrades between versions
The ordinary upgrade/impermanence-from-2abbe3cc1c10@v1 and @v2 cases validate a
production-origin container run through a candidate userspace switch. They pin
host source 2abbe3cc1c102a4b39022d3f73ef4a432ad4d616 with its stable/default
6.12.113 kernel, then activate the checkout being tested with its stable profile.
They ignore VPSADMINOS_CONFIG and retain the booted kernel and host boot ID.
The pin identifies a source boundary; it does not inventory live node kernels.
The two cgroup modes use one NixOS impermanent container each and carry both
upgrade and ci tags.
./test-runner.sh ls 'upgrade/impermanence-from-2abbe3cc1c10@*'
The stopped container's supported cpu-package none preference prevents an
incidental scheduler save. The production writer must naturally leave valid
saved run identity with stale base-dataset/cleanup metadata while a unique GC
owner and held live root identify the temporary dataset. Missing stale-state
coverage fails explicitly; the test does not synthesize metadata or retry it.
Candidate import must preserve that run, root and both payloads while repairing
only the saved selection. A daemon reload must retain the corrected file
without rewriting it. A new candidate-created run must already persist its
new selection before other setters, survive one reload with its payloads,
and retire temporary roots through ordinary stop/cleanup. Existing focused
units cover refusal, races and GC fencing; custom boot remains unit-only.
Userspace upgrade-test lifecycle
Name each top-level case or template
upgrade/<behavior>-from-<12charSHA>, with ordinary instance suffixes such as
@v1 and @v2. Use a kernel name only when the kernel identifies the
transition being tested. Pin each baseline to its full immutable revision;
never silently repin an existing case.
Add a comment header to each top-level case or template with these labels:
Baseline: full revision, profile and any intermediate upgrade hops.Purpose: the supported transition and data boundary the test protects.Target: the checkout being tested and its explicit profile.Removal condition: the support boundary or replacement coverage that permits retirement.Retirement evidence: confirmation references, or an explicit statement that the evidence is not established.
Exercise the actual transition and assert that expected state and data survive. A fresh target boot does not establish upgrade coverage. Retire a case only after its support boundary explicitly ends or verified replacement coverage preserves its guarantees. Remove its registration, exclusive helpers/specs and obsolete documentation together. Age, flaky failures and missing caches are not retirement grounds. Record the evidence confirming retirement in the removal commit.
The impermanence transition must remain until support for equivalent old saved/running state ends, including supported rollback and restoration paths, or a verified real-transition replacement covers old-writer repair, live run/root/payload continuity, reload stability, new-writer persistence and ordinary cleanup. Record the support/migration decision or the exact replacement and successful native results. This retirement evidence is not established. Deploying fixed userspace alone does not prove that old saved state is gone.
Keep focused writer, import, refusal, race and GC unit regressions while their behavior is supported. They do not replace the real transition test.
Numbered livepatch qualification
Numbered kernel livepatch releases use the catalog-driven process in Kernel livepatch release process. Maintainers discover one versioned plan, capture the selected target-bound producer once and verify that same capture in every consumer. The plan, capture and result contracts carry the detailed identity and coverage data; workflow job names and temporary artifact paths are not maintainer interfaces.
The stable entry points are tools/livepatch/discover.sh,
tools/livepatch/capture.sh, tools/livepatch/verify.sh and
tools/livepatch/run-profile.sh. Generic test selection uses
kernel/livepatch-batch@<row-id>, kernel/livepatch-set@<row-id> and
kernel/livepatch-fix@<row-id>. The canonical process defines the exact
commands, required AMD and Intel/KVM profiles, every-fix oracle coverage and
whole-stack forward/reverse qualification.
Stable-kernel NFS cancellation CI
The osctl/nfs-cancellation test is part of the ordinary ci selector. It
reads stableKernelVersion and the matching kernel record from
os/packages/linux/available-kernels.nix, then boots that kernel through the
normal vpsAdminOS stable-kernel package path. It has no separate kernel
revision, archive hash, ZFS pin or version-specific test registration. Updating
the stable kernel moves this regression to the new stable kernel automatically.
Before running examples, the guest verifies the actual kernel release and source revision. The suite covers NFSv3 and NFSv4 cancellation, per-server callback lifetime, namespace-wide terminal cancellation, NLM shutdown, sibling-client isolation, processless descendants, restart and admission behavior, and 25 forced stop/restart cycles. The teardown regression creates a private sysfs view and opens the shutdown control with initial-user-namespace authority. Its helper verifies and enters the user namespace that owns the container mount namespace, clones the mount tree without cross-owner locks, drops the NFS mount in the private slave clone, and re-enters the held original mount namespace for the final unmount.
The test uses the existing machines and per-example deadlines. It does not run in the livepatch qualification selectors.
Livepatch CPU hotplug diagnostics
Qualification checks kernel health after each hotplug and workload-progress check, using the example's original dmesg baseline. A warning before the transition fails that phase even when CPU online/offline and progress checks succeed. The ordinary workload, aging, ABI, quarantine and recovery requirements still apply.
Livepatch qualification shutdown diagnostics
Livepatch qualification guests retain a shutdown-only RSS diagnostic.
After workload cleanup, a return probe mirrors __mmdrop completion during
poweroff to the kernel console, with the return caller, task name and
online/dying CPU masks. Console tracing is allocated at boot but remains off
through the workload. Console verbosity is raised only after cleanup so the
records reach the serial console rather than only the dmesg ring. They help
investigate late accounting BUGs; they do not suppress faults or establish
that a particular BUG is harmless.
The counter-sum helper itself is not traceable on the pinned boot kernel;
the diagnostic respects that restriction and observes mm teardown instead.
If qualification already failed, its after-example hook reports unavailable diagnostics or cleanup without replacing the original example failure. A wedged cgroup migration can also prevent the guest from forking the commands needed to collect diagnostics. No successful cleanup or snapshot is inferred from this fallback. Cleanup exceptions after completed qualification still propagate and fail the test.
Guest kernel failures
Unexpected guest kernel failures remain fatal and are not retried. During
cleanup, the runner continues collecting the console so that a panic's later
stack traces and per-CPU tracing dumps can reach <machine>-console.log.
Collection stops at console EOF, after 30 seconds without new console bytes,
or 600 seconds after the first failure was detected, whichever comes first.
The runner then terminates the guest. New output, including repeated failure
messages, cannot extend that hard collection deadline.
This wait only affects failure cleanup: it does not suppress kernel-failure detection, permit further guest commands, or turn a failing test into a pass. A dump can still be incomplete if a collection limit is reached.
Console detection includes bad-RSS accounting BUGs reported during final shutdown. A successful script body and zero QEMU exit status do not clear a recorded kernel failure; the runner checks again after final console collection.
NFS cancellation lock diagnostics
The NFS cancellation test's remote-lock example enables a dedicated trace
instance while the FIFO-held lock and competing client are exercised. Its
nfs4, nfsd, sunrpc, and filelock event rings are bounded to 1 MiB per
virtual CPU. On failure, tracing is stopped and per-CPU loss statistics and
the retained events are collected before the holder is released, alongside
lock, mount, process, firewall and server-state snapshots. A wrapped ring is
not a complete operation history; use the recorded loss statistics.
The trace instance and enabled events are removed after the example. Capture and cleanup do not turn a failed lock assertion into a pass. Server diagnostics enter the server's mount, network and PID namespaces so its private proc mount can resolve the diagnostic process's network statistics.
Failure snapshots also retain each live flock process's descriptors and
fdinfo, plus device/inode/link-count observations for those descriptors and
the server/client lock paths. This distinguishes a live lock holder from a
stale or different path without changing the remote contention assertion.
These are post-failure observations: the short-lived contender has already
closed its descriptor, and path metadata can require fresh NFS requests.
Use the retained trace and its loss statistics to establish operation-time
filehandle identity; a matching pathname alone is not proof.
A separate transport ring starts before each protocol's first client mounts.
It records inet_sock_set_state, tcp_send_reset, and tcp_receive_reset,
with 1 MiB per virtual CPU. State changes and received resets are filtered
to source or destination port 2049. Sent resets have packed socket addresses
rather than scalar port fields, so they are retained without a port filter. A
callback connection can predate the lock example, so this history is not
cleared when the operation-time NFS trace starts. Lock failure collects both
rings and their loss statistics, plus server/client TCP socket snapshots.
The transport instance is removed after the protocol group. Socket snapshots
are post-failure observations, and an overwritten ring cannot establish that
an earlier disconnect or reset was absent. No lock or cancellation assertion
depends on a particular transport topology or on the diagnostic contents.
Preparing images before running tests
./test-runner.sh test --prebuild --jobs auto -t ci builds the configurations
and image closures for the selected tests before starting any selected VM.
Patterns, script selectors, tags, labels and metadata expressions keep their
usual meaning. Several selected scripts from one test share one prepared
configuration. The prepared JSON and its closure stay GC-rooted through all
attempts and VM cleanup; runtime loads those exact configurations without
invoking Nix. A test script or extension that explicitly starts a separate host
build is outside this boundary.
Prebuild is opt-in. --no-prebuild keeps per-test image preparation. Both modes
use the same test squashfs cache default and runtime resource scheduling.
debug continues to prepare one test when it starts. CI enables prebuild for
native tests; the manual prebuild boolean can disable it.
Preparation snapshots VM capacity before building, then refreshes assigned
limits before starting VMs. It chooses local Nix build jobs separately from
--jobs, using available host/cgroup memory bounded by assigned capacity and
--max-memory-mib. After subtracting --memory-reserve-mib (default 8192), it
divides the usable memory by --prebuild-memory-per-job-mib (default 8192),
rounds down, and caps the result at inherited local Nix max-jobs. The estimate
must be positive. Unknown headroom requires an explicit usable maximum;
insufficient capacity fails before any VM starts. Runtime memory overcommit is
not applied to preparation sizing. Inherited max-jobs=0 keeps remote-only or
substitution behavior; local headroom does not size remote builders.
The 8192 MiB estimate is an admission heuristic, not a derivation RSS limit.
Preparation changes neither Nix cores nor kernel/compression CPU parallelism.
Evaluation, substitution and daemon overhead, one unusually heavy derivation,
and unrelated builds can exceed the estimate. CI sizing assumes one worker job
per runner container without a competing test or build workload.
Preparation errors are reported as infrastructure/build failures, with the
selected identities, full build log path, and exit code or signal. They start
zero selected VMs and do not fabricate script assertion results. Cancellation
reaps the owned Nix client and releases its root; shared daemon work is not
killed. Retrying the command reuses valid cached outputs. Disk reset through
--fresh begins only after complete preparation succeeds.
Squashfs cache
vpsAdminOS test images default to system.squashfsMemoryMiB = 2048, passed to
both sqfstar and mksquashfs. This bounds the compressor cache, not total
build memory. Metadata, dictionaries, compression contexts and other processes
use additional memory. The setting preserves CPU parallelism, compression
settings (zstd level 10 by default) and image metadata. Override it when needed;
null restores the tool default. Production defaults to null. Historical
pinned OS sources and unrelated Nixpkgs image builders do not inherit this
setting.
Runtime resource scheduling
The scheduler budgets RAM, /dev/shm and CPUs across concurrent tests. If no
pending test fits and no test is running, it warns and runs one test alone,
even when that test exceeds a budget. Guest memory, CPU counts and shared-memory
requests are unchanged. The test may exhaust the host or fail from OOM.
Expected failure
A test can be expected to fail. The failure is shown, but it does not result in error exit status. If a test succeeds and we expected it to fail, it is considered as an error.
import ../make-test.nix ({ pkgs }: {
name = "my-failed-test";
description = ''
It's a great test indeed
'';
expectFailure = true;
machine = import ../machines/empty.nix pkgs;
testScript = ''
machine.start
machine.succeeds("shell command that fails...")
'';
})
Multiple test scripts
Each test can contain multiple test scripts. Test scripts are a part of test name and be run selectively.
import ../make-test.nix ({ pkgs }: {
name = "my-test";
description = ''
It's a great test indeed
'';
expectFailure = true;
machine = import ../machines/empty.nix pkgs;
testScripts = {
script1 = {
# Each script be expected to fail
# expectFailure = false;
# The test itself
script = ''
machine.start
machine.succeeds("uptime")
'';
};
script2 = {
script = ''
machine.succeeds("ps aux")
'';
};
};
})
./test-runner.sh ls 'my-test#*' would show these tests as:
my-test#script1
my-test#script2
Script name is separated from test name by a hash (#). Test scripts of
one test are run in the same environment and share the same virtual machines.
By default, they run one after the other in random order.
Tests can opt into parallel test script execution with testScriptJobs:
{
testScriptJobs = 2;
testScripts = {
script1.script = ''
machine.succeeds("uptime")
'';
script2.script = ''
machine.succeeds("ps aux")
'';
};
}
Parallel scripts share the same VM state, so they must avoid conflicting changes to the machine. Each running script gets its own test shell.
Machines can declare named extra shells for commands that should be able to run without blocking the script's default shell:
{
machine = {
shells = [ "first" "second" ];
};
testScript = ''
machine.shells['first'].succeeds("sleep 10")
machine.succeeds("uptime", shell: "second")
'';
}
Named shells share the same VM state as the default shell. Shell names can be
looked up as strings or symbols, e.g. machine.shells[:first].
Generated direct-boot test guests use flow-controlled virtserialport devices
for command traffic, separate from the serial console used for boot and kernel
diagnostics. Unlike virtconsole, these ports do not discard shell replies
when the host socket is temporarily full. Each guest service opens its named
/dev/virtio-ports/org.osvm.shellN device once and shares that descriptor for
stdin, stdout and stderr; it is not a TTY and does not use stty.
The machine JSON records testShellTransport, which can be virtserialport
or virtconsole. Old JSON without this field keeps the legacy /dev/hvcN
transport. Firmware-boot machines also default to virtconsole, because their
guest image is supplied separately. A machine can explicitly set
testShellTransport when its image needs a different transport; the guest
service and host port must agree. For separately built vpsAdminOS images, the
matching guest option is osctl.test-shell.transport.
Test templates
Templates can be used to create multiple instances of a test. The difference between templates and multiple test scripts is that tests created by templates are isolated, have their own virtual machines and can run in parallel.
import ../make-template.nix ({ distribution, version }: rec {
instance = "${distribution}-${version}";
test = { pkgs }: {
name = "my-template@${instance}";
description = ''
Test something on ${distribution}-${version}
'';
machine = import ../machines/tank.nix pkgs;
testScript = ''
machine.wait_for_osctl_pool("tank")
machine.wait_until_online
machine.succeeds("osctl ct new --distribution ${distribution} --version ${version} testct")
'';
};
})
Within all-tests.nix, the template would be listed as:
{ template = "my-template"; instances = distributions.all; }
./test-runner.sh ls 'my-template@*' would show these tests as:
my-template@debian-stable
my-template@ubuntu-24.04
...
RSpec expectations
Test scripts can use RSpec expectations and optionally also example groups similar to RSpec. We reuse rspec-expectations and provide our own implementation of rspec-core. We aim to be compatible with RSpec when possible.
The following test demostrates available RSpec features.
import ../make-test.nix ({ pkgs }: {
name = "my-test";
description = ''
Test with RSpec expectations
'';
machine = import ../machines/tank.nix pkgs;
testScript = ''
# Optional global configuration for example groups / examples
configure_examples do |config|
# `:defined`, `:rand`, instance of `Random` or `Integer` used as a seed
# Defaults to `:rand` if not set
config.default_order = :defined
end
before(:suite) do
puts 'block executed before all examples'
end
# Create an example group
describe 'machine' do
before(:context) do
puts 'block executed before examples in this group'
end
after(:context) do
puts 'block executed after examples in this group'
end
before(:example) do
puts 'block executed before each example'
end
after(:example) do
puts 'block executed after each example'
end
it 'can execute commands' do
_, output = machine.succeeds('echo hello')
expect(output.strip).to eq('hello')
end
example 'without a block is skipped'
skip 'examples created by skip are also skipped' do
puts 'this will not run'
end
example 'can be skipped from the code block' do
skip
skip('with a reason')
end
pending 'examples are expected to fail' do
# This example will fail if expectations will be met
expect(0).to eq(1)
end
example 'can be marked as pending from the code block' do
pending
pending('this is expected to fail')
expect(0).to eq(1)
end
context 'nested example group with custom order of evaluation', order: :rand do
example '#1'
example '#2'
end
end
after(:suite) do
puts 'block executed after all examples'
end
'';
})
Groups and examples are evaluated in random order unless configured otherwise.
Temporary config changes
It is possible to change all test machine configurations by creating
os/configs/tests.nix file, e.g. to change a kernel version used in tests:
{ config, ... }:
{
boot.kernelVersion = "6.1.30";
}