Testing #

Running Tests #


pulp test                    # Run all tests
ctest --test-dir build       # Equivalent, direct CTest
./validate-build.sh          # Detached clean worktree configure/build/test
./validate-build.sh --smoke  # Fast clean install/export preflight, no tests

Filter tests by name:


pulp test -R Gain            # Tests matching "Gain"
pulp test -R "golden"        # Golden-file audio tests
ctest --test-dir build -R clap-dlopen   # CLAP load tests

Use ./validate-build.sh --quiet when you want a detached clean-build outer loop that ignores incremental build state and only prints logs on failure. It validates committed HEAD in a detached worktree, then runs a fresh configure, build, install, an installed SDK find_package(Pulp) smoke configure, and an optional test pass. This is the quickest way to catch clean-build drift, basic install/export breakage, and missing setup dependencies before opening a PR. Run ./tools/check-docs.sh separately for manifest and README/docs consistency checks.

Use ./validate-build.sh --smoke when you want the fastest truthful preflight for build-system and SDK export work. Smoke mode still bootstraps dependencies, does a clean detached configure/build/install, and runs the installed-SDK find_package(Pulp) smoke configure, but disables tests, examples, and GPU in that clean build so you can catch install/export regressions before paying for a full validation run.

Repeat Until Fail #

When a test or command only fails intermittently, use the repeat helper instead of copy-pasting shell loops:


tools/scripts/repeat-until-fail.sh 100 -- ctest --test-dir build -R "SeqLock concurrent stress test" --output-on-failure
tools/scripts/repeat-until-fail.sh -- ctest --test-dir build -R "OSC sender/receiver loopback" --output-on-failure

The helper prints the current iteration, exports it as REPEAT_UNTIL_FAIL_ITERATION, and stops on the first failure. This is useful for debugging flaky concurrency, timing, path, and validator issues without inventing a new one-off loop every time.

Test Count #

The current branch has 1622+ registered tests covering:

Unit tests for subsystems (runtime, state, audio, midi, signal, format, events, canvas, render, view, osc)
Golden-file audio comparison tests
Plugin format validation (CLAP dlopen, VST3 pluginval, AU auval, optional local AAX validation)
Example plugin processing tests

Test Layers #

Layer	What	When
Unit tests	Individual functions and classes	Every build
Golden-file tests	DSP output matches reference audio files	Every build
Format validation	Plugins load and pass format validators	`pulp validate`
Example tests	Each example plugin has a processing test	Every build

Plugin Validation #


pulp validate

This runs:

CLAP: clap-validator if installed, otherwise falls back to dlopen checks
VST3: pluginval if installed, otherwise skips cleanly
AU: auval on macOS if the component is installed
AAX: DigiShell + AAX Validator on macOS/Windows if installed; otherwise the CLI reports a guided skip

Validation is separate from unit tests. Unit tests verify internal behavior; validation verifies that the built plugin binary loads and behaves correctly from the host's perspective. Public CI does not build or validate AAX because the SDK and validator are developer-supplied and not bundled by Pulp.

Design Tool Debugging #

The design tool has its own headless harness:


pulp design-debug --prompt "make the gain knob look like macOS 7" --target k1

Use this when you want reproducible before/after/diff screenshots plus JSON metadata for AI-driven restyling runs. The report captures provider/model/reasoning-effort, request text, token changes, widget look ids, screenshot diff metrics, and the render backend used for the capture.

The default --capture-backend skia path renders widget SkSL offscreen, which makes it useful for shader-aware pipeline verification. The remaining limitation is that it still does not prove final live GPU presentation parity in the interactive app.

Writing Tests #

Tests use Catch2. Test files live in test/ or alongside examples.

Example test structure:


#include <catch2/catch_test_macros.hpp>
#include <pulp/format/headless.hpp>
#include "my_processor.hpp"

TEST_CASE("MyPlugin processes audio") {
    pulp::format::HeadlessHost host(create_my_processor);
    host.prepare(48000.0, 512);

    pulp::audio::Buffer<float> in(2, 512), out(2, 512);
    // Fill input, process, check output...
}

Golden-File Tests #

Golden-file tests render known input through a processor and compare the output against a reference file. If the output differs beyond a tolerance, the test fails.

This catches unintended DSP regressions.

Test Expectations #

New public behavior needs tests
Bug fixes should add or adjust a test when practical
Example plugins each include a test file (test_pulp_*.cpp)