Skip to content

Release 0.17.0: Add expect-failure module for negative testing

Horizon Bot requested to merge add-expect-failure-module into master

Summary

This release adds a generic expect-failure module that dramatically simplifies negative testing in gitlab-ci.

Add expect-failure module

  • Generic module for testing jobs that are expected to fail
  • Automatically extends jobs with allow_failure and artifact capture
  • Generates verify-{jobName} jobs that check the job failed as expected
  • Eliminates massive boilerplate in negative tests
  • Reduces configuration from ~40 lines to ~5 lines per test

How it works

  1. List jobs that should fail: expect-failure.jobs.{jobName} = { description = "..."; }
  2. Module extends the job with failure handling (allow_failure, after_script, artifacts)
  3. Module generates a verify job that reads the artifact and confirms failure
  4. Verify job passes when the original job fails (negative test succeeds)

Example usage

imports = [
  inputs.gitlab-ci.modules.gitlab-ci.flake
  inputs.gitlab-ci.modules.gitlab-ci.expect-failure
];

flake.strict-inputs.enable = true;

expect-failure.jobs.flake-strict-inputs = {
  description = "strict-inputs validation correctly rejected nixos-unstable";
};

Refactored negative tests

  • Updated strict-inputs, flake-lock, and flake-show tests to use the module
  • Consistent pattern across all negative tests
  • Much cleaner and more maintainable

Changes

  • Add nix/modules/gitlab-ci/expect-failure.nix
  • Refactor test/negative/*/nix/configuration.nix to use expect-failure module
  • Update ChangeLog to 0.17.0

🤖 Generated with Claude Code

Merge request reports