Skip to content

Conversation

@gbudjeakp
Copy link
Contributor

@gbudjeakp gbudjeakp commented Nov 5, 2025

This PR implements QUnit.config.testFilter, a new configuration option that allows programmatic filtering of tests at runtime through a callback function.

Context:
Currently, QUnit provides filtering through CLI parameters (--filter, --module, etc.) and test-level methods test.only(), test.skip(), but there's no way to implement dynamic, programmatic filtering based on runtime conditions without modifying test code.

This feature enables several important use cases:

  • Quarantining flaky tests in CI without modifying test code
  • Parallel test execution by distributing tests across workers via hash-based sharding
  • Runtime capability detection to skip tests for unavailable browser features (WebGL, IndexedDB, etc.)
  • Dynamic test selection based on external data sources (APIs, configuration files)

Fixes #1814.

@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented Nov 5, 2025

CLA Signed

The committers listed above are authorized under a signed CLA.

  • ✅ login: gbudjeakp / name: Akpevwen Sebastian Gbudje (b795e77)

@gbudjeakp gbudjeakp marked this pull request as ready for review November 10, 2025 04:42
Krinkle added a commit that referenced this pull request Dec 30, 2025
…o()`

Improve discovery by adding likely search terms to the description
of these methods.

Ref #637.
Ref #1080.
Ref #1815.

1. **Test-level filters** run first: [`test.only()`](../QUnit/test.only.md), [`test.skip()`](../QUnit/test.skip.md), [`test.if()`](../QUnit/test.if.md)
2. **Programmatic filter** runs next: `QUnit.config.testFilter`
3. **CLI/URL filters** run last: [`--filter`](./filter.md), [`--module`](./module.md), [`--moduleId`](./moduleId.md), [`--testId`](./testId.md)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In terms of code and execution order it is:

  • Test methods test.only narrows irrevocably, removal is final.
  • Test methods test.if and test.skip disable irrevocably but don't remove (later layers see it).
  • config.moduleId, narrows irrevocably.
  • config.testId, narrows, and inclusion is immediate and final. Later layers don't even run.
  • config.module, narrows irrevocably.
  • config.filter, narrows irrevocably.
  • config.testFilter, narrows further, seeing ony what hasn't been filtered out before, except for if/skip, which do get seen here. A testFilter cannot re-enable a skipped tesrt, but it can decide whether to filter. The effect is whether or not the skipped test is reported. This makes sense when distributing tests, because a skipped test should be allocated and reported on, once, just like any other test.

The order documented here is different, but maybe the order doesn't matter? Is there a specific meaning conveyed with this order? Would it mean something different to you as a user, if the order was described as 1) CLI, 2) Programmatic, 3) Test-level? This isn't a suggestion, I'm only asking as example, to help understand intent.

Copy link
Contributor Author

@gbudjeakp gbudjeakp Dec 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I structured the docs conceptually (test-level → programmatic → CLI) thinking it would be easier to understand ( That's the way I was thinking about it at the time in my head with the scenarios I was thinking off). The order doesn't matter here but from a users perspective(especially for debugging), that might be a bit confusing. I should update the doc to match.

The correct order is:

  1. Test-level filters run first: test.only(), test.skip(), test.if()
  2. CLI/URL filters run last: --filter, --module, --moduleId, --testId
  3. Programmatic filter runs last: QUnit.config.testFilter

Copy link
Member

@Krinkle Krinkle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm landing the 2.x backport first, so the docs review is non-blocking. No rush :)

Krinkle pushed a commit that referenced this pull request Jan 1, 2026
@Krinkle Krinkle force-pushed the feature/test-filter branch from b773f42 to b795e77 Compare January 1, 2026 11:01
@Krinkle
Copy link
Member

Krinkle commented Jan 1, 2026

I've landed a minimal version in 1ee3e2c for the 2.25.0, docs site, and release notes. I've then rebased this PR to iterate on the docs further.

@Krinkle Krinkle changed the title Core: Add QUnit.config.testFilter Docs: Expand QUnit.config.testFilter docs Jan 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

Add new config callback to programmatically filter tests

2 participants