Proposal to delete t/old-bugs.{t,data}#1487
Merged
marc-vanderwal merged 2 commits intozonemaster:release/v2025.2.1from Feb 9, 2026
Merged
Proposal to delete t/old-bugs.{t,data}#1487marc-vanderwal merged 2 commits intozonemaster:release/v2025.2.1from
marc-vanderwal merged 2 commits intozonemaster:release/v2025.2.1from
Conversation
Merge pull request zonemaster#1483 from zonemaster/develop
These tests are very obsolete and depend on live data returned by name servers on the Internet. Because of that, they cannot be rerecorded anymore. The tests in question are also largely superseded by the newer scenario-based tests. Deleting these files does not affect the coverage of the test suite in general, except for two very specific places in the entire codebase of Zonemaster-Engine. After careful analysis, it turns out that restoring that coverage by means of scenarios isn’t trivial. The affected code implements the Delegation03 and Nameserver01 test cases. The associated specifications will need some rework and it is likely that the code will be entirely overhauled in the future.
marc-vanderwal
requested review from
MichaelTimbert,
matsduf,
mattias-p and
tgreenx
marc-vanderwal
added
the
RC-None
matsduf
previously approved these changes
Jan 8, 2026
Contributor
matsduf
left a comment
matsduf
left a comment
There was a problem hiding this comment.
Even if we loose some coverage in the short run, I think it is much better to focus on the new scenario model and remove legacy tests that we cannot control
matsduf
approved these changes
Jan 28, 2026
tgreenx
approved these changes
Feb 9, 2026
tgreenx
added
the
V-Patch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
This PR removes
t/old-bugs.tand its associated data file from the test suite.These tests are very obsolete and depend on live data returned by name servers on the Internet. Because of that, much of the data cannot be rerecorded anymore. The tests in question are also largely superseded by the newer scenario-based tests.
Deleting these files does not reduce the coverage of the test suite in general, except for two very specific places in the entire codebase of Zonemaster-Engine that probably need to be overhauled anyway.
After careful analysis, it turns out that the loss of coverage is the consequence of unclear or incomplete specifications and it would make more sense to rework the specifications and their implementations instead of adding band-aids to keep the coverage identical. The following issues were created in that context:
Context
While developing some of the features and fixes for the latest release (v2025.2), we’ve had problems due to some
.tfiles that were more or less difficult to fix because rerecording the corresponding data file was not always doable. Sometimes, I had to resort to extreme kludges to.datafiles to get tests passing again. One example is commit e24a21e, part of #1475.And
t/old-bugs.datais a prime example of a data file containing things that cannot ever be rerecorded, again, because the file is old and the Internet has moved on in the meantime.Changes
t/old-bugs.tandt/old-bugs.data.How to test this PR
Unit tests should still pass and coverage should stay unchanged, save for the two aforementioned places.
With
Devel::Coverinstalled, generate a coverage report on the codebase whilet/old-bugs.tand.dataare still there:Then check out this branch and generate another report:
Open the
cover_db.before/coverage.htmlandcover_db.after/coverage.htmlfiles side by side. The only difference you should see is before:and after (differences circled in red):
Open
cover_db.{before,after}/blib-lib-Zonemaster-Engine-Test-Delegation-pm--branch.htmlside by side. The only difference should be at line 502:(The circled area is green in the “before” report and red in the “after” report).
Open
http://localhost:8080/cover_db.after/blib-lib-Zonemaster-Engine-Test-Nameserver-pm--condition.htmlside by side. The only difference should be at line 700:(The area circled in red is green in the “before” report and red in the “after” report).