Skip to content

Comments

[fix][test] Made ProtobufNativeSchemaTest.testSchema order-independent#24805

Merged
lhotari merged 1 commit intoapache:masterfrom
LucasEby:testSchemaNondeterministic
Nov 11, 2025
Merged

[fix][test] Made ProtobufNativeSchemaTest.testSchema order-independent#24805
lhotari merged 1 commit intoapache:masterfrom
LucasEby:testSchemaNondeterministic

Conversation

@LucasEby
Copy link
Contributor

@LucasEby LucasEby commented Oct 1, 2025

Fixes #24804

Motivation

In ProtobufNativeSchemaTest.testSchema, the json's contents and the FileDescriptorSet field's contents do not have a deterministic order but the hardcoded string assertion assumes a deterministic order. The json serializer did not guarantee attribute order and inside FileDescriptorSet the contents can also be in different orders due to different generation paths or environments producing the contents in different orders despite the logical content being the same. Since the original test compared the raw strings/trees "as-is", harmless re-ordering could flip the test from pass to fail without any real schema change.

Modifications

We no longer compare raw strings/trees "as-is". Instead the fileDescriptorSet field is base64-decoded into a FileDescriptorSet, then converted into a JSONObject. We compare the expected and actual descriptors with JsonAssert in NON_EXTENSIBLE mode, which ignores field ordering but forbids missing or extra fields.

Next, we remove FileDescriptorSet from the outer JSON and compare the remainder in the NON_EXTENSIBLE mode again. We implemented this two stage approach because JSON assertion does not re-order the inner contents of the FileDescriptorSet (compares directly as strings instead) even if though it's decoded contents can be re-ordered.

By decoding and normalizing fileDescriptorSet first, we ensure the test remains robust to nondeterministic ordering inside the descriptor set, while still validating the schema structure strictly. This change keeps the spirit of the original test while eliminating failures caused solely by allowed reordering.

NOTE: Some additional assertTrue and assertFalse assertions were also added to ensure the test remains robust in the event that the hardcoded string is ever modified for some reason. This will prevent someone from modifying the string and then forgetting to modify the test afterwards, thus causing the fileDescriptorSet assertion to incorrectly pass from comparing empty strings.

Verifying this change

  • Make sure that the change passes the CI checks.

This change is already covered by existing tests, such as org.apache.pulsar.client.impl.schema.ProtobufSchemaTest#testSchema.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

PR in forked repository: LucasEby#1

@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Oct 1, 2025
@LucasEby LucasEby force-pushed the testSchemaNondeterministic branch from 3146821 to 1fbb84e Compare October 1, 2025 17:52
@LucasEby LucasEby force-pushed the testSchemaNondeterministic branch 3 times, most recently from d39852c to c84ac39 Compare October 4, 2025 00:15
@LucasEby
Copy link
Contributor Author

LucasEby commented Oct 8, 2025

@lhotari I incorporated your suggestions and made the code easier to follow. All checks passed on my fork: LucasEby#1
The code should be good to merge

@LucasEby LucasEby force-pushed the testSchemaNondeterministic branch from c84ac39 to 3473283 Compare October 17, 2025 18:00
@LucasEby LucasEby force-pushed the testSchemaNondeterministic branch from 3473283 to 6f31a65 Compare October 17, 2025 22:22
Copy link
Member

@lhotari lhotari left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lhotari
Copy link
Member

lhotari commented Nov 11, 2025

Closing and reopening to trigger the CI to run with latest master branch since updating PR branch wasn't allowed (please create PRs in a way where it's allowed, this is the default).

@lhotari lhotari closed this Nov 11, 2025
@lhotari lhotari reopened this Nov 11, 2025
@codecov-commenter
Copy link

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.33%. Comparing base (cc0eef9) to head (6f31a65).
⚠️ Report is 93 commits behind head on master.

Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff              @@
##             master   #24805      +/-   ##
============================================
- Coverage     74.34%   74.33%   -0.01%     
+ Complexity    33842    33667     -175     
============================================
  Files          1912     1920       +8     
  Lines        149072   150153    +1081     
  Branches      17299    17418     +119     
============================================
+ Hits         110833   111622     +789     
- Misses        29440    29646     +206     
- Partials       8799     8885      +86     
Flag Coverage Δ
inttests 26.18% <ø> (-0.06%) ⬇️
systests 22.82% <ø> (-0.01%) ⬇️
unittests 73.88% <ø> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 182 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@lhotari lhotari merged commit b8d0f14 into apache:master Nov 11, 2025
116 of 121 checks passed
lhotari pushed a commit that referenced this pull request Nov 11, 2025
lhotari pushed a commit that referenced this pull request Nov 11, 2025
manas-ctds pushed a commit to datastax/pulsar that referenced this pull request Nov 13, 2025
srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Nov 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] ProtobufNativeSchemaTest.testSchema order-independent

3 participants