-
Notifications
You must be signed in to change notification settings - Fork 3k
Spark 4.1: Fix Partition info not displayed in job description for RewritePositionDeleteFilesSparkAction #15175
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This PR fixes an issue where partition information was not properly displayed in job descriptions for the RewritePositionDeleteFilesSparkAction in Spark 4.1. The fix adds a description field to StructProjection to carry partition information through the projection process.
Changes:
- Modified
StructProjectionto store and return a description string - Updated
PartitionUtil.coercePartition()to pass partition description when wrappingPartitionData - Modified
RewritePositionDeleteFilesSparkActionto extract and use the partition description in job descriptions
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| api/src/main/java/org/apache/iceberg/util/StructProjection.java | Added description field and getter method, plus overloaded wrap method to accept description |
| core/src/main/java/org/apache/iceberg/util/PartitionUtil.java | Updated to pass partition description when wrapping PartitionData instances |
| spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/RewritePositionDeleteFilesSparkAction.java | Added helper method to extract partition description for job descriptions |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
...rk/src/main/java/org/apache/iceberg/spark/actions/RewritePositionDeleteFilesSparkAction.java
Outdated
Show resolved
Hide resolved
be1b165 to
5beb72d
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
...rk/src/main/java/org/apache/iceberg/spark/actions/RewritePositionDeleteFilesSparkAction.java
Show resolved
Hide resolved
api/src/main/java/org/apache/iceberg/util/StructProjection.java
Outdated
Show resolved
Hide resolved
5beb72d to
e72feb1
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| this.description = description; | ||
| return this; | ||
| } | ||
|
|
Copilot
AI
Jan 29, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The public method getDescription() lacks documentation explaining what the description represents, when it may be null, and its intended use case. Consider adding a Javadoc comment.
| /** | |
| * Returns the optional description associated with the currently wrapped {@link StructLike}. | |
| * | |
| * <p>The description is a human-readable identifier or context string supplied when calling | |
| * {@link #wrap(StructLike, String)}. It may be {@code null}, for example when | |
| * {@link #wrap(StructLike)} is used or when {@code null} is explicitly passed as the | |
| * description. | |
| * | |
| * <p>The description, if present, is propagated to projections created via | |
| * {@link #copyFor(StructLike)}. | |
| * | |
| * @return the description for the wrapped struct, or {@code null} if no description was set | |
| */ |
| public StructProjection wrap(StructLike newStruct) { | ||
| return wrap(newStruct, null); | ||
| } | ||
|
|
Copilot
AI
Jan 29, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The overloaded wrap method with description parameter lacks documentation explaining the purpose of the description parameter and when it should be provided versus using the no-argument version.
| /** | |
| * Wraps a new {@link StructLike} instance in this projection. | |
| * | |
| * <p>The {@code description} is an optional, human-readable label for the wrapped struct, | |
| * which may be used in diagnostics such as error messages or logging. Callers that do not | |
| * need to associate a description with the struct can use {@link #wrap(StructLike)} instead, | |
| * which is equivalent to passing {@code null} for this parameter. | |
| * | |
| * @param newStruct the struct to wrap with this projection, or {@code null} | |
| * @param description an optional description of the wrapped struct, or {@code null} | |
| * @return this projection, wrapping the given struct | |
| */ |
e72feb1 to
f85cd05
Compare
…writePositionDeleteFilesSparkAction
f85cd05 to
85a351f
Compare
Closes #12414
After Fix:
