refactor: consolidate snapshot expiration into MaintenanceTable by ForeverAngry · Pull Request #2143 · apache/iceberg-python

ForeverAngry · 2025-06-23T23:04:17Z

Rationale for this change

Consolidates snapshot expiration functionality from the standalone ExpireSnapshots class into the MaintenanceTable class for a unified maintenance API.
Resolves planned work left over from Added ExpireSnapshots Feature #1880, and closes refactor: consolidate snapshot expiration into MaintenanceTable #2142
Achieves feature and API parity with the Java implementation for snapshot retention and table maintenance.

Features & Enhancements

Introduces table.maintenance.expire_snapshots() as the unified entry point for snapshot expiration and future maintenance operations.
Retains the existing ExpireSnapshots implementation internally. The expire_snapshots() method on MaintenanceTable now returns an ExpireSnapshots object, preserving transaction semantics and supporting context manager usage:
```
with table.maintenance.expire_snapshots() as expire_snapshots:
    expire_snapshots.by_id(1)
    expire_snapshots.by_id(2)
```
Focuses this PR on refactoring and documentation improvements, while maintaining compatibility with the prior ExpireSnapshots interface.
Sets a foundation for future expansion of the MaintenanceTable abstraction to encapsulate additional maintenance operations.

Bug Fixes & Cleanups

ManageSnapshots Cleanup (#2151)
- Removes an unrelated instance variable from the ManageSnapshots class, aligning with the Java reference implementation.

Testing & Documentation

Testing:
- Tested the new API interface including:
  - Expiration by ID
  - Protection of branch/tag snapshots
Documentation:
- Added and updated documentation to describe:
  - API usage examples

Preview:

Are these changes tested?

Yes. All changes are tested.~~, with this PR predicated on the final changes from #1200.~~ This work builds on the framework introduced by @jayceslesar in #1200 for the MaintenanceTable.

Are there any user-facing changes?

Closes:

…p data

…h a new Expired Snapshot class. updated tests.

ValueError: Cannot expire snapshot IDs {3051729675574597004} as they are currently referenced by table refs.

Moved expiration-related methods from `ExpireSnapshots` to `ManageSnapshots` for improved organization and clarity. Updated corresponding pytest tests to reflect these changes.

Re-ran the `poetry run pre-commit run --all-files` command on the project.

Moved: the functions for expiring snapshots to their own class.

…ng it in a separate issue. Fixed: unrelated changes caused by afork/branch sync issues.

Co-authored-by: Fokko Driesprong <fokko@apache.org>

Implemented logic to protect the HEAD branches or Tagged branches from being expired by the `expire_snapshot_by_id` method.

ForeverAngry · 2025-06-23T23:07:52Z

@Fokko @jayceslesar let me know if you guys prefer i stack this pr into the #1200 or if you both would rather i wait until the #1200 is merged into main, and then rebase on the updated upstream/main, and then create the PR against apache/iceberg-python:main!

Fokko · 2025-06-24T08:07:51Z

Great seeing this PR @ForeverAngry, thanks again for working on this! I'm okay with first merging #1200, but we could also merge this first, and adapt the remove orphan files routine to use .maintenance. Let me follow up on the remove orphan files, because there are some open questions there.

ForeverAngry · 2025-07-03T00:32:35Z

@Fokko did you decide if you wanted me to stay stacked on the delete orphans pr, or go ahead and prepare the pr for this, to the main branch?

keep the table.maintenance.expire_snapshots() API signature Return the existing ExpireSnapshots class that extends UpdateTableMetadata Enable transaction semantics with context manager support Focus this PR on API refactoring, move complex retention logic to separate PR

ForeverAngry · 2025-08-10T11:24:19Z

@Fokko @kevinjqliu let me know if this commit looks like what you both were expecting.

…at original suggestions

kevinjqliu

thanks @ForeverAngry i think this pr is very close. we need to get it to a good shape for merge and then we can release 0.10 :)

mkdocs/docs/api.md

pyiceberg/table/inspect.py

pyiceberg/table/update/snapshot.py

tests/table/test_maintenance_api.py

…ected and unprotected snapshots

…mp, with updated docstrings

… clarity

mkdocs/docs/api.md

ForeverAngry · 2025-08-11T14:30:40Z

@Fokko can you kick off the workflows for me?

kevinjqliu

lgtm, lets revert the irrelevant docs and this thing is ready to merge!!

…r clarity

kevinjqliu · 2025-08-11T20:34:20Z

caught up with @ForeverAngry on slack, helped fix the merge issue with the documentation

kevinjqliu

LGTM!! Thanks a bunch of the persistence to get this PR into a good state :)

Fokko · 2025-08-12T17:44:12Z

mkdocs/docs/api.md

+
+# Method chaining
+table.maintenance.expire_snapshots().by_id(12345).commit()


This is the same example as above?

Suggested change

# Method chaining

table.maintenance.expire_snapshots().by_id(12345).commit()

Fokko · 2025-08-12T17:45:10Z

mkdocs/docs/api.md

+        for snapshot_id in snapshot_ids:
+            expire.by_id(snapshot_id)


Why not use the by_ids? This makes the example a bit more consise:

Suggested change

for snapshot_id in snapshot_ids:

expire.by_id(snapshot_id)

expire.by_ids(snapshot_ids)

Fokko

This looks good @ForeverAngry

Thanks for your work and all the patience 👍

@jayceslesar

…he#2143)   # Rationale for this change - Consolidates snapshot expiration functionality from the standalone `ExpireSnapshots` class into the `MaintenanceTable` class for a unified maintenance API. - Resolves planned work left over from apache#1880, and closes apache#2142 - Achieves feature and API parity with the Java implementation for snapshot retention and table maintenance. # Features & Enhancements - Introduces `table.maintenance.expire_snapshots()` as the unified entry point for snapshot expiration and future maintenance operations. - Retains the existing `ExpireSnapshots` implementation internally. The `expire_snapshots()` method on `MaintenanceTable` now returns an `ExpireSnapshots` object, preserving transaction semantics and supporting context manager usage: ```python with table.maintenance.expire_snapshots() as expire_snapshots: expire_snapshots.by_id(1) expire_snapshots.by_id(2) ``` - Focuses this PR on refactoring and documentation improvements, while maintaining compatibility with the prior `ExpireSnapshots` interface. - Sets a foundation for future expansion of the `MaintenanceTable` abstraction to encapsulate additional maintenance operations. # Bug Fixes & Cleanups - **ManageSnapshots Cleanup ([apache#2151](apache#2151 - Removes an unrelated instance variable from the `ManageSnapshots` class, aligning with the Java reference implementation. # Testing & Documentation - **Testing:** - Tested the new API interface including: - Expiration by ID - Protection of branch/tag snapshots - **Documentation:** - Added and updated documentation to describe: - API usage examples Preview: <img width="1686" height="1015" alt="Screenshot 2025-08-11 at 1 37 04 PM" src="https://github.com/user-attachments/assets/f469f3fc-b4b1-4ec9-b1ca-b9185e22643e" /> # Are these changes tested? Yes. All changes are tested.~, with this PR predicated on the final changes from apache#1200.~ This work builds on the framework introduced by @jayceslesar in apache#1200 for the `MaintenanceTable`. # Are there any user-facing changes? --- **Closes:** - Closes apache#2151 - Closes apache#2142 --------- Co-authored-by: Fokko Driesprong <fokko@apache.org> Co-authored-by: Kevin Liu <kevin.jq.liu@gmail.com>

ForeverAngry and others added 25 commits March 28, 2025 20:23

Added initial units tests and Class for Removing a Snapshot

0a94d96

Added methods needed to expire snapshots by id, and optionally cleanu…

5f0b62b

…p data

Update test_expire_snapshots.py

f995daa

Added the builder method to __init__.py, updated the snapshot api wit…

65365e1

…h a new Expired Snapshot class. updated tests.

Snapshots are not being transacted on, but need to re-assign refs

e28815f

ValueError: Cannot expire snapshot IDs {3051729675574597004} as they are currently referenced by table refs.

Fixed the test case.

4628ede

adding print statements to help with debugging

e80c41c

Draft ready

cb9f0c9

Applied suggestions to Fix CICD

ebcff2b

Merge branch 'main' into main

97399bf

Rebuild the poetry lock file.

95e5af2

Merge branch 'main' into main

5ab5890

Refactor implementation of ExpireSnapshots

5acd690

Moved expiration-related methods from `ExpireSnapshots` to `ManageSnapshots` for improved organization and clarity. Updated corresponding pytest tests to reflect these changes.

Fixed format and linting issues

d30a08c

Re-ran the `poetry run pre-commit run --all-files` command on the project.

Merge branch 'main' into main

e62ab58

Fixed format and linting issues

1af3258

Re-ran the `poetry run pre-commit run --all-files` command on the project.

Merge branch 'main' of https://github.com/ForeverAngry/iceberg-python

352b48f

Merge branch 'main' into main

382e0ea

rebased: from main

549c183

fixed: typo

386cb15

removed errant files

12729fa

Added: public method signature to the init table file.

ce3515c

Moved: the functions for expiring snapshots to their own class.

Removed: expire_snapshots_older_than method, in favor of implementi…

28fce4b

…ng it in a separate issue. Fixed: unrelated changes caused by afork/branch sync issues.

Update tests/table/test_expire_snapshots.py

2c3153e

Co-authored-by: Fokko Driesprong <fokko@apache.org>

Removed: unrelated changes, Added: logic to expire snapshot method.

27c3ece

Implemented logic to protect the HEAD branches or Tagged branches from being expired by the `expire_snapshot_by_id` method.

Fokko added this to the PyIceberg 0.10.0 milestone Jun 24, 2025

ForeverAngry force-pushed the refactor/consolidate-snapshot-expiration branch from a6c3b63 to 9937894 Compare July 5, 2025 01:10

ForeverAngry added 3 commits August 8, 2025 11:21

reverted api.md changes

3591c39

docs: add table maintenance section with snapshot expiration examples

c5ff202

ForeverAngry requested a review from Fokko August 10, 2025 11:22

refactor: rename expire_snapshot_by_id to by_id to align back with wh…

af78e52

…at original suggestions

kevinjqliu reviewed Aug 10, 2025

View reviewed changes

ForeverAngry added 4 commits August 10, 2025 21:08

feat: implement snapshot expiration functionality with tests for prot…

1c2f631

…ected and unprotected snapshots

feat: add methods to expire snapshots by IDs and older than a timesta…

64ba8f0

…mp, with updated docstrings

refactor: rename snapshot expiration methods for clarity and consistency

bf9427a

refactor: update snapshot expiration method calls for consistency and…

1b3a95c

… clarity

Fokko reviewed Aug 11, 2025

View reviewed changes

mkdocs/docs/api.md Show resolved Hide resolved

ForeverAngry added 2 commits August 11, 2025 10:17

fixed: unrelated changes

cab890f

fixed: unrelated changes

4df0e83

kevinjqliu reviewed Aug 11, 2025

View reviewed changes

ForeverAngry and others added 5 commits August 11, 2025 11:58

fix: update error messages for protected snapshot expiration tests fo…

44da743

…r clarity

refactor: remove outdated snapshot expiration documentation for clarity

36f89e6

linted update

d6ec64d

resolve merge conflict

3ba85f0

extra?

c980a16

kevinjqliu approved these changes Aug 11, 2025

View reviewed changes

kevinjqliu requested review from Fokko and jayceslesar August 11, 2025 20:54

Fokko reviewed Aug 12, 2025

View reviewed changes

Fokko approved these changes Aug 12, 2025

View reviewed changes

Fokko merged commit 4b961f7 into apache:main Aug 12, 2025
11 of 19 checks passed


		# Method chaining
		table.maintenance.expire_snapshots().by_id(12345).commit()

	for snapshot_id in snapshot_ids:
	expire.by_id(snapshot_id)
	expire.by_ids(snapshot_ids)

Conversation

ForeverAngry commented Jun 23, 2025 • edited by Fokko Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

Features & Enhancements

Bug Fixes & Cleanups

Testing & Documentation

Are these changes tested?

Are there any user-facing changes?

Uh oh!

ForeverAngry commented Jun 23, 2025

Uh oh!

Fokko commented Jun 24, 2025

Uh oh!

ForeverAngry commented Jul 3, 2025

Uh oh!

ForeverAngry commented Aug 10, 2025

Uh oh!

kevinjqliu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ForeverAngry commented Aug 11, 2025

Uh oh!

kevinjqliu left a comment

Choose a reason for hiding this comment

Uh oh!

kevinjqliu commented Aug 11, 2025

Uh oh!

kevinjqliu left a comment

Choose a reason for hiding this comment

Uh oh!

Fokko Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

Fokko Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

Fokko left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ForeverAngry commented Jun 23, 2025 •

edited by Fokko

Loading