-
Notifications
You must be signed in to change notification settings - Fork 1
Open
Description
Summary
Add configurable retention policy to automatically delete old query results and reclaim storage space.
Background
Query results are persisted as parquet files in S3 under cache/_runtimedb_internal/runtimedb_results/. These files accumulate indefinitely with no cleanup mechanism, causing storage to grow unbounded
Storage Structure
S3 Bucket
└── cache/_runtimedb_internal/runtimedb_results/
└── rslt{id}/
└── data.parquet/
└── part.1 (actual query output as parquet)
Requirements
-
Configuration
RUNTIMEDB_RESULT_RETENTION_DAYS=7 # Default: 7 days, 0 = disabled -
Cleanup Logic
- Delete results older than retention period
- Remove from both:
- Catalog DB: DELETE FROM results WHERE created_at < NOW() - INTERVAL 'X days'
- S3: Delete corresponding runtimedb_results/{result_id}/ folders
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels