Fix compaction error when upgrading from RocksDB v5.10.3 to v8.11.4 #14240
+76
−3
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When I upgraded RocksDB from version v5.10.1 to v8.11.4, a compaction error occurred:
2025/12/19-08:41:47.105378 139671781758720 [WARN] [db/compaction/compaction_job.cc:848] [default] [JOB 6] Compaction Total number of input records: 44912, but processed 44911 records.The root cause of this issue is that the SST metadata in RocksDB v5.10.3 does not include the
num_range_deletionfield, whereas this field was added in v8.11.4 with a default value of 0.After the upgrade, when compaction reads the metadata of legacy SST files, it fails to retrieve the
num_range_deletionsvalue and thus assigns it a default of 0. However, these legacy SST files actually contain DeleteRange operations.Upon completion of compaction, a validation check is performed:
// rocksdb-8.11.4/db/compaction/compaction_job.cc uint64_t expected = compaction_stats_.stats.num_input_records - num_input_range_del; uint64_t actual = compaction_job_stats_->num_input_records; if (expected != actual) { xxx // errorSince
num_range_deletionis assigned a value of 0, this validation check fails.Reproduction code as follows:
Generate data directory with v5.10.3:
int main() { Options options; options.create_if_missing = true; DB* db; Status s = DB::Open(options, "../db", &db); assert(s.ok()); auto generate_sst = [&]() { for (int i = 0; i < 100; ++i) { string key = "key" + to_string(i); string value = "value" + to_string(i); s = db->Put(WriteOptions(), key, value); assert(s.ok()); } s = db->DeleteRange(WriteOptions(), db->DefaultColumnFamily(), "key3", "key8"); assert(s.ok()); s = db->Flush(FlushOptions()); assert(s.ok()); }; // generate 2 sst files generate_sst(); generate_sst(); delete db; return 0; }Upgrade and verify with v8.11.4:
int main() { DB* db; Options options; options.create_if_missing = true; Status status = DB::Open(options, "../db/", &db); if (!status.ok()) { std::cout << "open db failed: " << status.ToString() << std::endl; return -1; } status = db->CompactRange(CompactRangeOptions(), nullptr, nullptr); if (!status.ok()) { std::cout << "compact db failed: " << status.ToString() << std::endl; return -1; } delete db; return 0; }// will print
compact db failed: Corruption: Compaction number of input keys does not match number of keys processed. Expected 92 but processed 90. Compaction summary: Base version 2 Base level 0, inputs: [10(1881B) 7(1881B)]The fix approach is as follows: when
num_range_deletionsis 0, check whether there arefragmented_range_dels. Iffragmented_range_delsexist, throw an error and skip the validation check following the original process.