From 5fce533f93aabce16fc5eafbae25a696b5eb5ee0 Mon Sep 17 00:00:00 2001 From: Lilian Lee Date: Thu, 20 Feb 2025 17:18:39 +0800 Subject: [PATCH 1/7] Add temp.md --- temp.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 temp.md diff --git a/temp.md b/temp.md new file mode 100644 index 0000000000000..af27ff4986a7b --- /dev/null +++ b/temp.md @@ -0,0 +1 @@ +This is a test file. \ No newline at end of file From fd2c809b26fa0c210b75e8eb8ec819dd2b45fcb0 Mon Sep 17 00:00:00 2001 From: Lilian Lee Date: Thu, 20 Feb 2025 17:18:43 +0800 Subject: [PATCH 2/7] Delete temp.md --- temp.md | 1 - 1 file changed, 1 deletion(-) delete mode 100644 temp.md diff --git a/temp.md b/temp.md deleted file mode 100644 index af27ff4986a7b..0000000000000 --- a/temp.md +++ /dev/null @@ -1 +0,0 @@ -This is a test file. \ No newline at end of file From a753eaea27ab297620847f7d998053a64b495fd8 Mon Sep 17 00:00:00 2001 From: lilin90 Date: Tue, 11 Mar 2025 15:30:36 +0800 Subject: [PATCH 3/7] toc, br: add compact log backup --- TOC.md | 1 + br/br-compact-log-backup.md | 92 +++++++++++++++++++++++++++++++++++++ 2 files changed, 93 insertions(+) create mode 100644 br/br-compact-log-backup.md diff --git a/TOC.md b/TOC.md index c1eeb4f4a4fde..38b2c0793063c 100644 --- a/TOC.md +++ b/TOC.md @@ -242,6 +242,7 @@ - [Use Overview](/br/br-use-overview.md) - [Snapshot Backup and Restore Guide](/br/br-snapshot-guide.md) - [Log Backup and PITR Guide](/br/br-pitr-guide.md) + - [Compact Log Backup](/br/br-compact-log-backup.md) - [Use Cases](/br/backup-and-restore-use-cases.md) - [Backup Storages](/br/backup-and-restore-storages.md) - BR CLI Manuals diff --git a/br/br-compact-log-backup.md b/br/br-compact-log-backup.md new file mode 100644 index 0000000000000..39a6119ef7dca --- /dev/null +++ b/br/br-compact-log-backup.md @@ -0,0 +1,92 @@ +--- +title: Compact Log Backup +summary: Learn how to improve Point-in-time Recovery (PITR) efficiency by compacting log backups into the SST format. +--- + +# Compact Log Backup + +This document describes how to improve the efficiency of point-in-time recovery (PITR) by compacting log backups into the [SST](/glossary.md#static-sorted-table--sorted-string-table-sst) format. + +## Overview + +Traditional log backups store write operations in a highly unstructured manner, which can lead to the following issues: + +- **Reduced recovery performance**: disordered data needs to be written to the cluster one by one through the Raft protocol. +- **Write amplification**: repeated write operations increase storage pressure. +- **Dependency on full backups**: frequent full backups are required to control the amount of recovery data, which can impact application operations. + +Starting from v9.0.0, the compact log backup feature provides offline reorganization capabilities, converting unstructured log backup data into structured SST files. This results in the following improvements: + +- **Enhanced recovery performance**: structured data supports efficient batch writes. +- **Optimized storage space**: reduces redundant data storage. +- **Extended full backup intervals**: minimizes the impact on application operations. + +## Limitations + +- Compact log backup is not a replacement for full backups. It must be used in conjunction with regular full backups. To ensure PITR capability, the compacting process retains MVCC data. Prolonged intervals without full backups can lead to excessive storage usage and recovery issues. +- Currently, compacting backups with local encryption enabled is not supported. + +## Use compact log backup + +Currently, only manual compaction of log backups is supported, and the process is complex. **It is recommended to use the coming TiDB Operator solution for compacting log backups in production environments.** + +### Manual compaction + +This section describes the steps for manually compacting log backups. + +#### Prerequisites + +Manual compaction of log backups requires two tools: `tikv-ctl` and `br`. + +#### Step 1: Encode storage to Base64 + +Execute the following encoding command: + +```shell +br operator base64ify --storage "s3://your/log/backup/storage/here" --load-creds +``` + +> **Note:** +> +> - If the `--load-creds` option is included when you execute the preceding command, the encoded Base64 string contains credential information loaded from the current BR environment. Note to ensure proper security and access control. +> - The `--storage` value should match the storage output from the `log status` command of the log backup task. + +Example command output: + +```text +Credentials are encoded to the base64 string. DON'T share this with untrusted people! +Gl8KEWh0dHA6Ly9taW5pbzo5MDAwEgl1cy1lYXN0LTEaBWFzdHJvIh50cGNjLTEwMDAtaW5jci13aXRoLWJvdW5kYXJpZXNCCm1pbmlvYWRtaW5KCm1pbmlvYWRtaW5QAQ== +``` + +#### Step 2: Execute log compaction + +With the Base64-encoded storage, you can initiate the compaction using `tikv-ctl`. Note that the default log level of `tikv-ctl` is `warning`. Use `--log-level info` to obtain more detailed information: + +```shell +tikv-ctl --log-level info compact-log-backup \ + --from "" --until "" \ + -s 'bAsE64==' -N 8 +``` + +Parameter descriptions: + +- `-s`: the Base64-encoded storage string obtained earlier. +- `-N`: the maximum number of concurrent log compaction tasks. +- `--from`: the start timestamp for compaction. +- `--until`: the end timestamp for compaction. + +The `--from` and `--until` parameters define the time range for the compaction operation. The compaction operation handles all log files containing write operations within the specified time range, so the generated SST files might include data outside this range. + +To obtain the timestamp for a specific point in time, execute the following command: + +```shell +echo $(( $(date --date '2004-05-06 15:02:01Z' +%s%3N) << 18 )) +``` + +> **Note:** +> +> If you are a macOS user, you need to install `coreutils` via Homebrew and use `gdate` instead of `date`. +> +> ```shell +> echo $(( $(gdate --date '2004-05-06 15:02:01Z' +%s%3N) << 18 )) +> ``` From 2491bf13bf7dc885396e2d9910f976f94ce28fdf Mon Sep 17 00:00:00 2001 From: lilin90 Date: Tue, 11 Mar 2025 15:34:02 +0800 Subject: [PATCH 4/7] Update wording --- br/br-compact-log-backup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/br/br-compact-log-backup.md b/br/br-compact-log-backup.md index 39a6119ef7dca..d1dccdad4ee4b 100644 --- a/br/br-compact-log-backup.md +++ b/br/br-compact-log-backup.md @@ -49,7 +49,7 @@ br operator base64ify --storage "s3://your/log/backup/storage/here" --load-creds > **Note:** > > - If the `--load-creds` option is included when you execute the preceding command, the encoded Base64 string contains credential information loaded from the current BR environment. Note to ensure proper security and access control. -> - The `--storage` value should match the storage output from the `log status` command of the log backup task. +> - The `--storage` value matches the storage output from the `log status` command of the log backup task. Example command output: From 3842770fe7d96883c0e59b8cc1daa92a06a1cf0d Mon Sep 17 00:00:00 2001 From: lilin90 Date: Tue, 11 Mar 2025 16:37:07 +0800 Subject: [PATCH 5/7] Remove example output --- br/br-compact-log-backup.md | 7 ------- 1 file changed, 7 deletions(-) diff --git a/br/br-compact-log-backup.md b/br/br-compact-log-backup.md index d1dccdad4ee4b..5e62b92438245 100644 --- a/br/br-compact-log-backup.md +++ b/br/br-compact-log-backup.md @@ -51,13 +51,6 @@ br operator base64ify --storage "s3://your/log/backup/storage/here" --load-creds > - If the `--load-creds` option is included when you execute the preceding command, the encoded Base64 string contains credential information loaded from the current BR environment. Note to ensure proper security and access control. > - The `--storage` value matches the storage output from the `log status` command of the log backup task. -Example command output: - -```text -Credentials are encoded to the base64 string. DON'T share this with untrusted people! -Gl8KEWh0dHA6Ly9taW5pbzo5MDAwEgl1cy1lYXN0LTEaBWFzdHJvIh50cGNjLTEwMDAtaW5jci13aXRoLWJvdW5kYXJpZXNCCm1pbmlvYWRtaW5KCm1pbmlvYWRtaW5QAQ== -``` - #### Step 2: Execute log compaction With the Base64-encoded storage, you can initiate the compaction using `tikv-ctl`. Note that the default log level of `tikv-ctl` is `warning`. Use `--log-level info` to obtain more detailed information: From e525f8af51c329d1dae0810b3e7240e6475b6365 Mon Sep 17 00:00:00 2001 From: Lilian Lee Date: Tue, 11 Mar 2025 18:10:32 +0800 Subject: [PATCH 6/7] Apply suggestions from code review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: 山岚 <36239017+YuJuncen@users.noreply.github.com> --- br/br-compact-log-backup.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/br/br-compact-log-backup.md b/br/br-compact-log-backup.md index 5e62b92438245..cb09e14bd3581 100644 --- a/br/br-compact-log-backup.md +++ b/br/br-compact-log-backup.md @@ -11,19 +11,19 @@ This document describes how to improve the efficiency of point-in-time recovery Traditional log backups store write operations in a highly unstructured manner, which can lead to the following issues: -- **Reduced recovery performance**: disordered data needs to be written to the cluster one by one through the Raft protocol. -- **Write amplification**: repeated write operations increase storage pressure. +- **Reduced recovery performance**: unordered data has to be written to the cluster one by one through the Raft protocol. +- **Write amplification**: all writes must be compacted from L0 to the bottommost level by level. - **Dependency on full backups**: frequent full backups are required to control the amount of recovery data, which can impact application operations. -Starting from v9.0.0, the compact log backup feature provides offline reorganization capabilities, converting unstructured log backup data into structured SST files. This results in the following improvements: +Starting from v9.0.0, the compact log backup feature provides offline compaction capabilities, converting unstructured log backup data into structured SST files. This results in the following improvements: -- **Enhanced recovery performance**: structured data supports efficient batch writes. -- **Optimized storage space**: reduces redundant data storage. -- **Extended full backup intervals**: minimizes the impact on application operations. +- SST files can be quickly imported into the cluster, **improving recovery performance**. +- Redundant data is removed during compaction, **reducing storage space consumption**. +- You can set longer full backup intervals while ensuring the Recovery Time Objective (RTO), **reducing the impact on applications**. ## Limitations -- Compact log backup is not a replacement for full backups. It must be used in conjunction with regular full backups. To ensure PITR capability, the compacting process retains MVCC data. Prolonged intervals without full backups can lead to excessive storage usage and recovery issues. +- Compact log backup is not a replacement for full backups. It must be used in conjunction with periodical full backups. To ensure PITR capability, the compacting process retains all MVCC versions. Failing to perform full backups for a long time can lead to excessive storage usage and might cause issues when restoring data later. - Currently, compacting backups with local encryption enabled is not supported. ## Use compact log backup From 03efbd0c62afa49daae82d4e252d40d6885a8348 Mon Sep 17 00:00:00 2001 From: Lilian Lee Date: Wed, 12 Mar 2025 15:13:09 +0800 Subject: [PATCH 7/7] Apply suggestions from code review Co-authored-by: xixirangrang --- br/br-compact-log-backup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/br/br-compact-log-backup.md b/br/br-compact-log-backup.md index cb09e14bd3581..159e4dfd5502f 100644 --- a/br/br-compact-log-backup.md +++ b/br/br-compact-log-backup.md @@ -5,7 +5,7 @@ summary: Learn how to improve Point-in-time Recovery (PITR) efficiency by compac # Compact Log Backup -This document describes how to improve the efficiency of point-in-time recovery (PITR) by compacting log backups into the [SST](/glossary.md#static-sorted-table--sorted-string-table-sst) format. +This document describes how to improve the efficiency of point-in-time recovery ([PITR](/glossary.md#point-in-time-recovery-pitr)) by compacting log backups into the [SST](/glossary.md#static-sorted-table--sorted-string-table-sst) format. ## Overview