From 1fced6de098a354109f96f200b5c957fbeb7aa9f Mon Sep 17 00:00:00 2001 From: Lilian Lee Date: Thu, 20 Feb 2025 17:18:39 +0800 Subject: [PATCH 1/9] Add temp.md --- temp.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 temp.md diff --git a/temp.md b/temp.md new file mode 100644 index 0000000000000..af27ff4986a7b --- /dev/null +++ b/temp.md @@ -0,0 +1 @@ +This is a test file. \ No newline at end of file From ac1d245dc704f2fe1b9a7e49ac7498f9621835e1 Mon Sep 17 00:00:00 2001 From: Lilian Lee Date: Thu, 20 Feb 2025 17:18:43 +0800 Subject: [PATCH 2/9] Delete temp.md --- temp.md | 1 - 1 file changed, 1 deletion(-) delete mode 100644 temp.md diff --git a/temp.md b/temp.md deleted file mode 100644 index af27ff4986a7b..0000000000000 --- a/temp.md +++ /dev/null @@ -1 +0,0 @@ -This is a test file. \ No newline at end of file From f429bb611f7f1f910b5cb4622f2c60f69548f7d2 Mon Sep 17 00:00:00 2001 From: lilin90 Date: Tue, 11 Mar 2025 15:30:36 +0800 Subject: [PATCH 3/9] toc, br: add compact log backup --- TOC.md | 1 + br/br-compact-log-backup.md | 92 +++++++++++++++++++++++++++++++++++++ 2 files changed, 93 insertions(+) create mode 100644 br/br-compact-log-backup.md diff --git a/TOC.md b/TOC.md index 73b8e996af9f6..88efe9f495827 100644 --- a/TOC.md +++ b/TOC.md @@ -246,6 +246,7 @@ - [Use Overview](/br/br-use-overview.md) - [Snapshot Backup and Restore Guide](/br/br-snapshot-guide.md) - [Log Backup and PITR Guide](/br/br-pitr-guide.md) + - [Compact Log Backup](/br/br-compact-log-backup.md) - [Use Cases](/br/backup-and-restore-use-cases.md) - [Backup Storages](/br/backup-and-restore-storages.md) - BR CLI Manuals diff --git a/br/br-compact-log-backup.md b/br/br-compact-log-backup.md new file mode 100644 index 0000000000000..39a6119ef7dca --- /dev/null +++ b/br/br-compact-log-backup.md @@ -0,0 +1,92 @@ +--- +title: Compact Log Backup +summary: Learn how to improve Point-in-time Recovery (PITR) efficiency by compacting log backups into the SST format. +--- + +# Compact Log Backup + +This document describes how to improve the efficiency of point-in-time recovery (PITR) by compacting log backups into the [SST](/glossary.md#static-sorted-table--sorted-string-table-sst) format. + +## Overview + +Traditional log backups store write operations in a highly unstructured manner, which can lead to the following issues: + +- **Reduced recovery performance**: disordered data needs to be written to the cluster one by one through the Raft protocol. +- **Write amplification**: repeated write operations increase storage pressure. +- **Dependency on full backups**: frequent full backups are required to control the amount of recovery data, which can impact application operations. + +Starting from v9.0.0, the compact log backup feature provides offline reorganization capabilities, converting unstructured log backup data into structured SST files. This results in the following improvements: + +- **Enhanced recovery performance**: structured data supports efficient batch writes. +- **Optimized storage space**: reduces redundant data storage. +- **Extended full backup intervals**: minimizes the impact on application operations. + +## Limitations + +- Compact log backup is not a replacement for full backups. It must be used in conjunction with regular full backups. To ensure PITR capability, the compacting process retains MVCC data. Prolonged intervals without full backups can lead to excessive storage usage and recovery issues. +- Currently, compacting backups with local encryption enabled is not supported. + +## Use compact log backup + +Currently, only manual compaction of log backups is supported, and the process is complex. **It is recommended to use the coming TiDB Operator solution for compacting log backups in production environments.** + +### Manual compaction + +This section describes the steps for manually compacting log backups. + +#### Prerequisites + +Manual compaction of log backups requires two tools: `tikv-ctl` and `br`. + +#### Step 1: Encode storage to Base64 + +Execute the following encoding command: + +```shell +br operator base64ify --storage "s3://your/log/backup/storage/here" --load-creds +``` + +> **Note:** +> +> - If the `--load-creds` option is included when you execute the preceding command, the encoded Base64 string contains credential information loaded from the current BR environment. Note to ensure proper security and access control. +> - The `--storage` value should match the storage output from the `log status` command of the log backup task. + +Example command output: + +```text +Credentials are encoded to the base64 string. DON'T share this with untrusted people! +Gl8KEWh0dHA6Ly9taW5pbzo5MDAwEgl1cy1lYXN0LTEaBWFzdHJvIh50cGNjLTEwMDAtaW5jci13aXRoLWJvdW5kYXJpZXNCCm1pbmlvYWRtaW5KCm1pbmlvYWRtaW5QAQ== +``` + +#### Step 2: Execute log compaction + +With the Base64-encoded storage, you can initiate the compaction using `tikv-ctl`. Note that the default log level of `tikv-ctl` is `warning`. Use `--log-level info` to obtain more detailed information: + +```shell +tikv-ctl --log-level info compact-log-backup \ + --from "" --until "" \ + -s 'bAsE64==' -N 8 +``` + +Parameter descriptions: + +- `-s`: the Base64-encoded storage string obtained earlier. +- `-N`: the maximum number of concurrent log compaction tasks. +- `--from`: the start timestamp for compaction. +- `--until`: the end timestamp for compaction. + +The `--from` and `--until` parameters define the time range for the compaction operation. The compaction operation handles all log files containing write operations within the specified time range, so the generated SST files might include data outside this range. + +To obtain the timestamp for a specific point in time, execute the following command: + +```shell +echo $(( $(date --date '2004-05-06 15:02:01Z' +%s%3N) << 18 )) +``` + +> **Note:** +> +> If you are a macOS user, you need to install `coreutils` via Homebrew and use `gdate` instead of `date`. +> +> ```shell +> echo $(( $(gdate --date '2004-05-06 15:02:01Z' +%s%3N) << 18 )) +> ``` From 0cec46dd84162449bdca7a87bd1a9d9eddccfdb7 Mon Sep 17 00:00:00 2001 From: lilin90 Date: Tue, 11 Mar 2025 15:34:02 +0800 Subject: [PATCH 4/9] Update wording --- br/br-compact-log-backup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/br/br-compact-log-backup.md b/br/br-compact-log-backup.md index 39a6119ef7dca..d1dccdad4ee4b 100644 --- a/br/br-compact-log-backup.md +++ b/br/br-compact-log-backup.md @@ -49,7 +49,7 @@ br operator base64ify --storage "s3://your/log/backup/storage/here" --load-creds > **Note:** > > - If the `--load-creds` option is included when you execute the preceding command, the encoded Base64 string contains credential information loaded from the current BR environment. Note to ensure proper security and access control. -> - The `--storage` value should match the storage output from the `log status` command of the log backup task. +> - The `--storage` value matches the storage output from the `log status` command of the log backup task. Example command output: From a259ba0b93cced0619f9d4a8be9a7bc7174aa50d Mon Sep 17 00:00:00 2001 From: lilin90 Date: Tue, 11 Mar 2025 16:37:07 +0800 Subject: [PATCH 5/9] Remove example output --- br/br-compact-log-backup.md | 7 ------- 1 file changed, 7 deletions(-) diff --git a/br/br-compact-log-backup.md b/br/br-compact-log-backup.md index d1dccdad4ee4b..5e62b92438245 100644 --- a/br/br-compact-log-backup.md +++ b/br/br-compact-log-backup.md @@ -51,13 +51,6 @@ br operator base64ify --storage "s3://your/log/backup/storage/here" --load-creds > - If the `--load-creds` option is included when you execute the preceding command, the encoded Base64 string contains credential information loaded from the current BR environment. Note to ensure proper security and access control. > - The `--storage` value matches the storage output from the `log status` command of the log backup task. -Example command output: - -```text -Credentials are encoded to the base64 string. DON'T share this with untrusted people! -Gl8KEWh0dHA6Ly9taW5pbzo5MDAwEgl1cy1lYXN0LTEaBWFzdHJvIh50cGNjLTEwMDAtaW5jci13aXRoLWJvdW5kYXJpZXNCCm1pbmlvYWRtaW5KCm1pbmlvYWRtaW5QAQ== -``` - #### Step 2: Execute log compaction With the Base64-encoded storage, you can initiate the compaction using `tikv-ctl`. Note that the default log level of `tikv-ctl` is `warning`. Use `--log-level info` to obtain more detailed information: From 5c05df811b33d582ad1d5f69864ac00c137dde45 Mon Sep 17 00:00:00 2001 From: Lilian Lee Date: Tue, 11 Mar 2025 18:10:32 +0800 Subject: [PATCH 6/9] Apply suggestions from code review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: 山岚 <36239017+YuJuncen@users.noreply.github.com> --- br/br-compact-log-backup.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/br/br-compact-log-backup.md b/br/br-compact-log-backup.md index 5e62b92438245..cb09e14bd3581 100644 --- a/br/br-compact-log-backup.md +++ b/br/br-compact-log-backup.md @@ -11,19 +11,19 @@ This document describes how to improve the efficiency of point-in-time recovery Traditional log backups store write operations in a highly unstructured manner, which can lead to the following issues: -- **Reduced recovery performance**: disordered data needs to be written to the cluster one by one through the Raft protocol. -- **Write amplification**: repeated write operations increase storage pressure. +- **Reduced recovery performance**: unordered data has to be written to the cluster one by one through the Raft protocol. +- **Write amplification**: all writes must be compacted from L0 to the bottommost level by level. - **Dependency on full backups**: frequent full backups are required to control the amount of recovery data, which can impact application operations. -Starting from v9.0.0, the compact log backup feature provides offline reorganization capabilities, converting unstructured log backup data into structured SST files. This results in the following improvements: +Starting from v9.0.0, the compact log backup feature provides offline compaction capabilities, converting unstructured log backup data into structured SST files. This results in the following improvements: -- **Enhanced recovery performance**: structured data supports efficient batch writes. -- **Optimized storage space**: reduces redundant data storage. -- **Extended full backup intervals**: minimizes the impact on application operations. +- SST files can be quickly imported into the cluster, **improving recovery performance**. +- Redundant data is removed during compaction, **reducing storage space consumption**. +- You can set longer full backup intervals while ensuring the Recovery Time Objective (RTO), **reducing the impact on applications**. ## Limitations -- Compact log backup is not a replacement for full backups. It must be used in conjunction with regular full backups. To ensure PITR capability, the compacting process retains MVCC data. Prolonged intervals without full backups can lead to excessive storage usage and recovery issues. +- Compact log backup is not a replacement for full backups. It must be used in conjunction with periodical full backups. To ensure PITR capability, the compacting process retains all MVCC versions. Failing to perform full backups for a long time can lead to excessive storage usage and might cause issues when restoring data later. - Currently, compacting backups with local encryption enabled is not supported. ## Use compact log backup From 42189e926d1c855e7505c206b7bad3f035fbd69d Mon Sep 17 00:00:00 2001 From: Lilian Lee Date: Wed, 12 Mar 2025 15:13:09 +0800 Subject: [PATCH 7/9] Apply suggestions from code review Co-authored-by: xixirangrang --- br/br-compact-log-backup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/br/br-compact-log-backup.md b/br/br-compact-log-backup.md index cb09e14bd3581..159e4dfd5502f 100644 --- a/br/br-compact-log-backup.md +++ b/br/br-compact-log-backup.md @@ -5,7 +5,7 @@ summary: Learn how to improve Point-in-time Recovery (PITR) efficiency by compac # Compact Log Backup -This document describes how to improve the efficiency of point-in-time recovery (PITR) by compacting log backups into the [SST](/glossary.md#static-sorted-table--sorted-string-table-sst) format. +This document describes how to improve the efficiency of point-in-time recovery ([PITR](/glossary.md#point-in-time-recovery-pitr)) by compacting log backups into the [SST](/glossary.md#static-sorted-table--sorted-string-table-sst) format. ## Overview From 56d8c95365dd84f705698c07767fdf927c93b6d0 Mon Sep 17 00:00:00 2001 From: Lilian Lee Date: Tue, 23 Dec 2025 15:41:01 +0800 Subject: [PATCH 8/9] Apply suggestions from code review Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --- br/br-compact-log-backup.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/br/br-compact-log-backup.md b/br/br-compact-log-backup.md index 159e4dfd5502f..4d0b4de7e783b 100644 --- a/br/br-compact-log-backup.md +++ b/br/br-compact-log-backup.md @@ -23,12 +23,12 @@ Starting from v9.0.0, the compact log backup feature provides offline compaction ## Limitations -- Compact log backup is not a replacement for full backups. It must be used in conjunction with periodical full backups. To ensure PITR capability, the compacting process retains all MVCC versions. Failing to perform full backups for a long time can lead to excessive storage usage and might cause issues when restoring data later. +- Compact log backup is not a replacement for full backups. It must be used in conjunction with periodic full backups. To ensure PITR capability, the compacting process retains all MVCC versions. Failing to perform full backups for a long time can lead to excessive storage usage and might cause issues when restoring data later. - Currently, compacting backups with local encryption enabled is not supported. ## Use compact log backup -Currently, only manual compaction of log backups is supported, and the process is complex. **It is recommended to use the coming TiDB Operator solution for compacting log backups in production environments.** +Currently, only manual compaction of log backups is supported, and the process is complex. **It is recommended to use the upcoming TiDB Operator solution for compacting log backups in production environments.** ### Manual compaction From 9446d14ab1b5176ac86d830afba3ccf20ab9bff1 Mon Sep 17 00:00:00 2001 From: Lilian Lee Date: Tue, 23 Dec 2025 15:41:35 +0800 Subject: [PATCH 9/9] Apply suggestions from code review --- br/br-compact-log-backup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/br/br-compact-log-backup.md b/br/br-compact-log-backup.md index 4d0b4de7e783b..9c969f1148e52 100644 --- a/br/br-compact-log-backup.md +++ b/br/br-compact-log-backup.md @@ -15,7 +15,7 @@ Traditional log backups store write operations in a highly unstructured manner, - **Write amplification**: all writes must be compacted from L0 to the bottommost level by level. - **Dependency on full backups**: frequent full backups are required to control the amount of recovery data, which can impact application operations. -Starting from v9.0.0, the compact log backup feature provides offline compaction capabilities, converting unstructured log backup data into structured SST files. This results in the following improvements: +Starting from v8.5.5, the compact log backup feature provides offline compaction capabilities, converting unstructured log backup data into structured SST files. This results in the following improvements: - SST files can be quickly imported into the cluster, **improving recovery performance**. - Redundant data is removed during compaction, **reducing storage space consumption**.