Implemented force-realloc-tail option for sync by Ralf1108 · Pull Request #31 · amadvance/snapraid

Ralf1108 · 2026-02-03T22:23:37Z

Follow up from #30

I implemented the suggested "force-realloc-tail" option for the sync command.
It works but it should be checked that this marking of files is always safe and screws nothing up.

Here the manual test:

Init test raid

snapraid.conf:

# dru1
data DRU1 C:\_SnapraidTest\dru1
content C:\_SnapraidTest\snapraid.content

# ppu1
parity D:\_SnapraidTest\ppu1\snapraid.parity
content D:\_SnapraidTest\ppu1\snapraid.content

exclude *.unrecoverable
exclude .content

DRU1:

Verzeichnis von C:\_SnapraidTest\dru1

03.02.2026  23:01                 1 Data1.txt
03.02.2026  23:01                 2 Data2.txt
               2 Datei(en),              3 Bytes

PPU1

empty

First sync

snapraid sync -c snapraid.conf

DRU1:

Verzeichnis von C:\_SnapraidTest\dru1

03.02.2026  23:01                 1 Data1.txt
03.02.2026  23:01                 2 Data2.txt
               2 Datei(en),              3 Bytes

PPU1:

Verzeichnis von D:\_SnapraidTest\ppu1

03.02.2026  23:04               246 snapraid.content
03.02.2026  23:04           524.288 snapraid.parity
               2 Datei(en),        524.534 Bytes

Adding big file ~ 20mb

snapraid sync -c snapraid.conf

DRU1:

Verzeichnis von C:\_SnapraidTest\dru1

29.01.2026  13:33        21.444.480 BigFile.txt
03.02.2026  23:01                 1 Data1.txt
03.02.2026  23:01                 2 Data2.txt
               3 Datei(en),     21.444.483 Bytes

PPU1:

Verzeichnis von D:\_SnapraidTest\ppu1

03.02.2026  23:06             1.602 snapraid.content
03.02.2026  23:06        22.020.096 snapraid.parity
               2 Datei(en),     22.021.698 Bytes

Adding small file to be at the parity tail

snapraid sync -c snapraid.conf

DRU1:

Verzeichnis von C:\_SnapraidTest\dru1

29.01.2026  10:57                 7 AnyFileAfterBigFile.txt
29.01.2026  13:33        21.444.480 BigFile.txt
03.02.2026  23:01                 1 Data1.txt
03.02.2026  23:01                 2 Data2.txt
               4 Datei(en),     21.444.490 Bytes

PPU1:

Verzeichnis von D:\_SnapraidTest\ppu1

03.02.2026  23:08             1.670 snapraid.content
03.02.2026  23:08        22.282.240 snapraid.parity
               2 Datei(en),     22.283.910 Bytes

Delete big file -> notice that the parity won't shrink

snapraid sync -c snapraid.conf

DRU1:

Verzeichnis von C:\_SnapraidTest\dru1

29.01.2026  10:57                 7 AnyFileAfterBigFile.txt
03.02.2026  23:01                 1 Data1.txt
03.02.2026  23:01                 2 Data2.txt
               3 Datei(en),             10 Bytes

PPU1:

Verzeichnis von D:\_SnapraidTest\ppu1

03.02.2026  23:09               317 snapraid.content
03.02.2026  23:08        22.282.240 snapraid.parity
               2 Datei(en),     22.282.557 Bytes

Checking for blocking files -> AnyFileAfterBigFile.txt blocks the shrinking because it was placed at the end of the parity file

snapraid locate -t 1m -c snapraid.conf

Output:

Loading state from C:/_SnapraidTest/snapraid.content...
SnapRAID locate report:

Locate files within the tail of 1000 kB of the parity

Current parity size is 22 MB
Collecting files with offset greater or equal to 21233664

Located data in this range: 7 B

       Offset         Span    Frags
    22020096       262144        1 AnyFileAfterBigFile.txt

Run sync with "--force-realloc-tail" to try to shrink parity file by at least 1 mb

snapraid sync --force-realloc-tail -t 1m -c snapraid.conf

DRU1:

Verzeichnis von C:\_SnapraidTest\dru1

29.01.2026  10:57                 7 AnyFileAfterBigFile.txt
03.02.2026  23:01                 1 Data1.txt.txt
03.02.2026  23:01                 2 Data2.txt.txt
               3 Datei(en),             10 Bytes

PPU1:

Verzeichnis von D:\_SnapraidTest\ppu1

03.02.2026  23:12               314 snapraid.content
03.02.2026  23:12           786.432 snapraid.parity
               2 Datei(en),        786.746 Bytes

Voila, parity shrunk to 3 files * 256kb blocksize/file => 768 kb
This operation won't have to realloc the whole raid which could save hours/days of time.
Also the moving of the files out of the raid, sync and move back and sync is also unnecessary. I would assume all data is now parity-protected all the time.
Via the "locate" operation it can be checked if this operation is necessary.

Feel free to check the code and adapt it to your snapraid guidelines 🙂

amadvance · 2026-02-06T12:43:59Z

Thanks! I'll review it next week!

Copilot

Pull request overview

This PR implements a new --force-realloc-tail (-X) option for the sync command that allows selective reallocation of only the files occupying the parity tail, rather than forcing a full parity reallocation. This feature aims to shrink oversized parity files more efficiently by reallocating only the blocks that prevent shrinking, avoiding the time-consuming process of either full reallocation or manual file movement.

Changes:

Added force_realloc_tail flag to the snapraid_option structure
Implemented command-line option -X/--force-realloc-tail with associated validation
Created state_locate_mark_tail_blocks_for_resync() function to mark tail blocks for reallocation during sync

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 17 comments.

File	Description
cmdline/state.h	Adds force_realloc_tail flag to snapraid_option struct
cmdline/snapraid.c	Implements command-line parsing, validation, and integration of force-realloc-tail option
cmdline/locate.h	Declares new state_locate_mark_tail_blocks_for_resync function
cmdline/locate.c	Implements tail block marking logic and refactors state_locate to extract common functionality

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

cmdline/snapraid.c

cmdline/locate.c

cmdline/snapraid.c

cmdline/locate.c

cmdline/snapraid.c

amadvance · 2026-02-06T12:53:52Z

Hi @Ralf1108

I started a Copilot review. Ignore all review comments about white spacing, they will be fixed automatically. Check the others. No idea if they are valid or not. Use your human common sense :)

Ralf1108 · 2026-02-06T14:21:36Z

cmdline/locate.c

+				struct snapraid_block* block = fs_file2block_get(file, f);
+
+				// TODO: check: is condition correct or should we always set BLOCK_STATE_REP?
+				if (block_state_get(block) == BLOCK_STATE_BLK) {


@amadvance please check if this is correct

Ralf1108 · 2026-02-06T14:22:37Z

cmdline/snapraid.c

+		// TODO: check if this is the correct place for marking, before or after regular scan?
+		// "scan_file_keep()"" will be called inside the following "state_scan()" so we have to mark the file before
+		if(state.opt.force_realloc_tail)
+			state_locate_mark_tail_blocks_for_resync(&state, parity_tail);


@amadvance please check if this is correct place to mark the blocks

Ralf1108 · 2026-02-06T14:23:59Z

Hi @amadvance,

I added 2 comments where you should check if the logic is correct and does not mess other stuff up 👍

Ralf1108

fixed all issues

Ralf1108 · 2026-02-06T14:39:02Z

btw. @amadvance can you convert the manual test in starting post into an automated test?

amadvance · 2026-02-09T13:48:53Z

@Ralf1108 Merged!

The only significant change is that, when a file needs to be reallocated, it must now be reallocated in full.
This is required to ensure that the blocks of each file remain in the correct order within the parity.

I've also added corresponding tests and updated the documentation.

Thanks again!

Ralf1108 · 2026-02-09T15:58:36Z

Thx for the merge!

Question regarding data safety during sync with "--force-realloc-tail": docs

WARNING! This option is for experts only, and it is highly
recommended not to use it.
You DO NOT have data protection during the \`sync\` operation
for the affected files.

I thought the parity of the affected files are valid until the parity file was shrunk which happens at the end of the sync process.
Or does the parity becomes unsafe when the content files are rewritten the first time during sync?

amadvance · 2026-02-10T09:08:00Z

The issue is that at the startup the files are reallocated, and the information of where was their parity is lost, even if the parity is still there. The program keeps track only of the new position in the parity.

Ralf1108 · 2026-02-10T13:28:01Z

So a hint added to the docs for the "--force-realloc-tail" option to precheck the affected files could be helpful.

implemented force-realloc-tail

47768f6

Ralf1108 marked this pull request as ready for review February 3, 2026 22:30

fixed comment

9a00971

amadvance requested a review from Copilot February 6, 2026 12:44

Copilot started reviewing on behalf of amadvance February 6, 2026 12:44 View session

Copilot AI reviewed Feb 6, 2026

View reviewed changes

Ralf1108 added 2 commits February 6, 2026 15:12

check if block is in normal state BLOCK_STATE_BLK

0477426

fixed comment

c1f8e25

Ralf1108 commented Feb 6, 2026

View reviewed changes

Ralf1108 added 2 commits February 6, 2026 15:31

added check to prevent using force_realloc_tail with force_full

fe6e85c

dont allow force-realloc-tail with --force-nocopy

2f7a040

Ralf1108 commented Feb 6, 2026

View reviewed changes

amadvance force-pushed the master branch from f4d19e4 to 60010a4 Compare February 9, 2026 13:40

amadvance closed this Feb 9, 2026

Conversation

Ralf1108 commented Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Init test raid

First sync

Adding big file ~ 20mb

Adding small file to be at the parity tail

Delete big file -> notice that the parity won't shrink

Checking for blocking files -> AnyFileAfterBigFile.txt blocks the shrinking because it was placed at the end of the parity file

Run sync with "--force-realloc-tail" to try to shrink parity file by at least 1 mb

Uh oh!

amadvance commented Feb 6, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

amadvance commented Feb 6, 2026

Uh oh!

Ralf1108 Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

Ralf1108 Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

Ralf1108 commented Feb 6, 2026

Uh oh!

Ralf1108 left a comment

Choose a reason for hiding this comment

Uh oh!

Ralf1108 commented Feb 6, 2026

Uh oh!

amadvance commented Feb 9, 2026

Uh oh!

Ralf1108 commented Feb 9, 2026

Uh oh!

amadvance commented Feb 10, 2026

Uh oh!

Ralf1108 commented Feb 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Ralf1108 commented Feb 3, 2026 •

edited

Loading