TesseraCT partial tile cleanup

[pimvanpelt]

Description

I archived IPng's halloumi2026h2 logshard with 4765491 leaves, which was consuming 146GB on disk.
Using Geomys' photocamera app, turned the log into a 4.0GB zipfile. Unzipping that, this yields 4.4GB of data on disk. The resulting unzipped data passes ct-fsck:

pim@summer:/ssd-vol0/logs$ du -sch tile/ issuer/
4.4G    tile/
946K    issuer/
4.4G    total
pim@summer:/ssd-vol0/logs$ ~pim/bin/ct-fsck -origin halloumi2026h2.log.ct.ipng.ch         -monitoring_url file:///ssd-vol0/logs/         -public_key="MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAENDhVrTtcQebbk/R6YW8Gt1RW+WZ8qscHUsI2/TW5f7bu+mEst2lskXbR9HEBQvyhkYKVTuigQjgZvDYqRhcn0Q=="         -N 20 -user_agent_info="ct-ops@ipng.ch"
I1205 00:12:37.997709 2546513 main.go:335] Using verifier string: halloumi2026h2.log.ct.ipng.ch+53348351+BTBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABDQ4Va07XEHm25P0emFvBrdUVvlmfKrHB1LCNv01uX+27vphLLdpbJF20fRxAUL8oZGClU7ooEI4Gbw2KkYXJ9E=
I1205 00:12:37.997827 2546513 main.go:112] Checking issuers CAS
I1205 00:12:37.998584 2546513 fsck.go:56] Fsck: checking log of size 4765491
I1205 00:12:54.667362 2546513 fsck.go:111] Successfully fsck'd log with size 4765491 and root ucxfVJxhzGDvsFzvHEmV/U+p3cvjWSHuwLhy+ljp17Q= (b9cc5f549c61cc60efb05cef1c4995fd4fa9ddcbe35921eec0b872fa58e9d7b4)
I1205 00:12:54.667397 2546513 main.go:82] OK
pim@summer:/ssd-vol0/logs$ cat checkpoint 
halloumi2026h2.log.ct.ipng.ch
4765491
ucxfVJxhzGDvsFzvHEmV/U+p3cvjWSHuwLhy+ljp17Q=

— halloumi2026h2.log.ct.ipng.ch UzSDUQAAAZrfajPWBAMARjBEAiBPRUhy0ldb3tJPq0liXcP1EQ+c+pseGzRFad6gNfBjhwIgai3IamIX6KlBeslhmnOp2YgY79jP0GiTqbOCCmx3d2U=

Looking at the source POSIX filesystem, I notice the majority of the files are partial tiles:

root@ctlog1:/ssd-vol0/logs/halloumi2026h2# find . -type f -ls | awk '{ total += $7} END { print total }'
270780557061

root@ctlog1:/ssd-vol0/logs/halloumi2026h2# find . -type f -ls | awk '$11 ~ /.*p\/.*/ { total += $7} END { print total }'
255180358712

And looking at the active IPng logs, comparing Gouda (Sunlight) and Halloumi (TesseraCT):

ssd-vol0/logs/gouda2025h2    569G  5.79T   569G  /ssd-vol0/logs/gouda2025h2
ssd-vol0/logs/gouda2026h1    465G  5.79T   465G  /ssd-vol0/logs/gouda2026h1

ssd-vol0/logs/halloumi2025h2   1.63T  2.85T  1.63T  /ssd-vol0/logs/halloumi2025h2
ssd-vol0/logs/halloumi2026h1   1.59T  2.85T  1.59T  /ssd-vol0/logs/halloumi2026h1

We will need to implement something like partial-aftersun for TesseraCT. If not, IPng's Halloumi logs will run out of usable ZFS space within ~14 weeks.

[phbnf]

Thanks for the report.

Garbage Collection is a Tessera feature, and although at the moment it is not configurable via TesseraCT, it should be enabled by default. It either needs fine tuning, or something prevents it from working at all.

Do you see any warning log message that would indicate that garbage collection doesn't work as expected? Similarly, can you see anything indicating that some files are successfully being deleted?

[pimvanpelt]

No such log messages since 2026-09-08 when the logshard was started:

ctlog@ctlog1:~$ sudo journalctl -u tesseract@halloumi2026h2a.service | head -1
Sep 08 21:51:25 ctlog1 systemd[1]: Started tesseract@halloumi2026h2a.service - Tesseract CT Log service for halloumi2026h2a.
ctlog@ctlog1:~$ sudo journalctl -u tesseract@halloumi2026h2a.service | grep -ci 'Garbage'
0
ctlog@ctlog1:~$ sudo journalctl -u tesseract@halloumi2026h2a.service | grep -ci 'rm '
0

[phbnf]

Tessera's Garbage Collection relies on tlog-tiles' entry path, which happens to be different for CT: it's tiles/entries for tlog-tiles logs and tiles/data for static-ct-api [0]. As a result, garbage collection does not delete partial data tiles it should delete. That's an oversight on our side, and we'll fix it.

To confirm this hypothesis, if you take a partial data path, could you check whether the matching level 0 tile has been garbage collected? Just to avoid any confusion, the rm logs that you greped for would only show up with a log level of 3, so maybe it's not set that high on halloumi's instances? (And that would be totally reasonable.)

I have observed the same behaviour on our test Arche log. As a matter of facts, I can see the same bug in the GCP and AWS implementations.

We'll follow up with a fix. Since garbage collection is stateful, it might require running some manual commands to reset that state and/or delete entries otherwise.

[0] This is a known and intentional spec divergence, since CT data tiles and tlog-tiles are encoded differently.

[pimvanpelt]

Confirmed we do not have verbose logging on Halloumi and Lipase tesseract shards.
I took a snapshot of the halloumi2026h2 log file structure:

ctlog@ctlog1:~$ find /ssd-vol0/logs/halloumi2026h2/data/ -ls -type f | gzip -9 > /tmp/halloumi2026h2.files.gz

It is attached to this comment. PTAL

halloumi2026h2.files.gz

[phbnf]

Thanks, this confirms what I thought was happening. I've got a fix baking here, we should also add a bit of Garbage Collection documentation about this both in Tessera and TesseraCT.

I'll merge this next week, test everything, and let you know when it's ready. Garbage collecting old bundles of currently running logs will require a bit of manual work, which I'd have to test first.