Initial clone takes FOREVER, suggest depth argument

[codefaux]

Hi there. Amazing work, love it.

I'm deploying a new DNS server, and wanted to use this repo for my current attempt.

My testbed was an RPi and it took FOREVER to do the first 'git clone' of the repo. Thinking it was "SD card slowdown" I decided to abort and move it onto my USB-C NVME SSD. It was only vaguely faster. I assumed it was a CPU horsepower thing.

Today, I deployed it on my primary workhorse, and was shocked to find it was the same.

I suggest a note for users to use a depth argument for deployments, if they're going to use the repo directly. The argument essentially ignores anything older than the specified depth, thus reducing the object count by more than several orders of magnitude.

I know this may not be the "ideal intended means" of deploying your hosts data, but I've seen several dozen scripts include this repo via git clone and I feel like it could be a welcome find for some folks. Even if it's just a closed issue, it's more easily discovered, so I figured I'd put it here.

PS, the time git clone command I started before typing this Issue out is at 23% and I have gigabit internet and a moderately competitive workstation running NVME SSDs; the slowdown is very real regardless of hardware spec.

Adding depth 1, to only retrieve the current version:

[faux@Hostname docker/coredns]$ time git clone --depth 1 https://github.com/StevenBlack/hosts.git
Cloning into 'hosts'...
remote: Enumerating objects: 341, done.
remote: Counting objects: 100% (341/341), done.
remote: Compressing objects: 100% (278/278), done.
remote: Total 341 (delta 78), reused 267 (delta 60), pack-reused 0 (from 0)
Receiving objects: 100% (341/341), 5.28 MiB | 1.07 MiB/s, done.
Resolving deltas: 100% (78/78), done.

real	0m7.093s
user	0m0.699s
sys	0m0.181s

And with no depth argument:

[faux@Hostname docker/coredns]$ time git clone https://github.com/StevenBlack/hosts.git
Cloning into 'hosts'...
remote: Enumerating objects: 89363, done.
remote: Counting objects: 100% (734/734), done.
remote: Compressing objects: 100% (216/216), done.
remote: Total 89363 (delta 693), reused 518 (delta 518), pack-reused 88629 (from 4)
Receiving objects: 100% (89363/89363), 1.49 GiB | 1.17 MiB/s, done.
Resolving deltas: 100% (56795/56795), done.

real	22m53.063s
user	4m56.170s
sys	0m24.311s

[StevenBlack]

Thank you for this suggestion @codefaux — I'm soaking what you're spilling, one hundred percent.

This clone-time cost has been bothering me for quite awhile.

I think I'll document this suggestion in the readme files.

[StevenBlack]

Thank you again @codefaux — here is what the repository home page, and all its internal readme files, look like starting now:

Closing.

[Rudxain]

Also worth noting the --filter option for blob-less or tree-less clones. I even have git-aliases for those (more tips at the start of the file)

[codefaux]

@StevenBlack - That's pretty fantastic, thank you for your positive response! Apologies for the slow reply. Glad I could help.

@Rudxain - Thanks for the additional feedback here. Would you be willing to help out with understanding it?

I looked at the article, and the config snippet you linked, but I'm afraid I'm unable to fully understand the implications of "blob-less" or "tree-less" clones. It seems it was written for someone with more git familiarity than I possess, lol.

The article indicates that blobs represent file contents, trees represent directories, but it doesn't so much explain the repercussions of using these features, and perhaps it's too early for me to be reading things like this but I'm not quite getting it, and I'm not entirely confident I'd be able to understand the full implications with just a little experimenting.

Would you be willing to clarify what's left out, or provide an example of when these would be useful?

[Rudxain]

I explain a bit more here:

• clone tips: partial, shallow, sparse jesseduffield/lazygit#4907
• Recommend blob-less Git clone KristalTeam/Shadow#26

Official docs here

I should warn that backfill is designed only for blob-less, and it will be unhelpful for tree-less. Converting tree-less to blob-less seems possible via --refetch (I haven't tested. Remember to change the filter in .git/config beforehand). Converting blob-less to tree-less is possible by doing a local-to-local clone. That's yet another reason to avoid tree-less.

The GH blog-post I linked also links to this post, whose title is technically-correct but misleading, and they even acknowledge it by linking here. Even more info can be found here (I recommend skipping to the end for a TLDR)