[Proposal] Sync files across multiple devices via Resilio Sync

[subsys-R9boq8]

First plz allow me to express my respect and my deepest concern of the project owner, recording history has been hard in this cursed land, and has never been harder since the pandemic started.

The current way of publishing on the GitHub cannot survive a state-level take down (and the collateral damage related to the project initiator's identity being compromised), if we sync documents via Resilio Sync, however, taking down will be impossible, and sync from multiple devices hosted by volunteers can provide the project initiator with extra anonymity.

A simple intro of Resilio Sync can be found in here.

[invishantom]

Thank you for your suggestion.
(Actually I'm just trying to preserve the great work done by their original authors as is. I un-forked the repos mainly driven by the idea to make them easier to discover. Honestly, I lack the energy to maintain them as their previous maintainers did.)

I consider distributed version control systems (DVCS) to be a superior option in archiving for the following reasons:

• They can provide version history, like a wiki. Although this may not seem necessary for archiving news articles since they normally won't change after publication, it can actually play a vital role in determining the authenticity of the repo content, if the original repo is somehow lost (eg. recent censorship activities). Users can verify a repo by comparing its commit hashes with those obtained elsewhere.
• Replicating a repo is easy. A git clone command is just what's needed to get a full copy of the repo. This is made even easier with the fork feature provided by GitHub.
• Git itself doesn't necessarily leaks your information. The author and committer metadata of a commit can be manipulated by users. Pushing commits to a git server usually requires a valid account, but they are not hard to get. However, users should pay attention to social engineering attacks in this case.

For Resilio Sync, I think it does have some advantages to git for simple file sharing usecases. It's a decentralized file-sharing tool, and suits program-think's need well (he has sole control over the content, others just need to sync with his version). It's also more intuitive to users than a CLI tool like git (although it has some GUI frontends).

However, it has some deficiencies that may make it unsuitable for collaboratively maintaining an archive:

• Lack of version control, which may lead to confusing divergence.
• Lack of fine-grained permission control, so a compromised contributor with R/W permission may purge the share folder or cause other damages. This is worsened by the lack of version control, that it may be cumbersome to revert to the last known good status. On GitHub, project owners can make a branch protected to prevent the history from being rewritten.
• Closed-source commercial tool, which might be undesirable for some paranoid users.

Platforms like GitHub may be susceptible to future government takedown requests (it's now a Microsoft subsidiary, who is friendly to the PRC government) or DMCA takedown notices. However, git itself is distributed by design and there are other hosting providers (GitLab, Bitbucket, SourceForge, etc.) It's always to possible to find a provider with less-strict regulatory requirements.

To me, a takeaway from recent repo vanishments is that they should be less reliant on some non-portable platform-specific features, like Issues and Pull Requests of GitHub. These artifacts won't be carried into forked repos, and are lost when the original repo is deleted. It's also a pity that Academic-nCoV/2019-nCoV chose to use the Wiki, which is also not carried when forking. Although wiki on GitHub can be manipulated like a git repo, no one seems to have done that, and they are now lost.
Although the aforementioned content can be seen in the archive.org Wayback Machine, it's likely to be incomplete, and will likely require some manual work if someone else wants to recreate the original content from these HTML archives. Its edit history seems not captured by the Wayback Machine and might be lost permanently.

TL;DR: To those brave archivists: Be careful hiding your identity, and consider maintaining multiple copies of the repo on different platforms if you decide to use git to manage the archive.

[wizardforcel]

ipfs or 0net is also a good idea.

and BT has a critical fault of needing centralized trackers, however ipfs doesn't.

[invishantom]

@wizardforcel I agree that decentralized networks like IPFS and ZeroNet can handle archival documents and might be easier for user to browse than a raw git repository if frontends like GitHub is not available.

However, I'm a bit worried about their scalability in terms of the amount of content. Some archival repos for the ongoing COVID-19 pandemic are quite big, especially for nCovMemory: A whopping size of more than 4.22 GiB. It will be much lighter if the screenshots of articles are stripped, but sometimes screenshot is the only feasible way to fully preserve an article as is.

ZeroNet has a default size limit of 10MB, although it can be loosened by users. I really doubt if a site of this size can be effectively distributed in this way.

I admit I'm not familiar with those tools. Maybe someone with more knowledge can try to publish archives with them.

[subsys-R9boq8]

@invishantom

Lack of version control
Lack of permission control

I think you probably got me wrong, instead of replacing git with a sync tool, by proposing syncing via resilio sync I intended to provide a way of backing up the whole repo and to provide another channel for submitting materials to the project owner (in a separate Sync, of course) for contributers unfamiliar with git operations in fact. The sole purpose is to avoid someone (but not the legitimate owner) with control of the administrative account delete the whole repo, and/or worse, confiscating all the project contributers' storage devices, thus making the latest version disappear on the Internet. In this case, the Sync would not need such version control/permission control anymore.

Please consider.

[invishantom]

@subsys-R9boq8 Thank you for your clarification.

First, I might have to reiterate my position stated in comment #1:

Honestly, I lack the energy to maintain them as their previous maintainers did.

I guess you have not cloned these huge archival repos like this one to your local machine, so I will try to explain what it looks like there. Git stores the content of every commit in its database, which by default lives in .git/ directory under the root of a repository. The size of this repo is 4.22 GiB, which is approximately the size you have to download if you git clone it.

However, the whole directory has a size of 8.71 GiB, since git will extract content of the current version to filesystem so that other programs can access it. Such duplication is not avoidable unless the underlying filesystem has block-level deduplication support (which means ZFS and NTFS currently).

Because of the issue described above, it doesn't make sense to share a normal git repo directly using conventional file-sharing applications. However, I think there are two workarounds:

• Share the content of .git/ directory directly, and users will need to configure git to use its local protocol to access the content.
• Strip .git/ and share the archive directly.

The first option is apparently non-optimal, since users will need to set-up Resilio Sync and git, and extracting the content from git needs the same huge amount of storage.

The second option is more realistic than the previous one, however that means the maintainer have to follow program-think's model: Someone or a group of people control a R/W key, and distribute the R/O key to the public. When there is update to the git repo, the maintainer will sync the modification to the shared folder

However, my concern of "lacking fine-grained permission control" still holds in this case. The permission model of Resilio Sync is so simple that it can't stop compromised R/W key holders from damaging the shared folder. The damage will be synced to everyone's shared folder, unless someone learns the news through other channels and stops the sync before damage happens. That is actually worse than the situation here in GitHub, since users can fork the repo to their account. Modifications done to the original repo won't affect the downstream unless users explicitly merge those changes, which can be easily tracked thanks to the nature of VCS.

As for "submitting materials to the project owner in another shared folder", I hope you can elaborate on this point. The maintainer(s) have to somehow distribute the R/W key to those potential contributors. If they just publish the key on the Internet, then everyone can mess with the shared folder. If they respond with a single common key upon request sent via email or PM, attackers can still damage the shared folder while contributors might be driven away because of the raised bar to contribute. Generating a new R/W key per request is too silly and doesn't scale.

I admit I haven't used Resilio Sync for years and lack detailed knowledge of it, so please feel free to correct me. Nevertheless thank you for your suggestion, and really appreciate your passion in making archives more robust and accessible to the general public.

[subsys-R9boq8]

As for "submitting materials to the project owner in another shared folder", I hope you can elaborate on this point.

I mean one creates a new Resilio Sync folder and submit the R/O key to the project owner (in issues or some where else) for him/her to receive the files, process, and complete the submission requests.

can't stop compromised R/W key holders from damaging the shared folder.

Then I propose Resilio Sync as a backup/more convenient (git, of course) way of distributing files but not the mainstream, not "the one and the only way of saving and sharing these files".

P.S. In fact, I cloned the whole repo in April 28 (to submit it to the Telegram Channel @tpv_documentation for snapshotting and spreading) and the proxy server died multiple times during the cloning process and screwed everything up, that's where the idea of utilizing Resilio Sync to ease the distribution process came from. XD