LFS data

[tbaudier]

Follow PR #48 (@dsarrut @SimonRit )

I open an issue to discuss about LFS data.
I gave up the PR to focus on rebase instead of revert.

I push the rebase of @jmletang 's commit but still some data are not correctly followed. Find the list bellow. It's .png files and Materials.db files.
For png files I will convert them to LFS. Does for Materials.db need to be LFS too because they are text files?

As we already rebase once I will find all blamed commit and rebase them to have a clean repository.

[tbaudier]

The problematic files:

Encountered 94 file(s) that should have been pointers, but weren't:
GATE-RTion/DoseActor/data/HFMaterials2014.db
GATE-RTion/DoseActor2-CT/data/HFMaterials2014_test.db
GATE-RTion/TPSPencilBeamSource/data/GateMaterials.db
GATE-RTion/TPSPencilBeamSource/data/MyMaterials2010.db
GATE-RTion/TPSPencilBeamSource/data/MyMaterials2013.db
GATE-RTion/TPSPencilBeamSource/data/patient-HUmaterials.db
dosimetry/DNA/GateMaterials.db
dosimetry/LET/data/GateMaterials.db
dosimetry/Radiotherapy/example1/data/GateMaterials.db
dosimetry/Radiotherapy/example10/data/GateMaterials.db
dosimetry/Radiotherapy/example10/data/phantom-HUmaterials.db
dosimetry/Radiotherapy/example11/data/Materials.db
dosimetry/Radiotherapy/example12/data/GateMaterials.db
dosimetry/Radiotherapy/example12/data/LinacMaterials.db
dosimetry/Radiotherapy/example2/data/GateMaterials.db
dosimetry/Radiotherapy/example3/data/GateMaterials.db
dosimetry/Radiotherapy/example3/data/PreciseMaterials.db
dosimetry/Radiotherapy/example3/data/patient-HUmaterials.db
dosimetry/Radiotherapy/example4/data/GateMaterials.db
dosimetry/Radiotherapy/example5/data/GateMaterials.db
dosimetry/Radiotherapy/example6/data/GateMaterials.db
dosimetry/Radiotherapy/example6/data/patient-HUmaterials.db
dosimetry/Radiotherapy/example7/data/GateMaterials.db
dosimetry/Radiotherapy/example8/data/GateMaterials.db
dosimetry/Radiotherapy/example9/data/GateMaterials.db
dosimetry/TEPCActor/data/myGateMaterials.db
dosimetry/UserSpectrum/data/GateMaterials.db
dosimetry/doseByRegions/data/AF_GateMaterials.db
dosimetry/doseactor/unvoxelized-example/data/GateMaterials.db
dosimetry/doseactor/voxelized-phantom/data/GateMaterials.db
dosimetry/dosimetry/brachytherapy/data/GateMaterials.db
dosimetry/dosimetry/electron_radiotherapy/data/GateMaterials.db
dosimetry/dosimetry/external-beam-therapy-photon/data/GateMaterials.db
dosimetry/dosimetry/molecular-therapy-I131/data/GateMaterials.db
dosimetry/dosimetry/protontherapy/data/GateMaterials.db
dosimetry/gaga-phsp/data/GateMaterials.db
imaging/CT/Fresnel_FFD/data/GateMaterials.db
imaging/CT/XRayRefraction/data/GateMaterials.db
imaging/CT/fixedForcedDetectionCT/data/ICRP_HUmaterials.db
imaging/ComptonCamera/data/GateMaterials.db
imaging/DICOM/mhd-dcm-comparison/data/GateMaterials.db
imaging/ExtendedVSource/CommonMacros/GateMaterials.db
imaging/ExtendedVSource/Example_1/histograms/EnergyDepositionAnnihilation_pPs.png
imaging/ExtendedVSource/Example_1/histograms/EnergyDepositionPrompt_pPs.png
imaging/ExtendedVSource/Example_1/histograms/EnergyDeposition_pPs.png
imaging/ExtendedVSource/Example_1/histograms/Lifetime_pPs.png
imaging/ExtendedVSource/Example_1/histograms/SourcePosition_pPs.png
imaging/ExtendedVSource/Example_1/histograms/energy_deposition_spectra_pPs.png
imaging/ExtendedVSource/Example_2/histograms/AngleDistributionScattered_oPs.png
imaging/ExtendedVSource/Example_2/histograms/AngleDistribution_oPs.png
imaging/ExtendedVSource/Example_2/histograms/EnergyDepositionAnnihilation_oPs.png
imaging/ExtendedVSource/Example_2/histograms/EnergyDepositionPrompt_oPs.png
imaging/ExtendedVSource/Example_2/histograms/EnergyDeposition_oPs.png
imaging/ExtendedVSource/Example_2/histograms/Lifetime_oPs.png
imaging/ExtendedVSource/Example_2/histograms/SourcePosition_oPs.png
imaging/ExtendedVSource/Example_2/histograms/energy_deposition_spectra_oPs.png
imaging/ExtendedVSource/Example_3/histograms/EnergyDepositionAnnihilation_Ps.png
imaging/ExtendedVSource/Example_3/histograms/EnergyDepositionPrompt_Ps.png
imaging/ExtendedVSource/Example_3/histograms/EnergyDeposition_Ps.png
imaging/ExtendedVSource/Example_3/histograms/Lifetime_Ps.png
imaging/ExtendedVSource/Example_3/histograms/SourcePosition_Ps.png
imaging/ExtendedVSource/Example_3/histograms/energy_deposition_spectra_Ps.png
imaging/ExtendedVSource/Example_4/histograms/EnergyDepositionAnnihilation_pPs.png
imaging/ExtendedVSource/Example_4/histograms/EnergyDepositionPrompt_pPs.png
imaging/ExtendedVSource/Example_4/histograms/EnergyDeposition_pPs.png
imaging/ExtendedVSource/Example_4/histograms/Lifetime_pPs.png
imaging/ExtendedVSource/Example_4/histograms/SourcePosition_pPs.png
imaging/ExtendedVSource/Example_4/histograms/energy_deposition_spectra_pPs.png
imaging/LUTDavisModel/GateMaterials.db
imaging/Optical/GateMaterials.db
imaging/ProtonRadiography/data/GateMaterials.db
imaging/SPECT_FFD/data/GateMaterials.db
imaging/SPECT_GPU/GateMaterials.db
imaging/Thermal/GateMaterials.db
imaging/UserFluenceSource/data/GateMaterials.db
imaging/fastY90/data/GateMaterials.db
imaging/fluorescence/data/myGateMaterials.db
imaging/garf/data/GateMaterials.db
imaging/garf/data/phantom_HUmaterials.db
imaging/vpgtle/stage0/data/GateMaterials.db
imaging/vpgtle/stage0/data/GateMaterialsElements.db
imaging/vpgtle/stage1/data/GateMaterials.db
imaging/vpgtle/stage1/data/patient-HUmaterials.db
imaging/vpgtle/stage2/data/GateMaterials.db
misc/GateMaterials.db
misc/GateMaterialsGPU.db
misc/TetrahedralMeshGeometry/data/GateMaterials.db
misc/TrackLengthFluenceExamples/TLFluence-Example1/Data/GateMaterials.db
misc/TrackLengthFluenceExamples/TLFluenceDistribution-Exampl1/Data/GateMaterials.db
misc/TrackLengthFluenceExamples/Track_Length_Actors.pdf
misc/geometry_STL/Suzanne/GateMaterials.db
misc/geometry_STL/kidneys/data/GateMaterials.db
misc/hybrid_volume_navigator/MergedVolume/GateMaterials.db
misc/hybrid_volume_navigator/Reference/GateMaterials.db

[dsarrut]

I think Materials.db should not be LFS because they are txt files.

[tbaudier]

They are listed in .gitattributes: https://github.com/OpenGATE/GateContrib/blob/master/.gitattributes#L15

...
*.TABLE filter=lfs diff=lfs merge=lfs -text
*.db filter=lfs diff=lfs merge=lfs -text

[dsarrut]

maybe there are other .db files that are binaries ?

[tbaudier]

They are all text files:

 ~/Software/GateContrib> file `find . -iname "*.db"`
./GATE-RTion/DoseActor/data/HFMaterials2014.db:                                        ASCII text
./GATE-RTion/DoseActor2-CT/data/HFMaterials2014_test.db:                               ASCII text
./GATE-RTion/TPSPencilBeamSource/data/GateMaterials.db:                                ASCII text
./GATE-RTion/TPSPencilBeamSource/data/MyMaterials2010.db:                              ASCII text
./GATE-RTion/TPSPencilBeamSource/data/MyMaterials2013.db:                              ASCII text
./GATE-RTion/TPSPencilBeamSource/data/patient-HUmaterials.db:                          ASCII text
./dosimetry/DNA/GateMaterials.db:                                                      ASCII text
./dosimetry/EMFields/data/HFMaterials2014.db:                                          ASCII text
./dosimetry/LET/data/GateMaterials.db:                                                 ASCII text
./dosimetry/Radiotherapy/example1/data/GateMaterials.db:                               ASCII text
./dosimetry/Radiotherapy/example10/data/GateMaterials.db:                              ASCII text
./dosimetry/Radiotherapy/example10/data/phantom-HUmaterials.db:                        ASCII text
./dosimetry/Radiotherapy/example11/data/Materials.db:                                  ASCII text
./dosimetry/Radiotherapy/example12/data/GateMaterials.db:                              ASCII text
./dosimetry/Radiotherapy/example12/data/LinacMaterials.db:                             ASCII text
./dosimetry/Radiotherapy/example2/data/GateMaterials.db:                               ASCII text
./dosimetry/Radiotherapy/example3/data/GateMaterials.db:                               ASCII text
./dosimetry/Radiotherapy/example3/data/PreciseMaterials.db:                            ASCII text
./dosimetry/Radiotherapy/example3/data/patient-HUmaterials.db:                         ASCII text
./dosimetry/Radiotherapy/example4/data/GateMaterials.db:                               ASCII text
./dosimetry/Radiotherapy/example5/data/GateMaterials.db:                               ASCII text
./dosimetry/Radiotherapy/example6/data/GateMaterials.db:                               ASCII text
./dosimetry/Radiotherapy/example6/data/patient-HUmaterials.db:                         ASCII text
./dosimetry/Radiotherapy/example7/data/GateMaterials.db:                               ASCII text
./dosimetry/Radiotherapy/example8/data/GateMaterials.db:                               ASCII text
./dosimetry/Radiotherapy/example9/data/GateMaterials.db:                               ASCII text
./dosimetry/TEPCActor/data/myGateMaterials.db:                                         ASCII text
./dosimetry/UserSpectrum/data/GateMaterials.db:                                        ASCII text
./dosimetry/doseByRegions/data/AF_GateMaterials.db:                                    ASCII text
./dosimetry/doseactor/unvoxelized-example/data/GateMaterials.db:                       ASCII text
./dosimetry/doseactor/voxelized-phantom/data/GateMaterials.db:                         ASCII text
./dosimetry/dosimetry/brachytherapy/data/GateMaterials.db:                             ASCII text
./dosimetry/dosimetry/electron_radiotherapy/data/GateMaterials.db:                     ASCII text
./dosimetry/dosimetry/external-beam-therapy-photon/data/GateMaterials.db:              ASCII text
./dosimetry/dosimetry/molecular-therapy-I131/data/GateMaterials.db:                    ASCII text
./dosimetry/dosimetry/protontherapy/data/GateMaterials.db:                             ASCII text
./dosimetry/gaga-phsp/data/GateMaterials.db:                                           ASCII text
./imaging/CT/Fresnel_FFD/data/GateMaterials.db:                                        ASCII text
./imaging/CT/XRayRefraction/data/GateMaterials.db:                                     ASCII text
./imaging/CT/fixedForcedDetectionCT/data/ICRP_HUmaterials.db:                          ASCII text
./imaging/ComptonCamera/data/GateMaterials.db:                                         ASCII text
./imaging/DICOM/mhd-dcm-comparison/data/GateMaterials.db:                              ASCII text
./imaging/ExtendedVSource/CommonMacros/GateMaterials.db:                               ASCII text
./imaging/LUTDavisModel/GateMaterials.db:                                              ASCII text
./imaging/Optical/GateMaterials.db:                                                    ASCII text
./imaging/ProtonRadiography/data/GateMaterials.db:                                     ASCII text
./imaging/SPECT_FFD/data/GateMaterials.db:                                             ASCII text
./imaging/SPECT_GPU/GateMaterials.db:                                                  ASCII text
./imaging/Thermal/GateMaterials.db:                                                    ASCII text
./imaging/UserFluenceSource/data/GateMaterials.db:                                     ASCII text
./imaging/fastY90/data/GateMaterials.db:                                               ASCII text
./imaging/fluorescence/data/myGateMaterials.db:                                        ASCII text
./imaging/garf/data/GateMaterials.db:                                                  ASCII text
./imaging/garf/data/phantom_HUmaterials.db:                                            ASCII text
./imaging/vpgtle/stage0/data/GateMaterials.db:                                         ASCII text
./imaging/vpgtle/stage0/data/GateMaterialsElements.db:                                 ASCII text
./imaging/vpgtle/stage1/data/GateMaterials.db:                                         ASCII text
./imaging/vpgtle/stage1/data/patient-HUmaterials.db:                                   ASCII text
./imaging/vpgtle/stage2/data/GateMaterials.db:                                         ASCII text
./misc/GateMaterials.db:                                                               ASCII text
./misc/GateMaterialsGPU.db:                                                            ASCII text
./misc/TetrahedralMeshGeometry/data/GateMaterials.db:                                  ASCII text
./misc/TrackLengthFluenceExamples/TLFluence-Example1/Data/GateMaterials.db:            ASCII text
./misc/TrackLengthFluenceExamples/TLFluenceDistribution-Exampl1/Data/GateMaterials.db: ASCII text
./misc/geometry_STL/Suzanne/GateMaterials.db:                                          ASCII text
./misc/geometry_STL/kidneys/data/GateMaterials.db:                                     ASCII text
./misc/hybrid_volume_navigator/MergedVolume/GateMaterials.db:                          ASCII text
./misc/hybrid_volume_navigator/Reference/GateMaterials.db:                             ASCII text

[SimonRit]

I would also have added .db files to regular git. I'm not aware of any binary db file and I don't see any in the repo.
Any clue how jm screwed up the repository with this .gitattributes file?

[tbaudier]

All these problems come from different commits. 39e62ad includes .db files inside the LFS system (with a commit it's easy to remove).
This commit 32c4770 includes a pdf file inside Github, so I need to rebase these commit to remove the Github history of this file and put it in the LFS.
And same problem with .png files for this commit d1b6993

[SimonRit]

39e62ad includes .db files inside the LFS system (with a commit it's easy to remove).

With a clear commit message explaining why...

This commit 32c4770 includes a pdf file inside Github, so I need to rebase these commit to remove the Github history of this file and put it in the LFS.
And same problem with .png files for this commit d1b6993

I think @dsarrut chose consciously to include some small binary files in Gate repositories.

[tbaudier]

I rewrote the history for these 3 previous commits with a rebase.
The updated repository can be found here: https://gitlab.in2p3.fr/tbaudier/gatecontrib/-/commits/master
And now the initial LFS error is not present.

But I saw that some binary are not staged by LFS (not present in gitattribute):

GATE-RTion/DoseActor/DoseActor_Validation sheet_2018-06-19.xlsx
GATE-RTion/DoseActor2-CT/template_results.ods
GATE-RTion/TPSPencilBeamSource/LGR_TPS-Source-Validation-Procedure.odt
dosimetry/Radiotherapy/example11/data/1000p.IAEAphsp

[tbaudier]

I updated the repository to remove the previous binary files from the history and put them in LFS. Everything can be seen here:
https://gitlab.in2p3.fr/tbaudier/gatecontrib
I had also update the readme to announce the rebase.
Is it OK, Can I pull inside this github repository ?

[SimonRit]

I was under the impression that with git lfs, we should have a small .git directory, with binary files being only pointers to files hosted on github. When I clone your repo, the .git directory is 686 MB. I don't understand what the problem is but when I run the command suggested here, with the title A blazingly fast shell one-liner, it seems that some large commits are still present but I don't understand how to clean this up. The commits which are large seem inaccessible so here stops my understanding of git. (When I run git clone --single-branch, it's worse, 952 MB!)

[tbaudier]

Thank you for the remarks. On my computer, I can see different things:

1. If I clone the gitlab repository (git clone), the total size is 2.6 GB with 1.6 GB for .git folder
2. If I clone the gitlab repository without pulling LFS (GIT_LFS_SKIP_SMUDGE=1 git clone), the total size of the repository is 694 MB with 686 MB for .git folder
3. If I clone the master branch only of the gitlab repository without pulling LFS (GIT_LFS_SKIP_SMUDGE=1 git clone --single-branch), the total size is 28 MB with 20 MB for .git folder
4. If I clone the master branch only of the gitlab repository (git clone --single-branch), the total size is 1.9 GB with 952 MB for .git folder

For the point 1/ and 2/, the size is important because I cloned all the branches (master, revert and testRebase) and the two last branches could be erased. They still contains binary files.
For the 3/ and 4/ points the size is less important, but we can still encounter old large files in the history. For example with this commit converted to LFS now or this one deleted now.

To clean the repository, I need to remove all binary files in the history. Your link could help me but:

• I need to run it on the repository clone like 3/ because If I download the LFS data, it returns the large LFS data first
• It return the files sorted according their size but some text file are quite huge (example with a deleted text file 2.25 MB)
• To identify the blaming commit I need to use this line
• This perl script can help me to identify suspicious commit. It gives me the largest commit in size with the number of modified files, eg:

...
1030936 6d5825e79836d3d366d56bb4272c7cd5e11e0963 2
...
3313787 e56954b1ff100c1a73816be0e934f9bd6c372bb2 1180

The first commit change 1MB in 2 files, the last 3.3 MB in 1180 files.

I will try to propose something using all these tools.

[SimonRit]

3\. If I clone the master branch only of the gitlab repository without pulling LFS (`GIT_LFS_SKIP_SMUDGE=1 git clone --single-branch`), the total size is 28 MB with 20 MB for `.git` folder

I had tried git clone --single-branch but without GIT_LFS_SKIP_SMUDGE. This is much better thanks. I would list all files in the history and check those we want to keep / remove the others with bfg. For example, the three following commands halved the size of the repo (19MB -> 9MB):

java -jar ~/Downloads/bfg-1.13.0.jar --delete-files SJAN_GC_2009.ppt
java -jar ~/Downloads/bfg-1.13.0.jar --delete-folders source
git reflog expire --expire=now --all && git gc --prune=now --aggressive