Build-up of the RAM using COSILike for LH fit

[GallegoSav]

When we try to run multiple times the LH fit in a loop (like in the spectral fit tutorial), the memory is somehow not reinitialize and the RAM build-up until it raise a OM error.

Note that we tried in a notebook or also in a regular python script.

The only solution found was to launch multiple instance of the spectral fit tutorial.

[GallegoSav]

Hi @jdbuhler , do you think you have time to look at this bug for DC4 ? I think you are the most suited for this one.

I tried to empty the memory with :

1. re-initialise the cosilike object to 0 at the end of the loop
2. try to use del
3. try to do this for loop of fitting in a scrip instead of jupyter to see if the RAM was still building-up (yes it is)

Your recent pr #470 might solve this though.

[jdbuhler]

I'll see what I can do. Do you have a minimal script that reproduces the bug?

[GallegoSav]

@jdbuhler I would use the crab spectral fit tutorial and put this part in a for loop :

for i in range(10):
    bkg_par = Parameter("background_cosi",                                         # background parameter
                     1,                                                        # initial value of parameter
                     min_value=0,                                              # minimum value of parameter
                     max_value=5,                                              # maximum value of parameter
                     delta=0.05,                                               # initial step used by fitting engine
                     desc="Background parameter for cosi")

    cosi = COSILike("cosi",                                                        # COSI 3ML plugin
                 dr = dr,                                                      # detector response
                 data = crab_bkg.binned_data.project('Em', 'Phi', 'PsiChi'),   # data (source+background)
                 bkg = bkg.binned_data.project('Em', 'Phi', 'PsiChi'),         # background model 
                 sc_orientation = sc_orientation,                              # spacecraft orientation
                 nuisance_param = bkg_par,                                     # background parameter
                 earth_occ = True)                                             # Option to account for Earth occultation

   # change the model flux each iteration
    spectrum.K.value = i*3e-5

    plugins = DataList(cosi) # If we had multiple instruments, we would do e.g. DataList(cosi, lat, hawc, ...)

    like = JointLikelihood(model, plugins, verbose = False)

    like.fit()

[jdbuhler]

I tried your suggestion in the current develop branch with a cosipy environment based on Python 3.10 and using the versions of all packages I get when I install cosipy from develop with 'pip -e' (so no development versions of threeML, astromodels, scoords, mhealpy, etc). Notebook attached below.

I was not able to see any memory growth after the second iteration. This is based on adding up the sizes of all the objects I can see from within the notebook.

If you've got a script version that I can run from the command line, I can just watch the top output and see if it is in fact growing.

fit-crab-overflow.ipynb

[israelmcmc]

If it help, you can convert jupyter notebooks to scripts with nbconvert. e.g.

jupyter nbconvert --to script fit-crab-overflow.ipynb

I'm attaching the output:
fit-crab-overflow.py

If you have troubles with get_ipython().run_cell_magic you can comment-out those lines in the script, or alternatively, remove the cell magics (% and %%) in the notebook before converting it.

[jdbuhler]

I converted the notebook to a script (after removing the magic) in the same environment as above. Memory usage is peaking between 10 and 11 GB in each iteration. There is not a clear signal that the usage rises from one iteration to the next, though I'm limited to the sampling frequency of 'top' in watching the usage change. If there is a leak, it's not a big leak.

[israelmcmc]

Thanks @jdbuhler. @GallegoSav can you also give it a try with the script + top monitoring? I vaguely remember @Yong2Sheng seing something like this at some point, and it seemed to depend on the particularities of the implementation for a given system (it was for the hdf5 library, iirc)

[Yong2Sheng]

Hi @israelmcmc, I remember that I used htop (or top) to monitor the RAM usage for TS fitting. I exported the RAM usage monitoring to a text file then plot RAM vs Time to see how the RAM usage is built up. Is this what you mentioned?

[GallegoSav]

Maybe this crab example is not the best to test that because it is using old data/response/ori file .

I share the test script I used using more recent data and response files.

RAM_Buildup_check.py

Gal_511.yaml

The response used is the 511keV response from your wasabi develop folder @jdbuhler and the DC3 orientation file
I attached the bkg file I used : https://seafile.rlp.net/seafhttp/f/b669e4d6e5e74ef8b088/?op=view

You can see the build-up of the RAM.

Here it is not a lot because this is a point source but for an extended source where the precomputed response is heavy to load, and if you want to run ~100-1k times the fit , you start to increase a lot the memory

[jdbuhler]

Where do I get the file gal_DC4_bg_511_binned_smoothed_fwhm_10.0deg.hdf5? I could not find it on wasabi.

(Never mind -- I see there was a link)