Make the photon a downloadable resource

[achiefa]

This PR addresses #2214

[achiefa]

Hi @scarlehoff , can you please start having a look? I'm pretty sure this needs to be tweaked a bit more, but it's a good starting point. I tried using a local photon set, generated using the new script fiatlux_exec.py, to check if I implemented the loading part correctly, and it works. Some major things are still missing:

• A proper remote folder, where the photon can be downloaded. Note that the code in download_photonQED is commented out because I'm waiting for instructions on this respect.
• Tests need to be updated. But I'd prefer to wait until a final version of this PR.

[scarlehoff]

A proper remote folder, where the photon can be downloaded. Note that the code in download_photonQED is commented out because I'm waiting for instructions on this respect.

Let's use the Milan server for this for now: /home/nnpdf/WEB/photons this is a sibling of e.g. /home/nnpdf/WEB/hyperscans

[scarlehoff]

I've only had a look at the download. The parallelization I need more time to be able to give meaningful comments.

Did you already tested and saw that 1) it was faster 2) it didn't make the memory explode 3) the results are correct (god knows whether fiatlux is threadsafe)

[scarlehoff]

I think the parallelization is ok as well (provided it runs, I didn't test it).

I think for now (before testing the download online) just make it so the download is a copy from some folder in your computer to the share folder. If that works the download from the server is a technicality and will work just the same.

I think this will already ease quite a bit the QED fits (e.g., all the positivity checks that are being done now could be done with QED for free thanks to this for the price of one single computation of the photon!)

[achiefa]

Please, have a look at this training report (link). In the new fit, I downloaded the photon previously computed with the script fiatlux-exec.

One thing that is left is whether we want to move fiatlux-exec into vp-setupfit fit. In general, we need to agree on how we want to compute and upload the photon PDF.

[scarlehoff]

Thanks for this. I've tried the light photons from the tests and seems to do ok. I'll try with some heavier photon, I'm curious about how heavy the eko will be in that case.

[scarlehoff]

I'm happy with the changes, just a few comments about the wording.

And a big comment about the docs! Could you please update the docs to include this functionality?

At the very least, explain it (and how to use it and where int he server are the photons) at the tutorial page https://docs.nnpdf.science/tutorials/run-qed-fit.html

Crucially it should mention:

1. What will happen when you use compute_in_setupfit
2. In which situations it makes sense to have it (best effort)
3. That the number of cores to use is the one in the runcard. FWIW, 64 cores is using 64 GB of RAM, perhaps this is also worth mentioning.

And any other info that you deem important.

I'm running a test 4.0-like QED fit with this branch. I don't expect to see any problems but probably will have finished by tomorrow so we can merge as soon as the docs are added if that doesn't crash smh

[achiefa]

I've added the documentation. Please, have a look and let me know.

[scarlehoff]

You went too specific :'( but ok for the rest

Look, I've submitted a job to create the photon for NNPDF40_nnlo_as_01180_qed I'll let you know when it is done.

[scarlehoff]

The new more official and cooler photon is ready in the server: photon_theoryID_40000100_fit_NNPDF40_nnlo_as_01180_qed.tar

edit: FWIW, while I think it would be better to have it in vp-upload (and I'll be very happy if you do it next) I think with the changes to the docs I mentioned this can be merged and used. The functionality itself is quite nice, orange is this branch:

(and, not seen in the blue, you remove the possibility of being unlucky and having some replicas that won't converge)

[achiefa]

Hi @scarlehoff, thanks for this report. I'm happy with the current status. I'll finalise the doc and merge it. Then I'll open a new PR for vp-upload.