Corsika on NERSC

[sophiemiddleton]

Opening an issue to keep track of future discussions on the on-going attempts to run the Corsika Campaign at NERSC.

Current status:

• S1 works on NERSC but only if files stored back at FNAL
• S2 failing due to Cat error (unrelated to NERSC)

Next steps:

• Run S2 normally
• Run S2 at NERSC

Further ahead:

• Store intermediate files at NERSC

[brownd1978]

It is important to try running S2 at NERSC; that’s where most of the cpu is spent. If we end up copying files back and forth that’s OK, we just need to make sure the files are accessed as local copies, not via XRootD, as that would use 1000X as much network bandwidth. Ray, can you help Sophie configure the S2 jobs?

On Sun, Nov 27, 2022 at 14:10 Sophie Middleton ***@***.***> wrote: Opening an issue to keep track of future discussions on the on-going attempts to run the Corsika Campaign at NERSC. Current status: - S1 works on NERSC but only if files stored back at FNAL - S2 failing due to Cat error (unrelated to NERSC) Next steps: - Run S2 normally - Run S2 at NERSC Further ahead: - Store intermediate files at NERSC — Reply to this email directly, view it on GitHub <#219>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABAH572R4BJLPZV5J5KF4KLWKPL3PANCNFSM6AAAAAASMXICOM> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- David Brown ***@***.*** Office Phone (510) 486-7261 Lawrence Berkeley National Lab M/S 50R5008 (50-6026C) Berkeley, CA 94720

[rlcee]

This is an important note for any job which does resampling - that you want to avoid multiple re-reading over the network. I'm happy to help if needed. I wonder if all MDC resampling was done this way, or was it re-read over the network? If needed, please point me to to the current relevant cfg.

[sophiemiddleton]

Hi Ray,

I think we should get the CFGs working in the current incarnation i.e. remove the S1 Cat atage and run to the end of S2 and then look at changing S2 to use NERSC. I want to separate issues with or actual FCL/Mu2e framework from issues related to the NERSC specific running.

I will post here when I have this completed and we are ready for the updates to get S2 working on NERSC. I think actually that part I can do. The big part will be chanigng both S1 and S2 to not move the file to FNAL, that will require Rays help.

thanks,
Sophie

[brownd1978]

Hi Ray,

On Mon, Nov 28, 2022 at 8:59 AM Ray Culbertson ***@***.***> wrote: This is an important note for any job which does resampling - that you want to avoid multiple re-reading over the network. I'm happy to help if needed. I wonder if all MDC resampling was done this way, or was it re-read over the network? If needed, please point me to to the current relevant cfg.

Most of our resampling campaigns are configured to run many jobs, but each job reads each input file approximately once per job, in which case I think using xrootd will be more efficient. Cosmic S2 jobs resample the same file many times and clearly would be more efficient copying the file, especially if run at NERSC. I don't think the current POMS scripts support that, I believe they are hard-coded to use xrootd (through the samListLocations interface). It would be good to make that configurable in the stage. Dave

…

— Reply to this email directly, view it on GitHub <#219 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABAH572RSSSV6ALAC2OXRWLWKTQGPANCNFSM6AAAAAASMXICOM> . You are receiving this because you commented.Message ID: ***@***.***>

-- David Brown ***@***.*** Office Phone (510) 486-7261 Lawrence Berkeley National Lab M/S 50R5008 (50-6026C) Berkeley, CA 94720

[sophiemiddleton]

As an update S1 + S2 are now working with S1 at NERSC and S2 at FNAL. Some issues remain in understanding the copying of files and moving S2 to NERSC eventually.