`overwrite` and `redo` options are underperforming

[wwieder]

ADF run type

Model vs. Model

What happened?

There seem to be a number of times with overwrite = false or redo = false seems to be ignored (or not used efficiently) in the clm_diags branch. Is this something that's been fixed on main, or can they be used more effectively to speed up debugging?

ADF Hash you are using

v0.11-576-gb46ed6f2

What machine were you running the ADF on?

CGD machine, CISL machine

What python environment were you using?

ADF-provided Conda env

Extra info

I'll fill out more when I'm less frusterated. Namely it seems like time series are always being overwritten, even when they already exist, but plotting features take longer than expected with they already exist?

[wwieder]

Happy to help on this for a bit tomorrow if there's energy.

[brianpm]

Hi @wwieder -- I tried to fix at least some of the redo problems in #358. I think that was for global maps and polar plots; if redo isn't working for other plots, I can try to take a look. This approach basically loops through everything to set up the plotting -- like checking paths and whether files exist and collecting all the options into a dictionary -- and then uses the gathered information to make plots. I think this is way potentially much more efficient than the previous implementation; I'm not sure my code is as clean as it could be, but it seems to be working (and redo definitely is working better now).

I haven't looked at the overwrite options, I thought those were working fine.

[wwieder]

Thanks @brianpm! I think we just need to merge these onto the clm-diags branch. This will greatly improve my quality of life