Integrate lp generation into snakemake workflow

[HauHe]

The generation of lp file can take quite some time for larger models. Hence it would be good to parallelize this part as well.
Do you think this would be very complicated @trevorb1?
I think in the original scripts I had it within the script called by snakemake.

[trevorb1]

Hey @HauHe! I think your original snakefile only included the preprocessing and solving? I do agree though, parallelizing the lp creation would be good. The only issue is that since we try to handle LPs that fail to build, it can be difficult because snakemake will flag them as missing input files for the solve step.

Maybe using the multiprocessing library would be a better option. But Ive never used it so I dont really know

[HauHe]

Well or we do it how I did it for solving in the old scripts and put the lo-generation into a python script that handles it and provides snakemake with an output file.

[HauHe]

@trevorb1, I'm tempted to just do this as I had it in the original scrips, i.e., have a python script handling the lp generation and call it from snakemake

[trevorb1]

@HauHe, that could be a good solution!

A couple of thoughts if this is implement though:

• Right now, if the LP generation fails, we remove the model run from the current step and all subsequent steps. I think this may be a little tricky to implement in snakemake? (Although, you have done it before, so I trust your experience!)
• As you mentioned we can create an LP file, regardless of if it completed successfully or not, and use the --keep-going flag to deal with failed solves. However, we will also need to re-implement the logic to track which LPs failed and remove all downstream steps between snakemake and passing residual capacity.
• Alternatively, we can use input functions probably to check if the LP is generated correctly or not, and define logic in Snakemake on what to do in each case. Im not sure what would be easier off the top of my head.
• Also, it seems like we are going towards a point of just snakemake'ing everything? I know that that the LP generation and solving are the big components, but maybe this update deserves further discussion around how to just snakemake all logic for each step?