Rework file publishing

[tcrombie]

This is a vague suggestion at this point, but I want to have it here on record. Throughput the pipeline there is are places where files are published outside of the work directory that do not need to be. I believe we can make the pipeline more efficient by limiting the file publishing and restrict more of the output files to the work directory. This would help with --resume and caching options too b/c currently the pipeline will purge directories in the output folder if it detects them to avoid overwriting files.