Conversation
d8ff896 to
cf9671e
Compare
tcatley
left a comment
There was a problem hiding this comment.
This does run and output the .csv files. However as @tobyallwood pointed out in #1290, there is now two folders generated for each file - one with the file name with the .spm extension (containing the tracing/filters/grains) and one without (containing the csv files). Ideally this would only be one folder per image
Closes #1290 Closes #1287 Because of problems that arise with the old format of Bruker files where the filenames have numerical extensions there was scope for data being over-written as `TopoStats.filename` uses the "stem", the part upto the file extension and so `file1.001`, `file1.002` and `file1.003` all write output to the same (nested) directory `file1`. To solve this we now include the file-extension in `TopoStats.filename`. However, this presents a problem when processing `.topostats` files as you can't have a directory and a file with the same name in the same location (manifested in the tests where we use `minicircle_small.topostats`). The solution is therefore to write output files (the `.topostats` version of every file) to the directory `<output_dir>/processed/topostats/<filename_without_extension>.topostats`. In doing so we also modify `io.save_topostats_file()` and drop the `filename` parameter as it can be obtained directly from the `topostats_object.filename` parameter passed in. We also introduce a step when saving files that don't already carry the `.topostats` extension (i.e. `.spm`, `.ibw`, `.006` etc.) to remove the last file extension from the filename and replace with `.topostats` for the target that is to be written to. This hopefully also addressed #1290 at the same time @tcatley if you are able to check with the `20240411_60ngTop2a_6ngpuc19_1mMATP_mgni.0_00024` file that uncovered the problem please that would be really useful and appreciated. Also - Removes some errant `print()` statements
cf9671e to
4c1fdf6
Compare
|
Thanks @tcatley I've now fixed that, took longer than I expected (as is often the case!). It stemmed from the disconnect between the code that writes all results out and the section that writes out per image statistics. I do wonder though, as I had this problem when working on #1284, what the utility of duplicating output is. The all statistics Basic data manipulation skills are invaluable and something all scientific researchers should have the in their skill set. An alternative if a single images output really is required is to just re-run TopoStats on just the single file. |
|
@ns-rse could it be possible the other set of files are left over from when we changed the names a few months back? Will take a look at the new code later am just out right now |
|
Agreed @ns-rse, I can't say I've ever actually used the per-image stats anyway - I would always just filter from the all_statistics.csv. If it is easier to just remove that output folder completely then maybe go with that |
|
@tobyallwood : No there is a specific function for writing them on a per image basis (it uses the @tcatley : Good to hear, thanks. I'll write up removal as an issue to be addressed. |
Closes #1290
Closes #1287
Because of problems that arise with the old format of Bruker files where the filenames have numerical extensions there
was scope for data being over-written as
TopoStats.filenameuses the "stem", the part upto the file extension and sofile1.001,file1.002andfile1.003all write output to the same (nested) directoryfile1. To solve this we nowinclude the file-extension in
TopoStats.filename.However, this presents a problem when processing
.topostatsfiles as you can't have a directory and a file with thesame name in the same location (manifested in the tests where we use
minicircle_small.topostats). The solution istherefore to write output files (the
.topostatsversion of every file) to the directory<output_dir>/processed/topostats/<filename_without_extension>.topostats.In doing so we also modify
io.save_topostats_file()and drop thefilenameparameter as it can be obtained directlyfrom the
topostats_object.filenameparameter passed in. We also introduce a step when saving files that don't alreadycarry the
.topostatsextension (i.e..spm,.ibw,.006etc.) to remove the last file extension from the filenameand replace with
.topostatsfor the target that is to be written to.This hopefully also addressed #1290 at the same time @tcatley if you are able to check with the
20240411_60ngTop2a_6ngpuc19_1mMATP_mgni.0_00024file that uncovered the problem please that would be really useful andappreciated.
Before submitting a Pull Request please check the following.