Error when running the training

[SRMLabHainu]

python train.py --datasets_folder noisy --n_epochs 5 --GPU 0 --train_datasets_size 60 --patch_x 80 --patch_t 80

A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.0.2 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.

Traceback (most recent call last): File "F:\Denoising_and_super-resolution\SRDTrans-main\SRDTrans-main\train.py", line 45, in
from SRDTrans import SRDTrans
File "F:\Denoising_and_super-resolution\SRDTrans-main\SRDTrans-main\SRDTrans_init_.py", line 4, in
from SRDTrans.SpatioTemporalTrans import SpatioTemporalTrans
File "F:\Denoising_and_super-resolution\SRDTrans-main\SRDTrans-main\SRDTrans\SpatioTemporalTrans_init_.py", line 5, in
from SRDTrans.SpatioTemporalTrans.SpatioiTrans import SpatioTransLayer
File "F:\Denoising_and_super-resolution\SRDTrans-main\SRDTrans-main\SRDTrans\SpatioTemporalTrans\SpatioiTrans.py", line 7, in
from timm.models.layers import DropPath, to_2tuple, trunc_normal_
File "D:\ProgramData\Anaconda3\envs\SRDTrans\lib\site-packages\timm_init_.py", line 2, in
from .layers import (
File "D:\ProgramData\Anaconda3\envs\SRDTrans\lib\site-packages\timm\layers_init_.py", line 1, in
from .fx import (
File "D:\ProgramData\Anaconda3\envs\SRDTrans\lib\site-packages\timm\layers_fx.py", line 8, in
from torchvision.models.feature_extraction import create_feature_extractor as create_feature_extractor
File "D:\ProgramData\Anaconda3\envs\SRDTrans\lib\site-packages\torchvision_init.py", line 5, in
from torchvision import datasets, io, models, ops, transforms, utils
File "D:\ProgramData\Anaconda3\envs\SRDTrans\lib\site-packages\torchvision\models_init.py", line 17, in
from . import detection, optical_flow, quantization, segmentation, video
File "D:\ProgramData\Anaconda3\envs\SRDTrans\lib\site-packages\torchvision\models\detection_init_.py", line 1, in
from .faster_rcnn import *
File "D:\ProgramData\Anaconda3\envs\SRDTrans\lib\site-packages\torchvision\models\detection\faster_rcnn.py", line 16, in
from .anchor_utils import AnchorGenerator
File "D:\ProgramData\Anaconda3\envs\SRDTrans\lib\site-packages\torchvision\models\detection\anchor_utils.py", line 10, in
class AnchorGenerator(nn.Module):
File "D:\ProgramData\Anaconda3\envs\SRDTrans\lib\site-packages\torchvision\models\detection\anchor_utils.py", line 63, in AnchorGenerator
device: torch.device = torch.device("cpu"),
D:\ProgramData\Anaconda3\envs\SRDTrans\lib\site-packages\torchvision\models\detection\anchor_utils.py:63: UserWarning: Failed to initialize NumPy: ARRAY_API not found (Triggered internally at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\torch\csrc\utils\tensor_numpy.cpp:77.)
device: torch.device = torch.device("cpu"),
D:\ProgramData\Anaconda3\envs\SRDTrans\lib\site-packages\timm\models\layers_init.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
warnings.warn(f"Importing from {name} is deprecated, please import via timm.layers", FutureWarning)
Training parameters ----->
Namespace(n_epochs=5, GPU='0', patch_x=80, patch_t=80, overlap_factor=0.5, train_datasets_size=60, datasets_path='datasets', pth_path='./pth', datasets_folder='noisy', output_path='./results', lr=0.0001, b1=0.5, b2=0.999, select_img_num=10000000000, test_datasize=10000000000, scale_factor=1, patch_y=80, gap_x=40, gap_y=40, gap_t=40, ngpu=1, batch_size=1)
ckp is saved in pth\noisy_202601290920
Image list for training ----->
All files are in -----> datasets\noisy
Total stack number -----> 7
Reading files...
Please check the shape of these image stacks, since some hyperstacks have unusual shapes. In that case, you just need to re-store these images by ImageJ.
clean_200Hz_2400frames_pxlsize30nm.tif -----> the shape is (24000, 328, 328)
noise_200Hz_2400frames_pxlsize30nm_-0.05dBSNR_24000x328x328.tif -----> the shape is (24000, 328, 328)
noise_200Hz_2400frames_pxlsize30nm_12.04dBSNR_24000x328x328.tif -----> the shape is (24000, 328, 328)
noise_200Hz_2400frames_pxlsize30nm_14.14dBSNR_24000x328x328.tif -----> the shape is (24000, 328, 328)
noise_200Hz_2400frames_pxlsize30nm_3.90dBSNR_24000x328x328.tif -----> the shape is (24000, 328, 328)
noise_200Hz_2400frames_pxlsize30nm_8.27dBSNR_24000x328x328.tif -----> the shape is (24000, 328, 328)
Traceback (most recent call last):
File "D:\ProgramData\Anaconda3\envs\SRDTrans\lib\site-packages\tifffile\tifffile.py", line 4281, in init
byteorder = {b'II': '<', b'MM': '>', b'EP': '<'}[header[:2]]
KeyError: b''

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "F:\Denoising_and_super-resolution\SRDTrans-main\SRDTrans-main\train.py", line 73, in
train_name_list, train_noise_img, train_coordinate_list, stack_index = train_preprocess_lessMemoryMulStacks(opt)
File "F:\Denoising_and_super-resolution\SRDTrans-main\SRDTrans-main\data_process.py", line 213, in train_preprocess_lessMemoryMulStacks
noise_im = tiff.imread(im_dir)
File "D:\ProgramData\Anaconda3\envs\SRDTrans\lib\site-packages\tifffile\tifffile.py", line 1245, in imread
with TiffFile(
File "D:\ProgramData\Anaconda3\envs\SRDTrans\lib\site-packages\tifffile\tifffile.py", line 4283, in init
raise TiffFileError(f'not a TIFF file {header!r}') from exc
tifffile.tifffile.TiffFileError: not a TIFF file b''
dear sir
When I used SMLM data and train.py to train the model, the above error occurred. May I ask how to solve this error? Or is that why I used SMLM data, but the shape of the data is (24000,328,328)? Is it because I set the parameters incorrectly?

[cabooster]

Hi, thanks for your question and for providing the detailed error log. Based on the traceback, this issue is not caused by the SMLM data shape (24000 × 328 × 328), and it is also not due to incorrect training parameters. This data shape is valid for SRDTrans as long as the input files are standard 3D TIFF stacks.

The real reason the training crashes is the following error: tifffile.TiffFileError: not a TIFF file b''. This indicates that tifffile.imread() attempted to read a file whose header is empty. In practice, this almost always means that at least one file in the datasets/noisy directory is not a valid TIFF file. Common causes include a corrupted or incomplete TIFF file, a zero-byte file, or a file that has a .tif extension but is not actually in standard TIFF format.

From the log, the program reports a total of 7 stacks, but only 6 files are successfully read and printed before the crash occurs. This strongly suggests that the last file being read is invalid and causes the failure.

To resolve this issue, please check all .tif/.tiff files in the datasets/noisy folder and make sure none of them are empty or corrupted. Try opening each file in ImageJ/Fiji. If a file cannot be opened, it needs to be re-exported or copied again from the original source. If it can be opened, it is recommended to re-save it using “File → Save As → TIFF” in ImageJ to ensure it is stored in a standard TIFF format. Also make sure that the dataset directory contains only TIFF files, with no subfolders or other file types that could be mistakenly read by the script.

In summary, this is a data integrity or file format issue rather than a limitation of SRDTrans or a problem with parameter settings.

[SRMLabHainu]

(SRDTrans) PS F:\Denoising_and_super-resolution\SRDTrans-main\SRDTrans-main> python train.py --datasets_folder noisy --n_epochs 5 --GPU 0 --train_datasets_size 60 --patch_x 80 --patch_t 80

A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.0.2 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.

Traceback (most recent call last): File "F:\Denoising_and_super-resolution\SRDTrans-main\SRDTrans-main\train.py", line 45, in
from SRDTrans import SRDTrans
File "F:\Denoising_and_super-resolution\SRDTrans-main\SRDTrans-main\SRDTrans_init_.py", line 4, in
from SRDTrans.SpatioTemporalTrans import SpatioTemporalTrans
File "F:\Denoising_and_super-resolution\SRDTrans-main\SRDTrans-main\SRDTrans\SpatioTemporalTrans_init_.py", line 5, in
from SRDTrans.SpatioTemporalTrans.SpatioiTrans import SpatioTransLayer
File "F:\Denoising_and_super-resolution\SRDTrans-main\SRDTrans-main\SRDTrans\SpatioTemporalTrans\SpatioiTrans.py", line 7, in
from timm.models.layers import DropPath, to_2tuple, trunc_normal_
File "D:\ProgramData\Anaconda3\envs\SRDTrans\lib\site-packages\timm_init_.py", line 2, in
from .layers import (
File "D:\ProgramData\Anaconda3\envs\SRDTrans\lib\site-packages\timm\layers_init_.py", line 1, in
from .fx import (
File "D:\ProgramData\Anaconda3\envs\SRDTrans\lib\site-packages\timm\layers_fx.py", line 8, in
from torchvision.models.feature_extraction import create_feature_extractor as create_feature_extractor
File "D:\ProgramData\Anaconda3\envs\SRDTrans\lib\site-packages\torchvision_init.py", line 5, in
from torchvision import datasets, io, models, ops, transforms, utils
File "D:\ProgramData\Anaconda3\envs\SRDTrans\lib\site-packages\torchvision\models_init.py", line 17, in
from . import detection, optical_flow, quantization, segmentation, video
File "D:\ProgramData\Anaconda3\envs\SRDTrans\lib\site-packages\torchvision\models\detection_init_.py", line 1, in
from .faster_rcnn import *
File "D:\ProgramData\Anaconda3\envs\SRDTrans\lib\site-packages\torchvision\models\detection\faster_rcnn.py", line 16, in
from .anchor_utils import AnchorGenerator
File "D:\ProgramData\Anaconda3\envs\SRDTrans\lib\site-packages\torchvision\models\detection\anchor_utils.py", line 10, in
class AnchorGenerator(nn.Module):
File "D:\ProgramData\Anaconda3\envs\SRDTrans\lib\site-packages\torchvision\models\detection\anchor_utils.py", line 63, in AnchorGenerator
device: torch.device = torch.device("cpu"),
D:\ProgramData\Anaconda3\envs\SRDTrans\lib\site-packages\torchvision\models\detection\anchor_utils.py:63: UserWarning: Failed to initialize NumPy: ARRAY_API not found (Triggered internally at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\torch\csrc\utils\tensor_numpy.cpp:77.)
device: torch.device = torch.device("cpu"),
D:\ProgramData\Anaconda3\envs\SRDTrans\lib\site-packages\timm\models\layers_init.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
warnings.warn(f"Importing from {name} is deprecated, please import via timm.layers", FutureWarning)
Training parameters ----->
Namespace(n_epochs=5, GPU='0', patch_x=80, patch_t=80, overlap_factor=0.5, train_datasets_size=60, datasets_path='datasets', pth_path='./pth', datasets_folder='noisy', output_path='./results', lr=0.0001, b1=0.5, b2=0.999, select_img_num=10000000000, test_datasize=10000000000, scale_factor=1, patch_y=80, gap_x=40, gap_y=40, gap_t=40, ngpu=1, batch_size=1)
ckp is saved in pth\noisy_202602021659
Image list for training ----->
All files are in -----> datasets\noisy
Total stack number -----> 7
Reading files...
Please check the shape of these image stacks, since some hyperstacks have unusual shapes. In that case, you just need to re-store these images by ImageJ.
clean_200Hz_2400frames_pxlsize30nm.tif -----> the shape is (24000, 328, 328)
noise_200Hz_2400frames_pxlsize30nm_-0.05dBSNR_24000x328x328.tif -----> the shape is (24000, 328, 328)
noise_200Hz_2400frames_pxlsize30nm_12.04dBSNR_24000x328x328.tif -----> the shape is (24000, 328, 328)
noise_200Hz_2400frames_pxlsize30nm_14.14dBSNR_24000x328x328.tif -----> the shape is (24000, 328, 328)
noise_200Hz_2400frames_pxlsize30nm_3.90dBSNR_24000x328x328.tif -----> the shape is (24000, 328, 328)
Traceback (most recent call last):
File "F:\Denoising_and_super-resolution\SRDTrans-main\SRDTrans-main\train.py", line 73, in
train_name_list, train_noise_img, train_coordinate_list, stack_index = train_preprocess_lessMemoryMulStacks(opt)
File "F:\Denoising_and_super-resolution\SRDTrans-main\SRDTrans-main\data_process.py", line 226, in train_preprocess_lessMemoryMulStacks
noise_im = noise_im.astype(np.float32) / args.scale_factor # no preprocessing
numpy._core._exceptions._ArrayMemoryError: Unable to allocate 9.62 GiB for an array with shape (24000, 328, 328) and data type float32
Dear sir, i follow your answer and restore the files. but it occurs new errors about memory. May I ask if the running of this code requires a high amount of computer memory? My computer is equipped with a 4070Ti graphics card, 16GB video memory, and 32GB memory module. In addition, the data I saved separately is only 4.81GB per tif file, not 5.3GB. May I ask how to solve it? Looking forward to the author's response