Whether to conduct data cheating in the experiment

[lzg1988]

1. sid and did use 0/* for training and you use 0/* +2/* for training?
2. sid and did use data config(0.1+0.04->10/30) and you use (0.1->10/30)?
   more training data and easier data complexity, is that fair????

[MohitLamba94]

Please note we use the same testing set up of SID i.e. (0.1+0.04->10/30) for benchmarking so it should be fine for we do not use the testing data for training. For training I merge the 0* and 2* both of which belong to training phase and none of it belongs to testing. As clearly noted in the ReadMe the scripts uploaded are only for demos since the patent is yet not granted. You will have to modify the training scripts for your project.

Hope this clarifies

[lzg1988]

1. testing issue

# from https://github.com/MohitLamba94/LLPackNet/blob/master/ablations/LLPackNet.ipynb
class get_data(Dataset):
    """Loads the Data."""
    
    def __init__(self,opt):
        self.train_files = glob.glob('/media/mohit/data/mohit/chen_dark_cvpr_18_dataset/Sony/short/1*_00_0.1s.ARW')
#         self.train_files = self.train_files + glob.glob('/media/mohit/data/mohit/chen_dark_cvpr_18_dataset/Sony/short/2*_00_0.1s.ARW')


        self.gt_files = []
        for x in self.train_files:
            self.gt_files =self.gt_files+ glob.glob('/media/mohit/data/mohit/chen_dark_cvpr_18_dataset/Sony/long/*'+x[-17:-12]+'*.ARW')
        
        self.to_tensor = torchvision.transforms.Compose([torchvision.transforms.ToTensor()])
        
        self.opt = opt

you used only 0.1s raw data for testing, why not use the below data for testing???

10003_00_0.04s.ARW  10030_00_0.04s.ARW  10045_00_0.04s.ARW  10178_00_0.04s.ARW  10193_00_0.04s.ARW  10217_00_0.04s.ARW
10006_00_0.04s.ARW  10032_00_0.04s.ARW  10054_00_0.04s.ARW  10185_00_0.04s.ARW  10198_00_0.04s.ARW  10226_00_0.04s.ARW
10011_00_0.04s.ARW  10034_00_0.04s.ARW  10055_00_0.04s.ARW  10187_00_0.04s.ARW  10199_00_0.04s.ARW  10227_00_0.04s.ARW
10016_00_0.04s.ARW  10035_00_0.04s.ARW  10068_00_0.04s.ARW  10191_00_0.04s.ARW  10203_00_0.04s.ARW  10228_00_0.04s.ARW
10022_00_0.04s.ARW  10040_00_0.04s.ARW  10069_00_0.04s.ARW  10192_00_0.04s.ARW  10213_00_0.04s.ARW
10178_00_0.033s.ARW  10187_00_0.033s.ARW  10192_00_0.033s.ARW  10198_00_0.033s.ARW  10203_00_0.033s.ARW  10217_00_0.033s.ARW  10227_00_0.033s.ARW
10185_00_0.033s.ARW  10191_00_0.033s.ARW  10193_00_0.033s.ARW  10199_00_0.033s.ARW  10213_00_0.033s.ARW  10226_00_0.033s.ARW  10228_00_0.033s.ARW

which was tested in sid and did.

# from https://github.com/parasmaharjan/DeepImageDenoising/blob/master/Pytorch_EDSR_Test.py#L120
in_files = glob.glob(input_dir + '%05d_00*.ARW' % test_id)
    for k in range(len(in_files)):

2. training issue

you said

For training I merge the 0* and 2* both of which belong to training phase

But for sid and did, they only use 0* data, totaly 161. BUT you use 181 data(more 20 data) for training?

# from https://github.com/cchen156/Learning-to-See-in-the-Dark/blob/master/train_Sony.py#L17
train_fns = glob.glob(gt_dir + '0*.ARW')
train_ids = [int(os.path.basename(train_fn)[0:5]) for train_fn in train_fns]

# from https://github.com/parasmaharjan/DeepImageDenoising/blob/master/Pytorch_EDSR.py#L39
train_fns = glob.glob(gt_dir + '0*.ARW')
train_ids = []
for i in range(len(train_fns)):
    _, train_fn = os.path.split(train_fns[i])
    train_ids.append(int(train_fn[0:5]))

Can you explain the above two issues?:)

[MohitLamba94]

yes, for my training I merged 0* and 2* because they are not used for testing. We are free to use validation set as we like during training phase. For testing ALL test images were used as mentioned in the SID project including 0.04 and 0.033.

For example

LLPackNet/inference.py

Line 29 in 90c33d8

self.paths_low = sorted(glob.glob(root+'short/1*_00_*.ARW'))

runs over all 1*_00* including 0.04 and 0.33

Feel free to expand this to include all 1*

Please understand, the files here are for DEMO only so that people can make quick experiments especially for those operating in RAM constrained environment. If you have large RAM feel free to edit the code and load more images, hope this helps.

[lzg1988]

thanks for your detailed reply. 👍

yes, for training, it "may" be ok to merge validation data set. But it's not allowed when strict.

Anyway, here are further discussions to figure out whether you work(LLPackNet BMCV 2020 and RED CVPR2021) are replicable.

Refer to Understanding Health Research, "It is very important that research can be replicated, because it means that other researchers can test the findings of the research. Replicability keeps researchers honest and can give readers confidence in research."

So

1. Can your provided model weights(LLPackNet and RED) reproduce your good results in paper
2. As you mentioned, you use 0* and 2* for training. So, follow your advice, if we use all(including 0.1s+0.033s + 0.04s) 0* and 2* data for training and your provided training code in LLPack github and RED, can we repruduce your work in papers LLPackNet BMCV 2020 and RED CVPR2021?

[MohitLamba94]

Hello,
My personal experience says that using just 0* or both 0*&2* does not make much of a difference, probably differences in 2nd decimal places.

But still, if you want, I will re-run the experiment with just 0* and see the metrics. But this will take some time as this is an old project, and I have shifted to another. Further, for your sake, for this experiment, I will only use the files uploaded to this GitHub repo except for the changes mentioned above. If time permits will repeat for LLPACKnet as well.

But I can almost say for sure that 0* or 0*&2* should not have much of a difference.

[lzg1988]

Thanks for your patience!! :)

Maybe there is no much difference between using 0*+2* and 0* for training.

However, the main issue is whether you use ONLY 0.1s data for training and testing. According to our experiments, we retrain sid using all data(0.1s+0.033s+0.04s), the psnr and ssim(28.89/0.787) is almost the same as their paper. But if we use 0.1s data ONLY, the psnr and ssim is much better(29. 5/0.814).

SO can your pretrained model reproduce your psnr and ssim using ALL testing data(0.1s+0.033s+0.04s) and is your model in your paper training and testing ALL data(0.1s+0.033s+0.04s) as SID or DID?

[lzg1988]

And we also notice other researcher also point out the same issue in RED. So we are also want to figure out whether your result is training and testing in all data.

MohitLamba94/Restoring-Extremely-Dark-Images-In-Real-Time#6

[lzg1988]

There is some strange thing in you work RED
MohitLamba94/Restoring-Extremely-Dark-Images-In-Real-Time@31248de
In 2022 May 16, you delete your detailed training path and testing paths(USE ONLY 0.1s data) and replace with some confusable '/path_to_SID_short_training'. We don't know what data you TRUELY train and test?