pretraining without downstream task

[navidivan]

Hi, this is more like a question than an issue.

How could one do a pretraining run, without a downstream task?
What I want to achieve is to pretrain a model (using maybe both contrastive losses) and then use this model for various downstream tasks later.

Another question would be, when pretraining, do we pretrain on all train, validation, and test datasets? or should we limit it to training dataset?

[navidivan]

Also in the manuscript, it does not show an ablation study on the pretraining, right? My question is what is the effect of doing pretraining in general using the Same dataset?
When you are doing finetuning using pretrained models, do you again pretrain on the target data as well?

Could you give exact command line code for setting the model and training to repreduce the sex classification accuracy?

[Padraig20]

(not an author)

Hi @navidivan,

it is not a general practice to do self-supervised pre-training and supervised training on the same dataset. In the most cases in ML/DL, I would expect the model to overfit drastically.

Pre-training in general is mostly used to account for small datasets.

Let's take image classification, for instance. The internet offers a lot of unlabelled images, while labelled images can be rather scarce. The general idea behind pretraining is to somehow use the unlabelled data to help the model learn some general structure of the data. As an example, in Natural Language Processing that might be simple grammatical structures and understanding how the language works.

After pretraining, the model's weights are already reasonably well-tuned, giving it a general understanding of the data. By applying supervised learning with labeled data, we can fine-tune these weights, optimizing the model for a specific downstream task.

For the command line code, I found the sample_scripts very helpful. Maybe look into issue 1, where someone already prepared the right flags for using pretraining. Still, I would not advise you to use the same dataset for supervised training and self-supervised pretraining since it will most likely end in overfitting :)

Hope that helps tho!

[Padraig20]

Oh, and to your question:

Another question would be, when pretraining, do we pretrain on all train, validation, and test datasets? or should we limit it to training dataset?

NEVER do any kind of training on the validation/test dataset! This is a great risk for data leakage and will bias your model's results -> Example: If you study for an exam by just learning the exam answers by heart, it won't show your actual understanding of the topics when you are being tested ;)

Generally:

• train set: do any kind of supervised training
• validation set: usually for tuning the model
• test set: for assessing the model's actual performance

For pretraining, we usually use a separate (large) dataset! Here, the performance is usually only evaluated during the training process :)

[navidivan]

Thank you for your responses. It does make sense. But there I have seen papers (maybe because of lack of external data on fmri and the scanner differences, etc.) that have performed pretraining and self-supervised stuff on the same data. Here is one
https://arxiv.org/abs/2112.05761v1

I think this could be called pretraining or training or ... but it does makes sense to train parts of the model in different ways or use different losses at different stages. Anyway I thought it could be interesting to see the effect here as well.

I can pretrain a model, but I could not rerun from checkpoint (actually use the pretraining model)
######################################## Myflags:
TRAINER_ARGS='--accelerator gpu --max_epochs 20 --precision 16 --num_nodes 1 --devices 1'
MAIN_ARGS='--loggername tensorcsv --classifier_module v6 --dataset_name T1000 --image_path /*** --resume_ckpt_path ***/output/default/10/last.ckpt'
DATA_ARGS='--batch_size 8 --num_workers 8 --input_type rest'
DEFAULT_ARGS=''
OPTIONAL_ARGS='--c_multiplier 2 --last_layer_full_MSA True --clf_head_version v1 --downstream_task sex'
COMMAND="python ***/SwiFT/project/main.py $TRAINER_ARGS $MAIN_ARGS $DEFAULT_ARGS $DATA_ARGS $OPTIONAL_ARGS --dataset_split_num 1 --seed 1 --learning_rate 5e-6 --model swin4d_ver7 --depth 2 2 6 2 --embed_dim 36 --sequence_length 20 --first_window_size 4 4 4 4 --window_size 4 4 4 4 --img_size 96 96 96 20"
$COMMAND

######################################### Where the error come from:
if temp_args.resume_ckpt_path is not None:
# resume previous experiment
from module.utils.neptune_utils import get_prev_args
args = get_prev_args(args.resume_ckpt_path, args)
exp_id = args.id
# override max_epochs if you hope to prolong the training
args.project_name = project_name
args.max_epochs = max_epochs
args.num_nodes = num_nodes
args.devices = devices
args.image_path = image_path
else:
exp_id = None

does pretraining actually need Neptune?

utils file:

import os
import torch

def load_ckpt(exp_id, root_dir):
path = os.path.join(root_dir, exp_id, "last.ckpt")
return torch.load(path, map_location="cpu")

def get_prev_args(ckpt_path, args):
ckpt = torch.load(ckpt_path, map_location='cpu')
ignored_args_list = ["data_dir", "default_root_dir", "max_epochs", "resume_ckpt_path", "adjust_thresh"]
print(f"Warning: You have to specify the following arguments list when you are running the process: {ignored_args_list}")
for k, v in ckpt["hyper_parameters"].items():
if k in ignored_args_list:
continue
setattr(args, k, v)
return args

[Padraig20]

Hi,

thanks for forwarding the paper! But careful, they did not pre-train and fine-tune it on the same dataset! They pre-trained it on the HCP dataset and fine-tuned it further on the COBRE/CNP datasets.

Careful with the flags! --resume_ckpt_path is for resuming checkpoints, and not for using a pre-trained model. You can use --load_model_path for that and it should work fine. This function loads SwiFT's state_dict, see main.py at line 153 for your reference!

Neptune is not required, but is a great tool for logging. You could use tensorboard as an alternative for terminal logging, it is also implemented in the project. If you want to use any other logger, like WandB, you'll need to implement it yourself!

Hope this helped! :)