added resnets that can be initialized with v1 weights. #2
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
MuawizChaudhary
requested review from
kamdh and
vivianwhite
and removed request for
kamdh and
vivianwhite
kamdh
reviewed
Dec 16, 2023
kamdh
requested changes
Dec 16, 2023
Contributor
kamdh
left a comment
kamdh
left a comment
There was a problem hiding this comment.
Comments on resnet implementation
V1-models/LC_models.py
Outdated
| for c, m in enumerate(self.resnet.modules()): | ||
| if isinstance(m, LearnableCov.FactConv2d): | ||
| scale = m.in_channels*m.kernel_size[0]*m.kernel_size[1] | ||
| center = (.3, .3) |
Contributor
There was a problem hiding this comment.
center should be (int(kernel_size[0]/2), int(kernel_size[1]/2))
Collaborator
Author
There was a problem hiding this comment.
int? but here its .3
| kernel_size=module.kernel_size, | ||
| stride=module.stride, padding=module.padding, | ||
| bias=module.bias) | ||
| setattr(model, n, new_module) |
Collaborator
Author
There was a problem hiding this comment.
Whats your definition of safe?
I've used this code for switching out batchnorm layers in resnets with another type of normalization layer (this was for my master's thesis work on test-time-adaptation.).
Also, the printed resnet gives the correct structure with Conv2d replaced with FactConv2d
kamdh
requested changes
Dec 16, 2023
Contributor
kamdh
left a comment
kamdh
left a comment
There was a problem hiding this comment.
Please move scripts into subdirectory under base directory rather than within V1-models
Collaborator
|
The arguments/combinations in the script for results look good to me! |
added resnet that can be initalized with v1 weights which are learnable and can be set to non learnable added scripts for launching experiments on the Mila cluster Added small sample, Cifar100, fixed v1 init commiting code for most recent set of experiments Rebutal experiments. fixes update committing my set of experiments changed alot of stuff in the conv modules in this commit. i need to come back and clean up. added changes tohooks. may need to revert back to using item. this bit takes up alot of time inrecording rms/gradient rms added all.sh files. need to organize a bit better. added change to freeze the 1x1 module and zero it. added kangliu codebase we're using for our experiments. also added output files. we need to organize this viv visualizing cov stuff initial commit for visualizing covariance script added functions for histograms and matrices vivian should be able to run these experiments and carry us to victory due to her larger fairshare adding subplots for spatial and channel plots added exp_diag and fixed eigenvector plots scripts for param scale test fixed mistakes for scale scripts added fixed parametric scattering included og stuff currently refactoring this. sending code to vivian added rainbow sampling scripts initial commit training and saving udpates fixed logging small changes commented out allow tf32 and fixed logger fixed trial dir adding width scale for resnets adding K Fact nets fixed replace_layers_K function implementing trunc normal adding 3 layer networks no affines resnets updated for loop in forward fixed typo adding icml folder adding todos add affine no bias no v1 code for kam to tweak code for kam to tweak og covariance rainbow sampling sequential model fixed sequential changes SOTA? currently running affines-no-bias weird non-functioning version fixed copy-pasted weights fixed typo affine study initial commit fixed typo i align the input with a forwards pre hook moved and renamed linear fact updating code to match fact conv fixed tri_vec_to_mat function imagenet setup removed reset_params adding imagenet script scripts for running imagenet script for width 8 run modifying save function take in resnets of different widths updates for different-width resnets Files for Debugging working on 2 gpus :D adding all scripts launched on narval fixed args adding linear module file cedar script added some cool stuff making sure this stuff is working... width 8 scripts updated imagenet more efficent way of calculating quantities debugged this. interesting note; aliging with the weight+learnable_cov performance ok! 85% for width 4. not the 89 we were getting when aligning the unlearnt random weight muawiz updates changes updates from narval removing old code, refactoring imagenet final boss time imagenet for vivian imagenet rainbowing for vivian starting refactor removed comments removing useless lines refactor and annotate. works for both conv and fact. @obilaniu why doesnt this speed up my code huh. a100 (40GB) renaming re-implementing their rainbow reimplemetning their rainbow sampling as presented in the notebook aca comments added, handle renamed moving stuff to functions refactor most of the way there. i just have to figure out some extrenous stuff about fitting their rainbow sampling stuff into ours and then im done refactored guth et als. colored covariance fitting into our stuff changes. includes some debug statements debugging weight_alignmnet fixed bug kam integration last commit code upload if wanted for verifcation purposes nicer weight alignment in kam stuff nicer weight alignment in kam stuff kam tweaks to alignment functions timing tests for 3060 gpus fancier forward in FactConv2dPreExp changes to time testing script Vivian pls run vivian pls run on 3060s timing stuff removed possible cuda synchronization point ( printing progress ) changes twin studies corrected twins singleton scripts initial commit of colab script added arguments for comparisons and hooks and stored the final array trying out patch
MuawizChaudhary
force-pushed
the
muawi_resnets
branch
from
fbd52c3 to
f43879b
Compare
Contributor
|
Close or merge? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
These v1 weights are learnable and can be set to non learnable. I use the resnets from a popular pytorch cifar codebase. These aren't in exact correspondence with the resnets used in Kaiming He's OG Resnet paper, nor with with the Resnets used by some models on RobustBench (at least those models used in the test time adaptation setting [1, 2], which are Resnet-26 and Resnet-20 respectively).
Currently I am focusing on creating some bash scripts for launching the experiments we wanna run on the Mila cluster.
From the email:
When I push the scripts it'd be nice to know if I got the correct arguments.
EDIT: I have pushed the scripts. please let me know that I got the arguments/combinations we want results for correct!