WPL #37

ROBOTICSENGINEER · 2021-10-28T23:01:29Z

Weibull Prototype Learning (WPL)

akshay-raj-dhamija · 2021-10-29T03:32:06Z

Since I already had a call about this work with Steve I had a vague idea of what this PR was about, but in the future, more information with details about the algorithm should be included with the PR.
I need some clarification here. Let's say we were training openmax for a dataset where we had 50 classes and each sample had 2048 dimensions. Then openmax model would contain 50 weibull models. Based on the current implementation WPL will create 50x2048 weibull models. Is this what is intended or do you mean to only create 2048 models (@tboult)?
I have made some suggestions for the fit_high function. I think it would be better if you incorporate these in the openmax.py and then just use the fit_high by calling from .openmax import fit_high. This should address these issues both for openmax and multimodal openmax too.
You need to import WPL inside vast/opensetAlgos/__init__.py
Looks like you haven't done a pip install of your clone of this repo. It is recommended you do that as it will install the git hooks that will correct a lot of inconsistent formatting for you.

vast/opensetAlgos/WPL.py

akshay-raj-dhamija · 2021-10-29T03:32:44Z

vast/opensetAlgos/WPL.py

+    WPL_params.add_argument(
+        "--default_scale",
+        nargs="+",
+        type=float,
+        default=[1.0],
+        help="Weibull scale to use when the number of uniqe element is less than 5 default: %(default)s",
+    )
+    WPL_params.add_argument(
+        "--default_shape",
+        nargs="+",
+        type=float,
+        default=[0.1],
+        help="Weibull shape to use when the number of uniqe element is less than 5 default:  %(default)s",
+    )


If these parameters are not supposed to be grid searched, they should not be a list rather floats. And if you are planning to grid search them, it should be noted that these do not impact the training in any way, since these are just the default values that need to be returned in a corner case. Possibly, it would be better if this was handled by whatever process is calling the inference function.

default_scale and default_shape are similar to tail_size, cover_threshold, and distance_multiplier of EVM. Please check the EVM.py in the main repo.

Algo Breakdown: Please try to simplify or break down your algorithm. Your WPL algo has two parts one is per dimension weibull and the other is how you address the case where weibull is needed for less than 5 samples/values. I am asking to separate this part from your core algorithm.
If tomorrow someone comes up with a new way to estimate default_scale and default_shape rather than fixing default values (let's say a mean from other class values, which can't be found until the algorithm has been run on the entire dataset) would you create a new WPL.py and call it a new algorithm even though the per dimension weibull idea is the same?

Reducing Computation: default_scale and default_shape are not similar to tail_size, cover_threshold, and distance_multiplier of EVM. These EVM parameters were being used as a list to reduce redundant computation. In your case, you will be recalling the mr.FitHigh function using the same parameters when your default_scale and default_shape change, even though they do not impact the result in this case. That is you are adding computational overhead rather than reducing it.
default_scale and default_shape are simply being used to replace scale and shape values where none could be found, not reduce computation.

Both of the comments Algo Breakdown and Reducing Computation are correct. @scruz13 please make required changes.

akshay-raj-dhamija · 2021-10-29T03:32:47Z

vast/opensetAlgos/WPL.py

+
+
+def fit_high(distances, distance_multiplier, tailsize, default_shape, default_scale):
+    distances = torch.unique(distances)


distances is always supposed to be a 2D tensor i.e.

no_of_samples x no_of_samples_to_which_distance_was_found

so the dim should be used when using unique
Also unique by default returns the sorted tensor, which might be redundant work taken care of in weibull.py

This seems to be trying to address a bigger problem, should it not be addressed in weibull.py?
@tboult: If we remove multiple occurrences of the same values, won't it result in different weibull models.

In the WPL, we have a prototype. i.e., no_of_samples_to_which_distance_was_found == 1 .

It is better to do not change weibull.py because it affects EVM and our papers

Even in openmax no_of_samples_to_which_distance_was_found=1 but still its fit_high function supports working on a 2D tensor absence of dim in your code above removes that support.

I will let you and @tboult decide if you want to incorporate this in weibull.py it is very much possible to incorporate this there without changing the default behaviour for any of the other algorithms. At least changing it there helps to quickly test the same idea for other algorithms as well.

Correction: I made an incorrect suggestion above, dim with unique doesn't do what I expected it to. So using unique in fit_high will by default remove the support for computation on a 2D tensor.

akshay-raj-dhamija · 2021-10-29T03:32:51Z

vast/opensetAlgos/WPL.py

+    tailsize = int(min(tailsize, distances.shape[1]))
+    mr = weibull.weibull()
+    if distances.shape[1] < 5:
+        pass


It might be better to simply create a weibull object where scale and shape tensors are nan and return it. The end-user can simply replace the nan values with the defaults you have in args.

args contains scale and shape of Weibull. So, why do not using it here? Why do you want to end-user replace it?

Please check my comment above #37 (comment)

vast/opensetAlgos/WPL.py

akshay-raj-dhamija · 2021-10-29T03:32:56Z

vast/opensetAlgos/WPL.py

+    with torch.no_grad():
+        for pos_cls_name in pos_classes_to_process:
+            features = features_all_classes[pos_cls_name].clone().to(f"cuda:{gpu}")
+            assert args.dimension == features.shape[1]


This is the first mention of dimension and it is not in your WPL_params function.
Also, why not just initialize it here and run for all dimensions by default, rather than leaving it as an argument?

akshay-raj-dhamija · 2021-10-29T03:32:58Z

vast/opensetAlgos/WPL.py

+            assert args.dimension == features.shape[1]
+
+            center = torch.mean(features, dim=0).to(f"cuda:{gpu}")
+            distances = torch.abs(features - center.view(1,args.dimension).repeat(features.shape[0], 1))


Looks like you are taking the L1 distance, please consider putting it in pairwisedistances.py and using it as was done in openmax.py, this ensures the same code being run in training and inference and also simplifies changing the distance computation by simply changing the argparse variable.

It is absolute value of feature from center (prototype). It cannot replace by other metric. For example, cosine distance of two scalar is not defined.

yes, you are right that this might not be replaceable by other metrics and neither would be a useful metric for other algorithms, which is what I was thinking.
BTW, you do not need .repeat above, even if needed it should be replaced with expand as it reduces memory consumption.

akshay-raj-dhamija · 2021-10-29T03:33:02Z

vast/opensetAlgos/WPL.py

+                args.tailsize, args.distance_multiplier, args.default_shape, args.default_scale
+            ):
+                  weibull_list = list()
+                  for k in args.dimension:


This for loop is not needed, you can just send the two-dimensional distance tensor into fit_high. This can reduce the compute time both during training and inference. This also means you will not need the weibull_list variable anymore.

I am confused! Does fit_high return a Weibull or list/tuple of many Weibulls?

You should also be able to use it to return multiple weibulls at once. If distances is a nxk tensor, where in your case n is the number of dimensions, then it can straightaway provide you n weibulls in a single mr object which will reduce both training and inference time.

vast/opensetAlgos/WPL.py

akshay-raj-dhamija · 2021-10-29T03:33:15Z

vast/opensetAlgos/WPL.py

+            distances = torch.abs(test_cls_feature - center.view(1,args.dimension).repeat(test_cls_feature.shape[0], 1))
+            weibull_list = models[class_name]["weibull_list"]
+            p = torch.empty(args.dimension)
+            for k in args.dimension:


As mentioned in the training function this for loop is avoidable.

Check comment above. #37 (comment)

akshay-raj-dhamija · 2021-10-29T18:19:41Z

vast/opensetAlgos/WPL.py

+        mr.sign = 1
+        mr.wbFits = torch.zeros(1, 2)
+        mr.wbFits[0, 1] = default_scale
+        mr.wbFits[0, 0] = default_shape
+        mr._ = torch.Tensor(0.0) # translate Amount
+        mr.smallScoreTensor =  torch.Tensor(0.0)#  small Score


You could just create this as done at

vast/vast/opensetAlgos/multimodal_openmax.py

Line 119 in 460480f

mr = weibull.weibull(

It might make it more readable

akshay-raj-dhamija · 2021-10-29T18:20:50Z

vast/opensetAlgos/WPL.py

+    dimension = None
+    for pos_cls_name in pos_classes_to_process:
+        features = features_all_classes[pos_cls_name].clone().to(f"cuda:{gpu}")
+        if dimension == None:
+            dimension = features.shape[1]
+         else:
+            assert dimension == features.shape[1]


dimension will always be None so will never reach the else statement.

akshay-raj-dhamija · 2021-10-29T18:23:18Z

vast/opensetAlgos/WPL.py

+        test_cls_feature = features_all_classes[batch_to_process].to(f"cuda:{gpu}")
+        assert test_cls_feature.shape[0] != 0
+        probs = []
+        for cls_no, cls_name in enumerate(models.keys())     


While you expect the models variable to be an ordered dict if a user instead passes a dict rather than letting the user know this will simply provide them with incorrect results. This can be avoided by simply using enumerate(sorted(models.keys())) instead of enumerate(models.keys())

ROBOTICSENGINEER added 3 commits October 28, 2021 14:22

Create WPL.py

27f95f7

adding Weibull prototype learning (WPL) algorithm

45b0e68

Update README.md

101fd4f

ROBOTICSENGINEER requested a review from akshay-raj-dhamija October 28, 2021 23:01

ROBOTICSENGINEER assigned scruz13 and RRabinow and unassigned scruz13 and RRabinow Oct 28, 2021

ROBOTICSENGINEER requested review from RRabinow, scruz13 and tboult October 28, 2021 23:03

akshay-raj-dhamija reviewed Oct 29, 2021

View reviewed changes

vast/opensetAlgos/WPL.py Outdated Show resolved Hide resolved

akshay-raj-dhamija reviewed Oct 29, 2021

View reviewed changes

vast/opensetAlgos/WPL.py Show resolved Hide resolved

akshay-raj-dhamija reviewed Oct 29, 2021

View reviewed changes

vast/opensetAlgos/WPL.py Outdated Show resolved Hide resolved

akshay-raj-dhamija reviewed Oct 29, 2021

View reviewed changes

vast/opensetAlgos/WPL.py Outdated Show resolved Hide resolved

akshay-raj-dhamija reviewed Oct 29, 2021

View reviewed changes

ROBOTICSENGINEER added 5 commits October 29, 2021 08:52

Adding the description to WPL

2b6cc7b

Update WPL.py

4a323c3

Update WPL.py

5b920e7

removing args.dimension from WPL

302744e

Update __init__.py

d1af419

akshay-raj-dhamija reviewed Oct 29, 2021

View reviewed changes



		def fit_high(distances, distance_multiplier, tailsize, default_shape, default_scale):
		distances = torch.unique(distances)

WPL #37

Are you sure you want to change the base?

WPL #37

Conversation

ROBOTICSENGINEER commented Oct 28, 2021

Uh oh!

akshay-raj-dhamija commented Oct 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

akshay-raj-dhamija Oct 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

akshay-raj-dhamija commented Oct 29, 2021 •

edited

Loading

akshay-raj-dhamija Oct 29, 2021 •

edited

Loading