Analyze and improve speed and memory consumption

[JanBenisek]

We had a use case at Argenta, where we worked with table of about 300 cols and ~2 mil. of rows.
There, the preprocessing took a lot of time and memory especially.

What we’d need is to find any dataset which is of similar size and is close to reality (mixture of categorical, flags and continuous variables, has missing) and see how much memory Cobra uses and how slow it is.

The issue occurs in preprocessor.fit() and preprocessor.transform() – but these guys do a lot behind, so I am trying to pinpoint the cause (is it the binning? Incidence replacement? Maybe the data types of intermediate tables are not efficient and it takes too much memory … ).

Once we find the cause, we can figure out how to fix it.

[MatthiasRoels]

I knew this issue would come up sooner or later. The problem is situated in CategoricalDataProcessor. More specifically in the part where you run chi-square tests. This takes a long time as you have to a test per category, per column (which takes a long time)...

One way to resolve this might be to add an alternative option to regroup small categories into a category other based on their size (in percentage, e.g. all categories with size less than 5% should be renamed to "other"). From experience, this approach would be more "aggressive" in the sense that you will end up with fewer categories but in my opinion this will not affect the model performance. The advantage is that we only have to do one computation per column.

[JanBenisek]

Thanks for mentioing that, it really helps! I will see what we can do.

I also saw that adding the column with split (train/selection/validation) is slow - in cases with bigger datasets, it takes too much memory because we are splitting and combining multiple large dataframes (easier would be just create a list and append). I sitill need to writ an issue for that.

[MatthiasRoels]

Thanks for mentioing that, it really helps! I will see what we can do.

I also saw that adding the column with split (train/selection/validation) is slow - in cases with bigger datasets, it takes too much memory because we are splitting and combining multiple large dataframes (easier would be just create a list and append). I sitill need to writ an issue for that.

No problem! And you are right, it could be that the train/selection/validation split becomes slow with bigger datasets. But then I wonder whether it is still feasible to use pandas (perhaps switching to Spark is the better option, but that really depends. You are the better judge here). Alternatively, we should indeed try to come up with something better, but then you loose the fact that you can really on scikit-learn...

As for the actual issue: CategoricalDataProcessor is always the bottleneck of the preprocessing part. Even with smaller datasets, it is slow so that should be the main focus here.

[JanBenisek]

I am just writing new split funciton (#53) because indeed when testing on large dataset, it is unnecessary slow. If you want, I tag you in the PR.

Yeah, there is a threshold when Python makes less sense. But average Cobra user will have some sort of ABT with maybe milion or two of rows and a few hundreds columns and then it should work in a reasonable amount time. I know we decided to rely on Pandas and sklearn and it was a good decision IMHO, but there is still some room for optimization.

CategoricalDataProcessor I am wondering how to speed this up - would paralelization work here? It might be overkill, but not sure where else there is. Or write your own chi-square test.

Additional consideration - we implemented Cobra in a way where we append the binned and encoded variables to the original one, which creates large table (plus with many object anf float64 data types). So I was wondering whether a) this should be reviewed and b) we preprocess columns which will not pass the univariate AUC selection, so we do not actually need them. However, I know that this would require larger refactoring.

[MatthiasRoels]

yes, you can add me as reviewer to that PR if you want to!

Regarding the CategoricalDataProcessor, parallelisation in pure Python is really hard! Implementing chi-square yourself could be an idea, but are we expecting to beat scipy in terms of performance there? The easiest would be to think of simpler alternatives to avoid chi-square I guess...

About your additional considerations: I think you are right and I thought about it really hard! The way I see it, you have three options:

1. Add additional columns to the original DataFrame, which will increase its size and memory consumption
2. Return a new DataFrame with only the processed columns, but then you have to stitch them all together at the end
3. Overwriting the original columns with processed ones (but then we have to make sure we copy the original DataFrame, otherwise we have to restart notebooks every time we want to try something different). So in the end we have two DataFrames which in my view will take up more memory than the first option (because you duplicate target column, index, ID column and some basic overhead that comes with the DataFrame class).

So as you can see, the first approach is in my view the better one in terms of memory consumption... And yes, we process much more columns than we use in the final model, but if you want a clear separation of the different pipeline steps, I don't see any alternative. Or do you have an idea?

[JanBenisek]

Agree with the implementation of CategoricalDataProcessor, this is an uphill battle we won't win.

I like 2 - just return processed (binned and incidence replacement) variables in a new dataframe. But then you have two large dataframes.

The issue (from the feedback I got) is mostly linked to memory - cobra is eating too much while processing the data. I'd like to check the internal memory usage. For example during the chi-square test, we create many dataframes, are those kept in memory or not?

Or the dataframe we return - does it have optimal data structures?

• Use categorical type instead of object.
• Every continuous variable is float64 which takes a lot of space, perhaps unnecessarily, and changing them to more appropriate ones helps).

[sandervh14]

The important commit here that should show up instead (forgot to mention "#27") is this one:
c787e8b.

It provides a method to create a big (2Mx300) housing price dataset to run the performance tests with.

I'm already pushing this so anyone could pick up this issue if I would die in a car crash tonight :-P, but I'll gladly have a look at the performance stuff mentioned in the posts above soon.

[sandervh14]

A lot of ideas were proposed. I'm summarizing them in descending order of feasibility and impact:

0. Solved (Improve speed of train/selection/validation split function #53): slow train/selection/validation split.
1. Slow CategoricalDataProcessor due to chi²:
   a. low-hanging fruit memory usage optimization: during the chi-square test, we create many dataframes, are those kept in memory or not? (del df_intermediary if no longer required) + optimal data structures: categorical > object and more appropriate column types than float64 for continuous vars (https://numpy.org/doc/stable/user/basics.types.html).
   b. Add additional columns to the original DataFrame, which will increase its size and memory consumption (but better than c. and d. in terms of memory consumption)
   c. Return a new DataFrame with only the processed columns, but then you have to stitch them all together at the end (preferred over b. and d. by Honza - "just return processed (binned and incidence replacement) variables in a new dataframe. But then you have two large dataframes.")
   d. Overwriting the original columns with processed ones (but then we have to make sure we copy the original DataFrame, otherwise we have to restart notebooks every time we want to try something different). So in the end we have two DataFrames which in my view will take up more memory than the first option (because you duplicate target column, index, ID column and some basic overhead that comes with the DataFrame class).
   e. regroup small categories into a category other based on their size (in percentage, e.g. all categories with size less than 5% should be renamed to "other") before computing chi² tests
   f. parallellization maybe, if not overkill.
   g. extra idea: offloading memory by writing some intermediary result dataframes to a cache folder on hard disk?
   h. Not planned: write chi² tests ourselves: unlikely faster than scipy.
2. Not currently planned: Speedup with spark: Cobra's spark branch - not the priority here. We focus on maximum use case that an average Cobra user would have, about 2Mx300cols, which should still be feasible with standard python (=no spark).)

I'll check 1a first.