BackwardStepwiseOLS as Intermediary Pipeline Step

[dennislabajo]

Hello,

Can you please advise how I could incorporate BackwardStepwiseOLS into a pipeline as an intermediary step to a classifier and then doing RandomizedSearchCV? What I'm hoping to do is retrofit BackwardStepwiseOLS so it only returns the features of X whose p-value is within a configurable value of .05. Using other techniques e.g. PCA, LDA, or KPCA works as an intermediary step but I'm just curious how a model performs if the dimensions were reduced instead by a function such BackwardStepwiseOLS.

Here's a snippet of what I'm trying to do:

steps=[
	('preprocessor', preprocessor), 
        ('stepwise_ols',  BackwardStepwiseOLS(fK=10)), 
        ('classifier', RandomForestClassifier())]

pipeline = Pipeline(steps=steps)

rs = RandomizedSearchCV(estimator=pipeline, param_distributions=my_hyper_parameter,
    scoring='accuracy', return_train_score=True)

rs.fit(X, y)

Thanks.

[xinhe97]

Hi,

I am excited to see you are interested in my project. This is the first Issue I've ever had on my original repository.

The Pipeline steps proposed in your comments are not practical, because the intermediate steps in pipelines should be transformers, such as PCA, LDA, usually unsupervised; the counterpart should be, I name it, predictors, such as OLS, RandomForestClassifier.

I have three examples, the point is that sklearn models LinearRegression and RandomForestClassifier cannot be an intermediate step in pipelines, neither the StepwiseOLS.

Codes:

from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LinearRegression
steps=[
        ('ols', LinearRegression()),
       ('classifier', RandomForestClassifier())]
pipeline = Pipeline(steps=steps)

Error Messages:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Applications/anaconda3/lib/python3.7/site-packages/sklearn/pipeline.py", line 135, in __init__
    self._validate_steps()
  File "/Applications/anaconda3/lib/python3.7/site-packages/sklearn/pipeline.py", line 185, in _validate_steps
    "'%s' (type %s) doesn't" % (t, type(t)))
TypeError: All intermediate steps should be transformers and implement fit and transform or be the string 'passthrough' 'LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None, normalize=False)' (type <class 'sklearn.linear_model.base.LinearRegression'>) doesn't

Codes:

steps=[
        ('bwd', BackwardStepwiseOLS()),
       ('classifier', RandomForestClassifier())]
pipeline = Pipeline(steps=steps)

Error Messages:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Applications/anaconda3/lib/python3.7/site-packages/sklearn/pipeline.py", line 135, in __init__
    self._validate_steps()
  File "/Applications/anaconda3/lib/python3.7/site-packages/sklearn/pipeline.py", line 185, in _validate_steps
    "'%s' (type %s) doesn't" % (t, type(t)))
TypeError: All intermediate steps should be transformers and implement fit and transform or be the string 'passthrough' 'BackwardStepwiseOLS(fK=3)' (type <class 'BackwardStepwiseOLS.BackwardStepwiseOLS'>) doesn't

Codes:

steps=[
       ('classifier', RandomForestClassifier()),
       ('ols', LinearRegression()),
]
pipeline = Pipeline(steps=steps)

Error Messages:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Applications/anaconda3/lib/python3.7/site-packages/sklearn/pipeline.py", line 135, in __init__
    self._validate_steps()
  File "/Applications/anaconda3/lib/python3.7/site-packages/sklearn/pipeline.py", line 185, in _validate_steps
    "'%s' (type %s) doesn't" % (t, type(t)))
TypeError: All intermediate steps should be transformers and implement fit and transform or be the string 'passthrough' 'RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',
                       max_depth=None, max_features='auto', max_leaf_nodes=None,
                       min_impurity_decrease=0.0, min_impurity_split=None,
                       min_samples_leaf=1, min_samples_split=2,
                       min_weight_fraction_leaf=0.0, n_estimators='warn',
                       n_jobs=None, oob_score=False, random_state=None,
                       verbose=0, warm_start=False)' (type <class 'sklearn.ensemble.forest.RandomForestClassifier'>) doesn't

The StepwiseOLS is supervised method. If you put it as an intermediate step in a pipeline, there is no Y to guide the algorithm. That's why intermediate steps are usually unsupervised.

Welcome more comments.

[dennislabajo]

Hi @xinhe97,

Thanks for the information and I apologize for the delayed response, multiple projects got in the way and was extremely busy.

Let me add more details on what I'm trying to achieve.

###################### Pipeline definition
# Standard Scaler for numeric feaures
num_transformer = Pipeline(steps=[
    ('scaler', StandardScaler())
])

# Categorical transformers
cat_transformer = Pipeline(steps=[
    ('onehot', OneHotEncoder())
])

# ColumnTransformer to apply a Pipeline to features selectively
preprocessor = ColumnTransformer(
    transformers=[
        ('cat', cat_transformer, cat_attribs), 
        ('num', num_transformer, num_attribs_all)
    ])

preprocessor.fit(df.iloc[:,:-1])
label_encoder = LabelEncoder().fit(df.class_)



###################### List of dimensionality reduction techniques to use
dimensionality_reduction = {
    'pca': PCA(n_components=8), 
    'lda': LDA(n_components=5),
    'backward_elimination': Backward_Elimination_Transformer(p_value, random_state=random_state)
}


###################### Classifiers to evaluate
classifiers = {
    'RandomForestClassifier': RandomForestClassifier(random_state=random_state),
    'GradientBoostingClassifier': GradientBoostingClassifier(random_state=random_state),
    'KNeighborsClassifier': KNeighborsClassifier(),
    'SVC': SVC()
}


######################  My implementation of backward elimination
class Backward_Elimination_Transformer(BaseEstimator):
    '''
    Wrapper class for performing backward elimination.
    '''
    def __init__(self, p_value=0.01, random_state=0): # no *args or **kargs
        self.p_value = p_value
        self.random_state = random_state

    def fit(self, X, y=None):
        self.y = y
        return self

    def transform(self, X=None, y=None):
        numVars = len(X[0])
        for i in range(0, numVars):
            obj_OLS = sm.OLS(self.y, X, random_state=self.random_state).fit()
            pvalues = obj_OLS.pvalues.astype(float)
            maxVar = max(pvalues).astype(float)
            if (maxVar > self.p_value):
                for j in range(0, numVars - i):
                    if (obj_OLS.pvalues[j].astype(float) == maxVar):
                        X = np.delete(X, j, 1)
                        break
        return X


######################  Run multiple models and evaluate results
for d_reduction in dimensionality_reduction:
    
    for classifier in classifiers.keys():
        
        print('Running {}, {}'.format(d_reduction, classifier))
        
        steps=[
            ('preprocessor', preprocessor), 
            ('dimensionality_reduction',  dimensionality_reduction[d_reduction]), 
            (classifier, classifiers[classifier])]
    
        pipeline = Pipeline(steps=steps)
            
        # Load the hyper-parameters for each classifier
        hyper_parameter = hyper_parameters[classifier]
            
        # Define the randomized search instance passing in the pipeline and parameters
        rs = RandomizedSearchCV(estimator=pipeline, 
                            param_distributions=hyper_parameter,
                            cv=cv,
                            scoring='accuracy', return_train_score=True,
                            n_iter=n_iter, n_jobs=-1)

        # Fit the randomized search using training data for each user
        rs_fit = rs.fit(X, y)

The code above runs perfectly where all the classifiers can run for each of the dimensionality reduction technique (PCA, LDA) but fails when the loop reaches backward_elimination.

The error I'm getting is: ValueError: endog and exog matrices are different sizes

To get around this error, I have to create a separate code just for handling backward elimination where I have to break apart the pipeline steps which is not elegant but it works. Below is what I did:

    # Pipeline 1 for backward elimination; includes column transformation
    backward_elimination_steps=[
        ('preprocessor', preprocessor), 
        ('dimensionality_reduction',  dimensionality_reduction[d_reduction]), 
    ]

    # Execute Pipeline 1
    backward_elimination_pipeline = Pipeline(steps=backward_elimination_steps)
    X_ = backward_elimination_pipeline.fit_transform(X, y)
    
    # Pipeline 2 for applying the classifier
    steps=[
        (classifier, classifiers[classifier])
    ]
    
    pipeline = Pipeline(steps=steps)
    hyper_parameter = hyper_parameters[classifier]
    
    # Run randomized search using pipeline #2 as the estimator
    rs = RandomizedSearchCV(estimator=pipeline, 
                            param_distributions=hyper_parameter,
                            cv=cv, 
                            scoring='accuracy', return_train_score=True, 
                            n_iter=n_iter, n_jobs=-1, random_state=random_state)

    rs_fit = rs.fit(X_, y)    # NOTE:  X_ is output from Pipeline 1

So this approach runs just fine but again, its just not elegant and not what I would really want to do. That's why I was searching for other ways to do it and that's when I came across your project. Ideally, I would like to use just one pipeline below without breaking it apart into two just to make backward elimination work. So my objective is to be able to run just one pipeline like below and should be able to work regardless if PCA, LDA, or Backward Elimination.

steps=[
            ('preprocessor', preprocessor), 
            ('dimensionality_reduction',  dimensionality_reduction[d_reduction]), 
            (classifier, classifiers[classifier])]

Basically what I'm hoping to achieve is to be able to identify a "champion" model based on for example, accuracy score and which dimensionality reduction technique was best for reducing my data's dimension. Below is the result running full training.

As you can see, the performance of backward elimination outperforms PCA and LDA.