Reproducibility of results

[varghesealex90]

Hi;

Thank you for sharing the reimplementation. Were you able to reproduce the results as stated in the paper?

[clu8]

Hi, thanks for reaching out. This is still a work in progress but I haven't been able to achieve the results stated in the paper so far.

It looks like you saw https://github.com/arnoweng/CheXNet also. They seem to have improved on the results in the paper.

It may also be interesting to take a look at this post: https://lukeoakdenrayner.wordpress.com/2018/01/24/chexnet-an-in-depth-review/. Apparently the paper used a different test set which has not been released.

[varghesealex90]

I did check Arnoweng's page. They did implement the classifier for the 14 class problem. I was more interested in differentiating between pneumonia and non-pneumonia. I assume that's the problem you are also trying the solve here. I did try to implement it, however, upon checking the confusion matrix during the training phase, the model was completely skewed toward non-pneuomnia class.

Patients with clinical findings with the tag "Pneumonia" was set as a class 1, while all other images ( No findings, hernia, pneuothroax etc) were set as class 2. Did you follow the same scheme as above or did you just consider No Finding as one class and "Pneumonia" as another.

Thanks in advance for your help

[clu8]

arnoweng's implementation, like ChexNet, should do something similar - since they output a probability between 0 and 1 for each of the 14 classes. It is not a softmax; each one of the 14 is a sigmoid output if I remember correctly. So it should be fairly similar to the results of training a binary classification model for pneumonia specifically (but maybe less performant since the 14 outputs need to share representations). Regardless, their performance on pneumonia even for the 14 class model is much better than what my reimplementation has achieved so far.

Yes, for my data, I labeled patients with "Pneumonia" as 1, and all other patients as 0.

About the skew - that should be expected since the fraction of patients is very small. Are you using the weighted binary cross entropy loss mentioned in the paper?

[varghesealex90]

I made a few changes, namely:

1. Made the number neurons in final layer of Densenet-121 as 2.
2. Made the loss as "CrossEntropy"
3. Assigned weights to class 0 (Non-Pneumonia) to 0.012 and weights to Pneumonia as 0.98.

Optimizer was adam, learning rate was set 0.001. I ran the model for 30 epochs and visualized the confusion matrix at the end of every epoch. Throughout the training phase, all images were being predicted as non-pneuomia.

If I am correct "CrossEntropy" is a special case of binary cross entropy. The only difference I see in your implementation is that the weight assigned to BCE varies every batch. I wonder if that has such a big impact.

[clu8]

All three of your changes seem about right. The reason that I have a different BCE weight vector per batch is because PyTorch requires it to be specified that way (a weight can't be assigned to a particular class).

Perhaps your issue is that you are setting the decision boundary to be at a particular threshold. For instance are you using 0.5 as your threshold? That might result in everything predicted as non-pneumonia. Instead you could simply take your output probabilities and calculate an AUROC using that.

[varghesealex90]

I tried running the code you have supplied over here.

80 % images labeled as pneumonia and non-pneuomia were kept in "image_train", while the rest was kept in "image_test". The path leading to either of them were updated in the config.py

The n_pos and n_neg were found to meaningful numbers/values.

Since I am using python 2.7, I converted the print(f) statements to print statements, for example
print(f'Validation loss (approximate): {val_loss}') was converted to
print('Validation loss (approximate):', val_loss)

I commented out few printf statements in such as print(f'Average precision score: {average_precision_score(all_labels, all_logits)}') has it was throwing up errors.

Line number 105 in your code estimates the validation error prior to first epoch, upon printing the value returned, I see the result as 0.0

A loss of 0.0 seems weird.

Any thoughts on why this is happening?

[clu8]

That's strange - I would recommend running in Python 3.6 and seeing if you have the same problem (I don't). It might be a case of integer division.

[varghesealex90]

I shall run it on python 3.6. Additionally I think I have an older version of torchvision installed . I will updated all packages and get back to you.

[varghesealex90]

Works without errors on python 3.6 version and latest torchvision packages

[varghesealex90]

Hi Charles,

I just completed training the model for 40 epochs.

The best validation score was the following

Average precision score: 0.027731635846752766
AUROC: 0.6785733951689875
Validation loss (approximate): 0.017031026797957052
Elapsed: 231.7613034248352

And the test scores were the following :
================= Evaluating on test set ==================
Loaded model params from model.pkl
Average precision score: 0.028192913236588693
AUROC: 0.7086328914568882
Validation loss (approximate): 0.014584553485454595
Elapsed: 453.84413290023804

[varghesealex90]

Is this the performance when you get?

[varghesealex90]

I wrote some lines of code to calculate the accuracy.

accuracy_counter =0
for i , (img,label) in enumerate(test_loader):
img, label_var = make_var(img, volatile=True), make_var(label, volatile=True)
logit = model(img)
scaler = nn.Sigmoid() ### scale values between 0 and 1
probabs = scaler(logit)
probabs = probabs.cpu().data.numpy()
probabs[np.where(probabs<0.5)] =0
probabs[np.where(probabs>=0.5)] =1
counter = np.sum(label.numpy()==probabs)
accuracy_counter = accuracy_counter+counter

print ('accuracy:', accuracy_counter/22423.0)

Output
accuracy:0.6129866654774115

[varghesealex90]

Code of people who are interested in finding the optimal threshold and visualizing the auc and confusion matrix

`import os
import statistics
import time

from sklearn.metrics import auc,roc_curve,average_precision_score, confusion_matrix, roc_auc_score
import torch
from torch.autograd import Variable
import torchvision.transforms as transforms

import config
from cxc_dataset import PneumoniaDataset
from model import PneumoniaNet
from torch import nn
import numpy as np
import matplotlib.pyplot as plt
def make_var(x, volatile=False):
return Variable(x.cuda(), volatile) if config.use_gpu else Variable(x, volatile)

def evaluate(model, loader):
start = time.time()
all_logits = []
all_labels = []
all_losses = []
for i, (img, label) in enumerate(loader):
img, label_var = make_var(img, volatile=True), make_var(label, volatile=True)
logit = model(img)
loss = model.loss(logit, label_var)

    all_logits.extend(list(logit.data.cpu().numpy()))
    all_labels.extend(list(label))
    all_losses.append(float(loss.data.cpu().numpy()[0]))
val_loss = statistics.mean(all_losses)

print(f'Average precision score: {average_precision_score(all_labels, all_logits)}')
print(f'AUROC: {roc_auc_score(all_labels, all_logits)}')
print(f'Validation loss (approximate): {val_loss}')
print(f'Elapsed: {time.time() - start}')
return val_loss

normalize = transforms.Normalize(
mean=[0.485, 0.456, 0.406],
std=[0.229, 0.224, 0.225]
)

transform = transforms.Compose([
transforms.Resize(224),
transforms.RandomHorizontalFlip(),
transforms.ToTensor(),
normalize
])

train_dset = PneumoniaDataset(
config.train_path, '/home/uavws/chest_xray/Data_Entry_2017.csv', transform,
lambda pid: pid % config.val_proportion != 0
)
val_dset = PneumoniaDataset(
config.train_path,'/home/uavws/chest_xray/Data_Entry_2017.csv' , transform,
lambda pid: pid % config.val_proportion == 0
)
num_pos = sum(train_dset.labels)
num_neg = len(train_dset) - num_pos

test_dset = PneumoniaDataset(config.test_path, '/home/uavws/chest_xray/Data_Entry_2017.csv', transform)

test_loader = torch.utils.data.DataLoader(
test_dset,
batch_size=config.val_batch_size,
shuffle=False,
num_workers=config.workers,
pin_memory=True
)

model = PneumoniaNet(config.use_gpu, class_counts=(num_pos, num_neg), verbose=True)
model.load_state_dict(torch.load('/home/uavws/chest_xray_data/gitv/model.pkl'))

accuracy_counter =0
all_labels=[]
all_logits=[]
for i , (img,label) in enumerate(test_loader):
img, label_var = make_var(img, volatile=True), make_var(label, volatile=True)
logit = model(img)
# scaler = nn.Sigmoid()
# probabs = scaler(logit)

all_logits.extend(list(logit.data.cpu().numpy()))
all_labels.extend(list(label))

probabs =logit
probabs = probabs.cpu().data.numpy()
probabs[np.where(probabs<0.12)] =0
probabs[np.where(probabs>=0.12)] =1

counter = np.sum(label.numpy()==probabs)
accuracy_counter = accuracy_counter+counter

print(f'AUROC: {roc_auc_score(all_labels, all_logits)}')
fpr,tpr,thresholds= roc_curve(all_labels,all_logits)
roc_auc = auc(fpr, tpr)
lw = 2
plt.plot(fpr, tpr, color='darkorange',
lw=lw, label='ROC curve (area = %0.2f)' % roc_auc)
plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic example')
plt.legend(loc="lower right")
plt.show()

optimal_difference = tpr-fpr

optimal_index = np.where(optimal_difference==optimal_difference.max())

optimal_threshold = thresholds[optimal_index]

print ('Sensitivity', tpr[optimal_index])

print
print ('1- Specificity',fpr[optimal_index])

copy_predictions = np.array(all_logits)
copy_predictions[np.where(copy_predictions<=optimal_threshold)] = 0
copy_predictions[np.where(copy_predictions>optimal_threshold)] = 1

true_labels = np.array(all_labels)

cm =confusion_matrix(true_labels,copy_predictions)
print ('confusion_matrix')
print
print (cm)

`