Negative Visit Forward Cross-Entropy on MIMIC-III

First of all, thank you for making your code available. This is a very interesting line of research.

In order to better understand your work, I rewrote med2vec in Python with TensorFlow rather than Theano. I then compared my results to yours on the MIMIC-III data set, with the same parameters used, expecting the results to be close. However, I discovered some issues:

1. In the calculation of visit forward cross-entropy, there are negative values. This leads to some cancellations and hence a visit cost of 300-400. Should negative values be considered here? In Tensorflow, the negative values are mapped to 0, giving a visit cost of ~4,000. What sort of cost values have you seen on MIMIC-III and other data sets?

2. My emb cost values are roughly equal to yours, but the value is < 10. Since visit cost >> emb cost, won't total cost just optimize for visits and not emb?

Below are some images related to visit cost calculation.

I'd be happy to make my code available to you, if you like. Thank you for your continued work on medical data analytics and I look forward to hearing back from you.

[mp2893]

Hi Brian,

Thanks for taking interest in our work.
I think the negative values coming from the loss is caused by the small value (logEps 1e-8) I added to the cross-entropy to prevent negative infinity.
I haven't run any experiments on MIMIC-III with med2vec, so I can't give you any concrete numbers.
The balance between emb cost and the visit cost is actually an interesting issue. I haven't done the hyperparameter search on that aspect, but the balance between the two depends on many factors including number of input codes and number of output codes.
We can talk more on email or Skype as you suggested, and maybe take a look at your code.

Thanks,
Ed