The discontinuity between batches using SpatialBatchNormalization?

[xjwxjw]

Hi!
Very excellent work, thanks for your effort!
Here I have a question: now that the SpatialBatchNormalization is used in the autoencoder architecture, the mean and variance varies between different batches. So this results in discontinuity of the output of encoder, and this may have a serious impact on the training process of LSTM. Do you have any idea on how to solve this problem?
Thanks very much!

[ArturoDeza]

My opinion on this: Since the Adversarially trained AutoEncoder and LSTM are trained separately, I think it should have little effect. The Decoder is only trained to invert the latent representations independent of whatever goes inside the LSTM. It's the LSTM's job to figure out how to modify these latent representations over time through its cell + hidden states.