Lack of variability of ANP result

[QhelDIV]

So here are two results (two columns) of ANP after 50000+ iterations.
The first row is the default plot and the plots in the second row contains 16 samples of the curve (given same context x & y). And the red curve is the mean of these 16 curves. As we can see, there has very limited variation.

This is true also for NP. However, if I set use_deterministic_path=False, the variation start to emerge:

My guess is that during training the decoder only prefer the deterministic path and just ignore the latent path. Then no matter what latent code is delivered the output won't change much. What's your opinion?