conv2d running very slowly

[shihux]

Hi,

I am running the tf_train.py and tf_utils code out of the box. Our tensorflow's version is 1.3.0 and GPU is GeForce GTX TITAN X. The conv2d function in tf_utils/layers.py is running very slowly. Specifically, the following two lines in conv2d take a long time:
_ = tf.get_variable("g", initializer=tf.log(scale_init) / 3.0)
_ = tf.get_variable("b", initializer=-m_init * scale_init)

I think due to lazy evaluation, what is actually taking time is this line:
m_init, v_init = tf.nn.moments(x_init, [0, 2, 3])
as both m_init and scale_init depend on the moments.

When running conv2d, nvidia-smi shows 'No running processes found' and 'GPU-Util Compute M.' is 0%. The CPU utilization is ~ 95%, which means it isn't exploiting the multi-core CPU architecture either. I wonder how I can speed it up.

Thank you!