loss is nan

[xunfeng2zkj]

Model: "graphmodel_1"

---

Layer (type) Output Shape Param # Connected to

atom (InputLayer) (None, 1) 0

---

squeeze_1 (Squeeze) (None,) 0 atom_index[0][0]
atom[0][0]
n_pro[0][0]

---

atom_embedding (Embedding) (None, 256) 2560 squeeze_1[1][0]

---

dense_1 (Dense) (None, 256) 65536 atom_embedding[0][0]

---

connectivity (InputLayer) (None, 2) 0

---

gather_atom_to_bond_1 (GatherAt (None, 256) 0 dense_1[0][0]
connectivity[0][0]

---

gather_atom_to_bond_2 (GatherAt (None, 256) 0 dense_1[0][0]
connectivity[0][0]

---

distance_rbf (InputLayer) (None, 256) 0

---

concatenate_1 (Concatenate) (None, 768) 0 gather_atom_to_bond_1[0][0]
gather_atom_to_bond_2[0][0]
distance_rbf[0][0]

---

dense_2 (Dense) (None, 512) 393728 concatenate_1[0][0]

---

dense_3 (Dense) (None, 256) 131328 dense_2[0][0]

---

dense_4 (Dense) (None, 256) 65792 dense_3[0][0]

---

dense_5 (Dense) (None, 256) 65792 dense_4[0][0]

---

add_1 (Add) (None, 256) 0 dense_5[0][0]
distance_rbf[0][0]

---

multiply_1 (Multiply) (None, 256) 0 gather_atom_to_bond_1[0][0]
add_1[0][0]

---

reduce_bond_to_atom_1 (ReduceBo (None, 256) 0 multiply_1[0][0]
connectivity[0][0]

---

dense_6 (Dense) (None, 256) 65792 reduce_bond_to_atom_1[0][0]

---

dense_7 (Dense) (None, 256) 65792 dense_6[0][0]

---

add_2 (Add) (None, 256) 0 dense_1[0][0]
dense_7[0][0]

---

dense_8 (Dense) (None, 256) 65536 add_2[0][0]

---

gather_atom_to_bond_3 (GatherAt (None, 256) 0 dense_8[0][0]
connectivity[0][0]

---

gather_atom_to_bond_4 (GatherAt (None, 256) 0 dense_8[0][0]
connectivity[0][0]

---

concatenate_2 (Concatenate) (None, 768) 0 gather_atom_to_bond_3[0][0]
gather_atom_to_bond_4[0][0]
add_1[0][0]

---

dense_9 (Dense) (None, 512) 393728 concatenate_2[0][0]

---

dense_10 (Dense) (None, 256) 131328 dense_9[0][0]

---

dense_11 (Dense) (None, 256) 65792 dense_10[0][0]

---

dense_12 (Dense) (None, 256) 65792 dense_11[0][0]

---

add_3 (Add) (None, 256) 0 dense_12[0][0]
add_1[0][0]

---

multiply_2 (Multiply) (None, 256) 0 gather_atom_to_bond_3[0][0]
add_3[0][0]

---

reduce_bond_to_atom_2 (ReduceBo (None, 256) 0 multiply_2[0][0]
connectivity[0][0]

---

dense_13 (Dense) (None, 256) 65792 reduce_bond_to_atom_2[0][0]

---

dense_14 (Dense) (None, 256) 65792 dense_13[0][0]

---

add_4 (Add) (None, 256) 0 dense_8[0][0]
dense_14[0][0]

---

dense_15 (Dense) (None, 256) 65536 add_4[0][0]

---

gather_atom_to_bond_5 (GatherAt (None, 256) 0 dense_15[0][0]
connectivity[0][0]

---

gather_atom_to_bond_6 (GatherAt (None, 256) 0 dense_15[0][0]
connectivity[0][0]

---

concatenate_3 (Concatenate) (None, 768) 0 gather_atom_to_bond_5[0][0]
gather_atom_to_bond_6[0][0]
add_3[0][0]

---

dense_16 (Dense) (None, 512) 393728 concatenate_3[0][0]

---

dense_17 (Dense) (None, 256) 131328 dense_16[0][0]

---

dense_18 (Dense) (None, 256) 65792 dense_17[0][0]

---

dense_19 (Dense) (None, 256) 65792 dense_18[0][0]

---

add_5 (Add) (None, 256) 0 dense_19[0][0]
add_3[0][0]

---

multiply_3 (Multiply) (None, 256) 0 gather_atom_to_bond_5[0][0]
add_5[0][0]

---

reduce_bond_to_atom_3 (ReduceBo (None, 256) 0 multiply_3[0][0]
connectivity[0][0]

---

dense_20 (Dense) (None, 256) 65792 reduce_bond_to_atom_3[0][0]

---

dense_21 (Dense) (None, 256) 65792 dense_20[0][0]

---

atom_index (InputLayer) (None, 1) 0

---

n_pro (InputLayer) (None, 1) 0

---

add_6 (Add) (None, 256) 0 dense_15[0][0]
dense_21[0][0]

---

reduce_atom_to_pro_1 (ReduceAto (None, 256) 0 add_6[0][0]
squeeze_1[0][0]
squeeze_1[2][0]

---

dense_22 (Dense) (None, 256) 65792 reduce_atom_to_pro_1[0][0]

---

dense_23 (Dense) (None, 256) 65792 dense_22[0][0]

---

dense_24 (Dense) (None, 128) 32896 dense_23[0][0]

---

atomwise_shift (Embedding) (None, 1) 10 squeeze_1[1][0]

---

dense_25 (Dense) (None, 1) 129 dense_24[0][0]

---

reduce_atom_to_pro_2 (ReduceAto (None, 1) 0 atomwise_shift[0][0]
squeeze_1[0][0]
squeeze_1[2][0]

---

add_7 (Add) (None, 1) 0 dense_25[0][0]
reduce_atom_to_pro_2[0][0]

Total params: 2,728,459
Trainable params: 2,728,459
Non-trainable params: 0

---

2023-10-31 06:41:41.641883: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2023-10-31 06:41:41.773098: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-10-31 06:41:41.773275: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:
name: NVIDIA GeForce RTX 3090 major: 8 minor: 6 memoryClockRate(GHz): 1.725
pciBusID: 0000:05:00.0
totalMemory: 23.69GiB freeMemory: 23.43GiB
2023-10-31 06:41:41.773292: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0
2023-10-31 06:41:42.223463: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2023-10-31 06:41:42.223495: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0
2023-10-31 06:41:42.223501: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N
2023-10-31 06:41:42.223603: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 22717 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 3090, pci bus id: 0000:05:00.0, compute capability: 8.6)
{'n_pro': array([24, 21, 7, 27, 12, 10, 10, 20]), 'atom': array([2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 2, 3, 2, 2, 2, 2, 2,
2, 2, 2, 2, 3, 2, 2, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 2, 3, 2, 2, 2, 2, 2, 2, 2,
2, 3, 2, 3, 2, 2, 3, 2, 3, 2, 2, 2, 3, 2, 2, 2, 4, 2, 4, 2, 4, 4,
4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
4, 4, 4, 4, 2, 2, 2, 2, 5, 3, 2, 2, 2, 3, 3, 4, 6, 4, 4, 4, 4, 4,
4, 4, 4, 4, 4, 4, 4, 4, 4, 2, 2, 2, 2, 2, 2, 2, 6, 2, 2, 2, 2, 2,
2, 3, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 4, 4, 4, 4, 4, 4,
4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 6, 2, 6, 2, 5, 2, 2, 2, 5,
2, 2, 2, 2, 2, 3, 3, 2, 2, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
4, 4, 4, 4, 4, 4, 4, 4, 4, 6, 2, 6, 2, 5, 2, 2, 3, 2, 2, 2, 2, 5,
2, 2, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 2, 2, 2, 3, 2, 2,
2, 3, 3, 3, 6, 2, 3, 2, 6, 6, 2, 2, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4,
4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
6, 2, 2, 2, 2, 3, 3, 2, 2, 6, 6, 2, 2, 3, 4, 4, 4, 4, 4, 4, 4, 4,
4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4]), 'bond': array([2, 3, 4, ..., 0, 0, 0]), 'connectivity': array([[ 0, 31],
[ 0, 5],
[ 0, 1],
...,
[342, 298],
[342, 310],
[342, 322]]), 'atom_index': array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, -1,
-1, -1, 12, -1, 13, 14, 15, 16, 17, 18, 19, 20, 21,
-1, 22, 23, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
-1, -1, -1, -1, -1, 24, -1, 25, 26, 27, 28, 29, 30,
31, 32, -1, 33, -1, 34, 35, -1, 36, -1, 37, 38, 39,
-1, 40, 41, 42, -1, 43, -1, 44, -1, -1, -1, -1, -1,
-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 45, 46, 47,
48, -1, -1, 49, 50, 51, -1, -1, -1, -1, -1, -1, -1,
-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 52, 53,
54, 55, 56, 57, 58, -1, 59, 60, 61, 62, 63, 64, -1,
65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77,
78, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
-1, -1, -1, -1, -1, -1, -1, -1, 79, -1, 80, -1, 81,
82, 83, -1, 84, 85, 86, 87, 88, -1, -1, 89, 90, -1,
-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
-1, -1, -1, -1, -1, -1, -1, -1, -1, 91, -1, 92, -1,
93, 94, -1, 95, 96, 97, 98, -1, 99, 100, -1, -1, -1,
-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 101, 102,
103, -1, 104, 105, 106, -1, -1, -1, -1, 107, -1, 108, -1,
-1, 109, 110, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 111, 112, 113,
114, 115, 116, 117, 118, 119, 120, 121, 122, -1, 123, 124, 125,
126, -1, -1, 127, 128, -1, -1, 129, 130, -1, -1, -1, -1,
-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
-1, -1, -1, -1, -1]), 'node_graph_indices': array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4,
4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
4, 4, 4, 4, 4, 4, 4, 4, 4, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5,
5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 6, 6, 6, 6, 6, 6,
6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6,
6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,
7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,
7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7]), 'distance_rbf': array([[8.28731687e-06, 6.52437027e-05, 4.20537142e-04, ...,
0.00000000e+00, 0.00000000e+00, 0.00000000e+00],
[5.55421160e-09, 7.92017194e-08, 9.24672414e-07, ...,
0.00000000e+00, 0.00000000e+00, 0.00000000e+00],
[3.65965223e-09, 5.37801536e-08, 6.47061206e-07, ...,
0.00000000e+00, 0.00000000e+00, 0.00000000e+00],
...,
[6.50888555e-97, 7.20740461e-93, 6.53419777e-89, ...,
0.00000000e+00, 0.00000000e+00, 0.00000000e+00],
[4.27906304e-97, 4.78068100e-93, 4.37292421e-89, ...,
0.00000000e+00, 0.00000000e+00, 0.00000000e+00],
[3.26498268e-99, 4.04323416e-95, 4.09937841e-91, ...,
0.00000000e+00, 0.00000000e+00, 0.00000000e+00]])}
[[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]
[99.590904]] (131, 1)
/root/miniconda3/envs/my-rdkit-env/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py:112: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
Epoch 1/1200
139/807 [====>.........................] - ETA: 20s - loss: nan