Do you have the evaluation code?

[XUHAN314]

I didn't find the evaluation or test code in the repo. How do you test the generative model?

[bengioe]

Hi, experiments typically produce a file called sampled_mols.pkl.gz:

gflownet/mols/gflownet.py

Lines 384 to 400 in 831a698

	def save_stuff():
	pickle.dump([i.data.cpu().numpy() for i in model.parameters()],
	gzip.open(f'{exp_dir}/params.pkl.gz', 'wb'))
	pickle.dump(dataset.sampled_mols,
	gzip.open(f'{exp_dir}/sampled_mols.pkl.gz', 'wb'))
	pickle.dump({'train_losses': train_losses,
	'test_losses': test_losses,
	'test_infos': test_infos,
	'time_start': time_start,
	'time_now': time.time(),
	'args': args,},
	gzip.open(f'{exp_dir}/info.pkl.gz', 'wb'))
	pickle.dump(train_infos,
	gzip.open(f'{exp_dir}/train_info.pkl.gz', 'wb'))

What's reported in the paper is the rolling top-k, which we compute after the fact by loading the molecules and computing the top-k-so-far during learning.

The results are stored as a list of 4-tuples:

gflownet/mols/gflownet.py

Line 210 in 831a698

self.sampled_mols.append((r, m, trajectory_stats, inflow))

So to get the ground reward you can take the r value and inverse-exponentiate it back to the reward prediction of the proxy r ** (1 / args.reward_exp).

It would be nice to add a notebook with that kind of code. It shouldn't be too hard for me to take existing notebooks and clean them up, but I don't have time right now. I'll mark this issue as enhancement.

[XUHAN314]

Thanks for your reply. There are some plots in the paper, I am wondering how do you make them. It would be nice if you could create a notebook to show that. Thanks! This is a great work!

[XUHAN314]

Does Top K mean you selected K mols with the highest rewards? How do you select K?

[bengioe]

By Top K we mean something like topk = lambda x, k: x[np.argsort(x)[-k:]]. In the paper we report for k 10, 100, and 1000, where x is the set of rewards sampled so far.

[XUHAN314]

Thanks! It helps a lot.

[58191554]

which file should I run first, in other words, where is the beginning?

[bengioe]

Simply running python gflownet.py in the mols directory should train a model with the default hyperparameters.