I implemented the code but the Performance is not similar to the paper. #7
Description
After many tries. I have implemented the code.
I have two questions.
- How did you select the lowest ppl checkpoint? In my case, I trained the model twice, first to see the checkpoints and evals results, then I manually break the training on the checkpoint with the lowest ppl, then train and evaluate the model. Below is the eval result. (the lowest ppl is 16.21 around)
- Why my eval result is high? Is there something I might be missing? Despite the hyperparameters are the same as in paper.
This is the evaluation result, after training in the same parameter in the paper exp. setup:
{"length": 19.346938775510203, "dist-1": 3.9611914472871095, "dist-2": 17.547699947726084, "dist-3": 32.19769174782518, "bleu-1": 19.537415536727647, "bleu-2": 8.045721170784967, "bleu-3": 4.084835823220594, "bleu-4": 2.3918514292851243, "f1": 20.91022932286169, "rouge-l": 17.504459395975523}