I left out the precomputed BRDF for the specular term. This should also be considered https://learnopengl.com/PBR/IBL/Specular-IBL