get rid off the pytorch dependency that comes with lm_eval harness

right now we introduce a pytorch (cpu) dependency because lm-eval harness depends on torch. We only depend on lm_eval harness because we want to be able to download the eval datasets a-priori to scheduling evals since compute nodes don't have internet access. We should be able to get that functionality out of lm eval harness somehow to reduce the install size of our package