-
Notifications
You must be signed in to change notification settings - Fork 17
Description
When using a GridWorker, only the output of the logger of the current Worker (e.g. OfflineTrainer or Tester) is captured. The loggers of the Model & Problem are not able to write to the console or to the .log file.
This is fairly annoying as these loggers have some important information (e.g. whether the Sampler was properly instantiated, when is saved the Model etc).
Not sure yet how to fix this. I think this is related to the logging.config and also how the output of a child process is captured in the GridWorker:
mi-prometheus/miprometheus/grid_workers/grid_trainer_cpu.py
Lines 316 to 317 in 3d4f868
| with open(os.devnull, 'w') as devnull: | |
| result = subprocess.run(command_str.split(" "), stdout=devnull) |
We should probably change stdout=os.devnull and do something instead like
with open(worker.log, 'w') as devnull:
result = subprocess.run(command_str.split(" "), stdout=devnull)