You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Feb 26, 2025. It is now read-only.
As of today, most parameters we provide to Spark are based on either indications provided by the official documentation, or mostly on trial-and-error tests after we find issues. Some of these parameters, such as the memory consumption limit per driver / executor, can be automized based on the input parameters. For instance, we can launch a mock simulation just to understand the input sizes and then configure Spark properly. It would also be good to recover after we detect a timeout exception, tweaking whatever parameters caused the problem automatically. The final .properties file can then be stored, to be used again if needed in other simulations of the same campaign.
As a consequence, the goal of this task is to explore such functionality in the context of the functionalizer, preventing out-of-memory exceptions, timeout exceptions, and so on.
This should indeed be tuned to account for the available resources.