Skip to content
This repository was archived by the owner on Feb 26, 2025. It is now read-only.
This repository was archived by the owner on Feb 26, 2025. It is now read-only.

Automate parameter selection #11

@matz-e

Description

@matz-e

From JIRA:

As of today, most parameters we provide to Spark are based on either indications provided by the official documentation, or mostly on trial-and-error tests after we find issues. Some of these parameters, such as the memory consumption limit per driver / executor, can be automized based on the input parameters. For instance, we can launch a mock simulation just to understand the input sizes and then configure Spark properly. It would also be good to recover after we detect a timeout exception, tweaking whatever parameters caused the problem automatically. The final .properties file can then be stored, to be used again if needed in other simulations of the same campaign.

As a consequence, the goal of this task is to explore such functionality in the context of the functionalizer, preventing out-of-memory exceptions, timeout exceptions, and so on.

This should indeed be tuned to account for the available resources.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions