From 6d67ae1b149bdd50d52120286c2f214402e020a9 Mon Sep 17 00:00:00 2001 From: Sergiy Matusevych Date: Tue, 11 Mar 2025 18:12:12 -0700 Subject: [PATCH 01/35] add docstrings on Environment Parameterization and Variable Propagation --- .../mlos_bench/environments/__init__.py | 79 +++++++++++++++++-- 1 file changed, 74 insertions(+), 5 deletions(-) diff --git a/mlos_bench/mlos_bench/environments/__init__.py b/mlos_bench/mlos_bench/environments/__init__.py index 6c8170bac2..1280c9e6b8 100644 --- a/mlos_bench/mlos_bench/environments/__init__.py +++ b/mlos_bench/mlos_bench/environments/__init__.py @@ -28,7 +28,7 @@ This lets Environments be very flexible in what they can accomplish. Environments can be stacked together with the :py:class:`.CompositeEnv` class to -represent complex setups (e.g., an appication running on a remote VM with a +represent complex setups (e.g., an application running on a remote VM with a benchmark running from a local machine). See below for the set of Environments currently available in this package. @@ -37,6 +37,29 @@ :py:class:`.Environment` class and referencing them in the :py:mod:`json configs ` using the ``class`` key. +Environment Parameterization +++++++++++++++++++++++++++++ + +Each environment can have a set of parameters that define the environment's configuration. +These parameters can be _constant_ (i.e., immutable from one trial run to the next) or +_tunable_ (i.e., suggested by the optimizer or provided by the user). +The following clauses in the environment configuration are used to declare these parameters: + + - `tunable_params`: A list of tunable parameters' _groups_. + The environment will obtain these parameters from the outside (e.g., from the optimizer). + - `const_args`: A dictionary of constant parameters along with their values. + - `required_args`: A list of constant parameters supplied to the environment externally + (i.e., from a parent environment, global config file, or command line). + +During the setup and run phases, MLOS will combine the constant and tunable parameters and their +values into a single dictionary and pass it to the corresponding method. + +Values of constant parameters defined in the environment config can be overridden with the values +from the command line and/or external config files. +That allows MLOS users to have reusable immutable environment configurations and move all +experiment-specific or sensitive data outside of the version-controlled files. +We discuss the parameter propagation mechanism in the section below. + Environment Tunables ++++++++++++++++++++ @@ -72,6 +95,45 @@ dynamically change the set of active TunableGroups for a given Experiment using only `globals`, allowing for configs to be more modular and composable. +Variable Propagation +++++++++++++++++++++ + +Parameters declared in the `const_args` or `required_args` sections of the environment config can +be overridden with values of the corresponding parameters of the parent environment or specified +in the external config files or the command line. In fact, `const_args` or `required_args` +sections can be viewed as placeholders for the parameters that are being pushed to the environment +from the outside. + +Variable replacement happens in the bottom-up manner. That is, if a certain parameter is present +in the parent (composite) environment, it will replace the corresponding parameter in the child, +and so on. Note that the parameter _must_ appear in the child environment `const_args` or +`required_args` section; if a parameter is not present in one of these placeholders of the child +environment config, it will not be propagated. This hierarchy allows MLOS users to have small +immutable environment configurations at the lower levels and combine and parameterize them at the +higher levels. + +Taking it to the next level outside of the environment configs, the parameters can be defined in +the external key-value JSON config files (usually referred to as "global config files" in MLOS +lingo). Users can have multiple global config files; at runtime, parameters from these files will +be combined into a single dictionary and pushed to the root environment. This way, users can keep +their experiment-specific parameters separately from the environment configs making them more +reusable. Another common use of global config files is to store sensitive data (e.g., passwords, +tokens, etc.) that should not be version-controlled. The global config files are specified in the +`globals` section of the top-level CLI config, or in the `--globals` command line parameter. + +Finally, any global or environment parameter can be overridden from the command line, by simply +specifying `--PARAMETER_NAME PARAMETER_VALUE`. + +We can summarize the parameter propagation rules as follows: +1. Child environment will only get the parameters defined in its `const_args` or + `required_args` sections. +1. Value of the parameter defined in the `const_args` section of the parent environment will + override the value of the corresponding parameter in the child environments. +1. Values of the parameters defined in the global config files will override the values of the + corresponding parameters in all environments. +1. Values of the command line parameters take precedence over values defined in the global or + environment configs. + Environment Services ++++++++++++++++++++ @@ -86,10 +148,17 @@ used in different settings (e.g., local machine, SSH accessible machine, Azure VM, etc.) without having to change the Environment config. -Variable Propagation -++++++++++++++++++++ -TODO: Document how variable propagation works in the script environments using -required_args, const_args, etc. +Variable propagation rules described in the previous section for the environment configs also +apply to the service configurations. That is, every parameter defined in the service config can be +overridden by a corresponding parameter from the global config or the command line. + +All global configs, command line parameters, environment `const_args` and `required_args` +sections, and service config parameters thus form one flat name space of parameters. This imposes +a certain risk of name clashes, but also simplifies the configuration process and allows users to +keep all experiment-specific data in a few human-readable files. + +We will discuss the examples of such global and local configuration parameters in the +documentation of the concrete services and environments. Examples -------- From 37cc18b77e7f640bd19d91b30461b09a98b68d96 Mon Sep 17 00:00:00 2001 From: Sergiy Matusevych Date: Wed, 12 Mar 2025 14:49:01 -0700 Subject: [PATCH 02/35] fix formatting issues --- .../mlos_bench/environments/__init__.py | 43 ++++++++++--------- 1 file changed, 23 insertions(+), 20 deletions(-) diff --git a/mlos_bench/mlos_bench/environments/__init__.py b/mlos_bench/mlos_bench/environments/__init__.py index 1280c9e6b8..7fe84026c8 100644 --- a/mlos_bench/mlos_bench/environments/__init__.py +++ b/mlos_bench/mlos_bench/environments/__init__.py @@ -45,11 +45,13 @@ _tunable_ (i.e., suggested by the optimizer or provided by the user). The following clauses in the environment configuration are used to declare these parameters: - - `tunable_params`: A list of tunable parameters' _groups_. - The environment will obtain these parameters from the outside (e.g., from the optimizer). - - `const_args`: A dictionary of constant parameters along with their values. - - `required_args`: A list of constant parameters supplied to the environment externally - (i.e., from a parent environment, global config file, or command line). +- ``tunable_params``: A list of tunable parameters' _groups_. + The environment will obtain these parameters from the outside (e.g., from the optimizer). + +- ``const_args``: A dictionary of constant parameters along with their values. + +- ``required_args``: A list of constant parameters supplied to the environment externally + (i.e., from a parent environment, global config file, or command line). During the setup and run phases, MLOS will combine the constant and tunable parameters and their values into a single dictionary and pass it to the corresponding method. @@ -98,16 +100,16 @@ Variable Propagation ++++++++++++++++++++ -Parameters declared in the `const_args` or `required_args` sections of the environment config can -be overridden with values of the corresponding parameters of the parent environment or specified -in the external config files or the command line. In fact, `const_args` or `required_args` -sections can be viewed as placeholders for the parameters that are being pushed to the environment -from the outside. +Parameters declared in the ``const_args`` or ``required_args`` sections of the environment config +can be overridden with values of the corresponding parameters of the parent environment or +specified in the external config files or the command line. In fact, ``const_args`` or +``required_args`` sections can be viewed as placeholders for the parameters that are being pushed +to the environment from the outside. Variable replacement happens in the bottom-up manner. That is, if a certain parameter is present in the parent (composite) environment, it will replace the corresponding parameter in the child, -and so on. Note that the parameter _must_ appear in the child environment `const_args` or -`required_args` section; if a parameter is not present in one of these placeholders of the child +and so on. Note that the parameter _must_ appear in the child environment ``const_args`` or +``required_args`` section; if a parameter is not present in one of these placeholders of the child environment config, it will not be propagated. This hierarchy allows MLOS users to have small immutable environment configurations at the lower levels and combine and parameterize them at the higher levels. @@ -119,19 +121,20 @@ their experiment-specific parameters separately from the environment configs making them more reusable. Another common use of global config files is to store sensitive data (e.g., passwords, tokens, etc.) that should not be version-controlled. The global config files are specified in the -`globals` section of the top-level CLI config, or in the `--globals` command line parameter. +``globals`` section of the top-level CLI config, or in the ``--globals`` command line parameter. Finally, any global or environment parameter can be overridden from the command line, by simply -specifying `--PARAMETER_NAME PARAMETER_VALUE`. +specifying ``--PARAMETER_NAME PARAMETER_VALUE``. We can summarize the parameter propagation rules as follows: -1. Child environment will only get the parameters defined in its `const_args` or - `required_args` sections. -1. Value of the parameter defined in the `const_args` section of the parent environment will + +1. Child environment will only get the parameters defined in its ``const_args`` or + ``required_args`` sections. +2. Value of the parameter defined in the ``const_args`` section of the parent environment will override the value of the corresponding parameter in the child environments. -1. Values of the parameters defined in the global config files will override the values of the +3. Values of the parameters defined in the global config files will override the values of the corresponding parameters in all environments. -1. Values of the command line parameters take precedence over values defined in the global or +4. Values of the command line parameters take precedence over values defined in the global or environment configs. Environment Services @@ -152,7 +155,7 @@ apply to the service configurations. That is, every parameter defined in the service config can be overridden by a corresponding parameter from the global config or the command line. -All global configs, command line parameters, environment `const_args` and `required_args` +All global configs, command line parameters, environment ``const_args`` and ``required_args`` sections, and service config parameters thus form one flat name space of parameters. This imposes a certain risk of name clashes, but also simplifies the configuration process and allows users to keep all experiment-specific data in a few human-readable files. From eea4aeb6a01c9145aa2907b8035c9b0b8644eec0 Mon Sep 17 00:00:00 2001 From: Sergiy Matusevych Date: Wed, 12 Mar 2025 15:43:31 -0700 Subject: [PATCH 03/35] minor docstring improvements --- .../mlos_bench/environments/__init__.py | 24 +++++++++++-------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/mlos_bench/mlos_bench/environments/__init__.py b/mlos_bench/mlos_bench/environments/__init__.py index 7fe84026c8..53036c7294 100644 --- a/mlos_bench/mlos_bench/environments/__init__.py +++ b/mlos_bench/mlos_bench/environments/__init__.py @@ -40,19 +40,22 @@ Environment Parameterization ++++++++++++++++++++++++++++ -Each environment can have a set of parameters that define the environment's configuration. -These parameters can be _constant_ (i.e., immutable from one trial run to the next) or -_tunable_ (i.e., suggested by the optimizer or provided by the user). +Each :py:class:`~mlos_bench.environments.Environment` can have a set of parameters that define the +environment's configuration. These parameters can be _constant_ (i.e., immutable from one trial +run to the next) or _tunable_ (i.e., suggested by the optimizer or provided by the user). The following clauses in the environment configuration are used to declare these parameters: -- ``tunable_params``: A list of tunable parameters' _groups_. - The environment will obtain these parameters from the outside (e.g., from the optimizer). +- ``tunable_params``: A list of tunable parameters' _groups_. At each trial, the environment will + obtain the new values of these parameters from the outside (e.g., from the optimizer). -- ``const_args``: A dictionary of constant parameters along with their values. +- ``const_args``: A dictionary of _constant_ parameters along with their values. -- ``required_args``: A list of constant parameters supplied to the environment externally +- ``required_args``: A list of _constant_ parameters supplied to the environment externally (i.e., from a parent environment, global config file, or command line). +That is, tunable parameters change on every trial, while constant parameters stay fixed for the +entire experiment. + During the setup and run phases, MLOS will combine the constant and tunable parameters and their values into a single dictionary and pass it to the corresponding method. @@ -123,8 +126,8 @@ tokens, etc.) that should not be version-controlled. The global config files are specified in the ``globals`` section of the top-level CLI config, or in the ``--globals`` command line parameter. -Finally, any global or environment parameter can be overridden from the command line, by simply -specifying ``--PARAMETER_NAME PARAMETER_VALUE``. +Finally, any global or :py:class:`~mlos_bench.environments.Environment` parameter can be +overridden from the command line, by simply specifying ``--PARAMETER_NAME PARAMETER_VALUE``. We can summarize the parameter propagation rules as follows: @@ -161,7 +164,8 @@ keep all experiment-specific data in a few human-readable files. We will discuss the examples of such global and local configuration parameters in the -documentation of the concrete services and environments. +documentation of the concrete :py:mod:`~mlos_bench.services` and +:py:mod:`~mlos_bench.environments`. Examples -------- From 0a4d0500a05a9c8c2526f5d5078de4b9e438d615 Mon Sep 17 00:00:00 2001 From: Sergiy Matusevych Date: Wed, 12 Mar 2025 16:04:31 -0700 Subject: [PATCH 04/35] minor docstring fixes --- .../mlos_bench/environments/__init__.py | 28 +++++++++++-------- 1 file changed, 17 insertions(+), 11 deletions(-) diff --git a/mlos_bench/mlos_bench/environments/__init__.py b/mlos_bench/mlos_bench/environments/__init__.py index 53036c7294..54e65f2388 100644 --- a/mlos_bench/mlos_bench/environments/__init__.py +++ b/mlos_bench/mlos_bench/environments/__init__.py @@ -42,28 +42,30 @@ Each :py:class:`~mlos_bench.environments.Environment` can have a set of parameters that define the environment's configuration. These parameters can be _constant_ (i.e., immutable from one trial -run to the next) or _tunable_ (i.e., suggested by the optimizer or provided by the user). -The following clauses in the environment configuration are used to declare these parameters: +run to the next) or _tunable_ (i.e., suggested by the optimizer or provided by the user). The +following clauses in the environment configuration are used to declare these parameters: -- ``tunable_params``: A list of tunable parameters' _groups_. At each trial, the environment will - obtain the new values of these parameters from the outside (e.g., from the optimizer). +- ``tunable_params``: + A list of tunable parameters' _groups_. At each trial, the environment will obtain the new + values of these parameters from the outside (e.g., from the optimizer). -- ``const_args``: A dictionary of _constant_ parameters along with their values. +- ``const_args``: + A dictionary of _constant_ parameters along with their values. -- ``required_args``: A list of _constant_ parameters supplied to the environment externally +- ``required_args``: + A list of _constant_ parameters supplied to the environment externally (i.e., from a parent environment, global config file, or command line). -That is, tunable parameters change on every trial, while constant parameters stay fixed for the +Again, tunable parameters change on every trial, while constant parameters stay fixed for the entire experiment. During the setup and run phases, MLOS will combine the constant and tunable parameters and their values into a single dictionary and pass it to the corresponding method. Values of constant parameters defined in the environment config can be overridden with the values -from the command line and/or external config files. -That allows MLOS users to have reusable immutable environment configurations and move all -experiment-specific or sensitive data outside of the version-controlled files. -We discuss the parameter propagation mechanism in the section below. +from the command line and/or external config files. That allows MLOS users to have reusable +immutable environment configurations and move all experiment-specific or sensitive data outside of +the version-controlled files. We discuss the parameter propagation mechanism in the section below. Environment Tunables ++++++++++++++++++++ @@ -109,6 +111,10 @@ ``required_args`` sections can be viewed as placeholders for the parameters that are being pushed to the environment from the outside. +The same parameter can be present in both ``const_args`` and ``required_args`` sections. +``required_args`` is just a way to emphasize the importance of the parameter and create a +placeholder for it when no default value can be specified the ``const_args`` section. + Variable replacement happens in the bottom-up manner. That is, if a certain parameter is present in the parent (composite) environment, it will replace the corresponding parameter in the child, and so on. Note that the parameter _must_ appear in the child environment ``const_args`` or From ccda090bf43164c0538da04880c2bb6b8b9a981f Mon Sep 17 00:00:00 2001 From: Brian Kroth Date: Thu, 13 Mar 2025 13:41:58 -0500 Subject: [PATCH 05/35] crossref resolution fixups --- mlos_bench/mlos_bench/environments/__init__.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mlos_bench/mlos_bench/environments/__init__.py b/mlos_bench/mlos_bench/environments/__init__.py index 54e65f2388..ac1a8ee69c 100644 --- a/mlos_bench/mlos_bench/environments/__init__.py +++ b/mlos_bench/mlos_bench/environments/__init__.py @@ -34,13 +34,13 @@ See below for the set of Environments currently available in this package. Note that additional ones can also be created by extending the base -:py:class:`.Environment` class and referencing them in the :py:mod:`json configs +:py:class:`~.Environment` class and referencing them in the :py:mod:`json configs ` using the ``class`` key. Environment Parameterization ++++++++++++++++++++++++++++ -Each :py:class:`~mlos_bench.environments.Environment` can have a set of parameters that define the +Each :py:class:`~.Environment` can have a set of parameters that define the environment's configuration. These parameters can be _constant_ (i.e., immutable from one trial run to the next) or _tunable_ (i.e., suggested by the optimizer or provided by the user). The following clauses in the environment configuration are used to declare these parameters: @@ -132,7 +132,7 @@ tokens, etc.) that should not be version-controlled. The global config files are specified in the ``globals`` section of the top-level CLI config, or in the ``--globals`` command line parameter. -Finally, any global or :py:class:`~mlos_bench.environments.Environment` parameter can be +Finally, any global or :py:class:`~.Environment` parameter can be overridden from the command line, by simply specifying ``--PARAMETER_NAME PARAMETER_VALUE``. We can summarize the parameter propagation rules as follows: From 9f8826f56cb4eb71dadda70b665dccedca439dd7 Mon Sep 17 00:00:00 2001 From: Brian Kroth Date: Thu, 13 Mar 2025 13:49:54 -0500 Subject: [PATCH 06/35] add another crossref link --- mlos_bench/mlos_bench/config/__init__.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/mlos_bench/mlos_bench/config/__init__.py b/mlos_bench/mlos_bench/config/__init__.py index 508d09c7a4..aac6b65e5d 100644 --- a/mlos_bench/mlos_bench/config/__init__.py +++ b/mlos_bench/mlos_bench/config/__init__.py @@ -199,7 +199,8 @@ "location": "eastus", } -There are additional details about variable propagation in the +There are additional details about `Variable Propagation +<../environments/index.html#variable-propagation>`_ in the :py:mod:`mlos_bench.environments` module. Well Known Variables From 1972c161bffeaf14595ce6c2c95062a13ba2834a Mon Sep 17 00:00:00 2001 From: Brian Kroth Date: Thu, 13 Mar 2025 14:13:15 -0500 Subject: [PATCH 07/35] Add some comments regarding tunables_params_map --- mlos_bench/mlos_bench/config/__init__.py | 23 ++++++++++++++++++- .../environments/remote/remote_env.py | 2 +- mlos_bench/mlos_bench/tunables/__init__.py | 3 ++- 3 files changed, 25 insertions(+), 3 deletions(-) diff --git a/mlos_bench/mlos_bench/config/__init__.py b/mlos_bench/mlos_bench/config/__init__.py index aac6b65e5d..e6fd18a172 100644 --- a/mlos_bench/mlos_bench/config/__init__.py +++ b/mlos_bench/mlos_bench/config/__init__.py @@ -197,6 +197,20 @@ "current_dir": "$PWD", "some_expanded_var": "$some_var: $experiment_id", "location": "eastus", + + // This can be specified in the CLI config or the globals config + "tunable_params_map": { + // a map of tunable_params variables to their covariant group names + "environment1_tunables": [ + "covariant_group_name", + "another_covariant_group_name" + ], + "environment2_tunables": [ + // empty list means no tunables + // are enabled for this environment + // during this experiment + // (e.g., only use defaults for this environment) + ], } There are additional details about `Variable Propagation @@ -217,6 +231,13 @@ - ``$trial_runner_id``: A unique identifier for the ``TrialRunner``. This can be useful when running multiple trials in parallel (e.g., to provision a numbered VM per worker). +- ``$tunable_params_map``: A map of ``tunable_params`` ``$name`` to their list of covariant group names. + This is usually used in a CLI ``--config`` CLI config or ``--globals`` + (e.g., "experiment") config file and is used to control what the + ``"tunable_params": $tunable_group_name`` specified in the the + :py:mod:`mlos_bench.environments` JSONC configs resolves to. + This can be used to control which tunables are enabled for tuning for an + experiment without having to change the underlying Environment config. Tunable Configs ^^^^^^^^^^^^^^^ @@ -354,4 +375,4 @@ `mlos_bench/tests/config/README.md `_ for additional documentation and examples in the source tree. -""" +""" # pylint: disable=line-too-long # noqa: E501 diff --git a/mlos_bench/mlos_bench/environments/remote/remote_env.py b/mlos_bench/mlos_bench/environments/remote/remote_env.py index d74b7fbbbf..2e08a64b19 100644 --- a/mlos_bench/mlos_bench/environments/remote/remote_env.py +++ b/mlos_bench/mlos_bench/environments/remote/remote_env.py @@ -7,7 +7,7 @@ e.g. Application Environment -TODO: Documentat how variable propogation works in the remote environments. +TODO: Document how variable propogation works in the remote environments. """ import logging diff --git a/mlos_bench/mlos_bench/tunables/__init__.py b/mlos_bench/mlos_bench/tunables/__init__.py index 0e163d4b1a..ae6bbc3187 100644 --- a/mlos_bench/mlos_bench/tunables/__init__.py +++ b/mlos_bench/mlos_bench/tunables/__init__.py @@ -93,7 +93,8 @@ Environment config. Then individual covariant groups can be enabled via the ``tunable_params`` and -``tunable_params_map`` properties, possibly via ``globals`` variable expansion. +``tunable_params_map`` properties, possibly via ``globals`` `Variable Expansion +<../config/index.html#globals-and-variable-substitution>`_. See the :py:mod:`mlos_bench.config` and :py:mod:`mlos_bench.environments` module documentation for more information. From cf5e1f93fea7fcfba146c39eca72dad9b448604b Mon Sep 17 00:00:00 2001 From: Brian Kroth Date: Thu, 13 Mar 2025 14:26:54 -0500 Subject: [PATCH 08/35] reformatting and restructuring --- mlos_bench/mlos_bench/config/__init__.py | 17 +++++ .../mlos_bench/environments/__init__.py | 67 ++++++++++--------- 2 files changed, 52 insertions(+), 32 deletions(-) diff --git a/mlos_bench/mlos_bench/config/__init__.py b/mlos_bench/mlos_bench/config/__init__.py index e6fd18a172..529c292c2a 100644 --- a/mlos_bench/mlos_bench/config/__init__.py +++ b/mlos_bench/mlos_bench/config/__init__.py @@ -213,6 +213,23 @@ ], } +Users can have multiple global config files, each specified with a ``--globals`` +CLI arg or ``"globals"`` CLI config property. + + +At runtime, parameters from these files will be combined into a single +dictionary, in the order they appear, and pushed to the root +:py:class:`~mlos_bench.environments.Environment`. + +Any global or :py:class:`~.Environment` parameter can also be overridden from +the command line, by simply specifying ``--PARAMETER_NAME PARAMETER_VALUE``. + +Another common use of global config files is to store sensitive data (e.g., +passwords, tokens, etc.) that should not be version-controlled. + +This way, users can keep their experiment-specific parameters separately from +the Environment configs making them more reusable. + There are additional details about `Variable Propagation <../environments/index.html#variable-propagation>`_ in the :py:mod:`mlos_bench.environments` module. diff --git a/mlos_bench/mlos_bench/environments/__init__.py b/mlos_bench/mlos_bench/environments/__init__.py index ac1a8ee69c..5d94a8ba06 100644 --- a/mlos_bench/mlos_bench/environments/__init__.py +++ b/mlos_bench/mlos_bench/environments/__init__.py @@ -105,41 +105,40 @@ Variable Propagation ++++++++++++++++++++ -Parameters declared in the ``const_args`` or ``required_args`` sections of the environment config -can be overridden with values of the corresponding parameters of the parent environment or -specified in the external config files or the command line. In fact, ``const_args`` or -``required_args`` sections can be viewed as placeholders for the parameters that are being pushed -to the environment from the outside. +Parameters declared in the ``const_args`` or ``required_args`` sections of the +Environment config can be overridden with values of the corresponding parameters +of the parent Environment or specified in the external config files or the +command line. + +In fact, ``const_args`` or ``required_args`` sections can be viewed as +placeholders for the parameters that are being pushed to the environment from +the outside. The same parameter can be present in both ``const_args`` and ``required_args`` sections. ``required_args`` is just a way to emphasize the importance of the parameter and create a placeholder for it when no default value can be specified the ``const_args`` section. -Variable replacement happens in the bottom-up manner. That is, if a certain parameter is present -in the parent (composite) environment, it will replace the corresponding parameter in the child, -and so on. Note that the parameter _must_ appear in the child environment ``const_args`` or -``required_args`` section; if a parameter is not present in one of these placeholders of the child -environment config, it will not be propagated. This hierarchy allows MLOS users to have small -immutable environment configurations at the lower levels and combine and parameterize them at the -higher levels. - -Taking it to the next level outside of the environment configs, the parameters can be defined in -the external key-value JSON config files (usually referred to as "global config files" in MLOS -lingo). Users can have multiple global config files; at runtime, parameters from these files will -be combined into a single dictionary and pushed to the root environment. This way, users can keep -their experiment-specific parameters separately from the environment configs making them more -reusable. Another common use of global config files is to store sensitive data (e.g., passwords, -tokens, etc.) that should not be version-controlled. The global config files are specified in the -``globals`` section of the top-level CLI config, or in the ``--globals`` command line parameter. - -Finally, any global or :py:class:`~.Environment` parameter can be -overridden from the command line, by simply specifying ``--PARAMETER_NAME PARAMETER_VALUE``. +Variable replacement happens in the bottom-up manner. That is, if a certain +parameter is present in the parent (:py:class:`~.CompositeEnv`) Environment, it +will replace the corresponding parameter in the child, and so on. + +Note that the parameter _must_ appear in the child Environment ``const_args`` or +``required_args`` section; if a parameter is not present in one of these +placeholders of the child Environment config, it will not be propagated. This +hierarchy allows MLOS users to have small immutable Environment configurations +at the lower levels and combine and parameterize them at the higher levels. + +Taking it to the next level outside of the Environment configs, the parameters +can be defined in the external key-value JSON config files (usually referred to +as `global config files +<../config/index.html#globals-and-variable-substitution>`_ in MLOS lingo). +See :py:mod:`mlos_bench.config` for more details. We can summarize the parameter propagation rules as follows: 1. Child environment will only get the parameters defined in its ``const_args`` or ``required_args`` sections. -2. Value of the parameter defined in the ``const_args`` section of the parent environment will +2. Value of the parameter defined in the ``const_args`` section of the parent Environment will override the value of the corresponding parameter in the child environments. 3. Values of the parameters defined in the global config files will override the values of the corresponding parameters in all environments. @@ -160,14 +159,18 @@ used in different settings (e.g., local machine, SSH accessible machine, Azure VM, etc.) without having to change the Environment config. -Variable propagation rules described in the previous section for the environment configs also -apply to the service configurations. That is, every parameter defined in the service config can be -overridden by a corresponding parameter from the global config or the command line. +Variable propagation rules described in the previous section for the environment +configs also apply to the :py:mod:`Service ` +configurations. + +That is, every parameter defined in the Service config can be overridden by a +corresponding parameter from the global config or the command line. -All global configs, command line parameters, environment ``const_args`` and ``required_args`` -sections, and service config parameters thus form one flat name space of parameters. This imposes -a certain risk of name clashes, but also simplifies the configuration process and allows users to -keep all experiment-specific data in a few human-readable files. +All global configs, command line parameters, Environment ``const_args`` and +``required_args`` sections, and Service config parameters thus form one flat +name space of parameters. This imposes a certain risk of name clashes, but also +simplifies the configuration process and allows users to keep all +experiment-specific data in a few human-readable files. We will discuss the examples of such global and local configuration parameters in the documentation of the concrete :py:mod:`~mlos_bench.services` and From e6d83f8237ee02abe8381ca5e9b31c43511db541 Mon Sep 17 00:00:00 2001 From: Brian Kroth Date: Thu, 13 Mar 2025 14:29:10 -0500 Subject: [PATCH 09/35] fixups --- mlos_bench/mlos_bench/config/__init__.py | 3 +-- mlos_bench/mlos_bench/environments/remote/remote_env.py | 2 +- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/mlos_bench/mlos_bench/config/__init__.py b/mlos_bench/mlos_bench/config/__init__.py index 529c292c2a..9743e5e104 100644 --- a/mlos_bench/mlos_bench/config/__init__.py +++ b/mlos_bench/mlos_bench/config/__init__.py @@ -216,10 +216,9 @@ Users can have multiple global config files, each specified with a ``--globals`` CLI arg or ``"globals"`` CLI config property. - At runtime, parameters from these files will be combined into a single dictionary, in the order they appear, and pushed to the root -:py:class:`~mlos_bench.environments.Environment`. +:py:class:`Environment `. Any global or :py:class:`~.Environment` parameter can also be overridden from the command line, by simply specifying ``--PARAMETER_NAME PARAMETER_VALUE``. diff --git a/mlos_bench/mlos_bench/environments/remote/remote_env.py b/mlos_bench/mlos_bench/environments/remote/remote_env.py index 2e08a64b19..184df75cf6 100644 --- a/mlos_bench/mlos_bench/environments/remote/remote_env.py +++ b/mlos_bench/mlos_bench/environments/remote/remote_env.py @@ -7,7 +7,7 @@ e.g. Application Environment -TODO: Document how variable propogation works in the remote environments. +TODO: Document how variable propagation works in the remote environments. """ import logging From 3b0061864ebd94db8efb2318b0adc2206dccc29e Mon Sep 17 00:00:00 2001 From: Brian Kroth Date: Thu, 13 Mar 2025 14:37:55 -0500 Subject: [PATCH 10/35] rst syntax instead of markdown --- mlos_bench/mlos_bench/environments/__init__.py | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/mlos_bench/mlos_bench/environments/__init__.py b/mlos_bench/mlos_bench/environments/__init__.py index 5d94a8ba06..cbf3fc788a 100644 --- a/mlos_bench/mlos_bench/environments/__init__.py +++ b/mlos_bench/mlos_bench/environments/__init__.py @@ -41,8 +41,8 @@ ++++++++++++++++++++++++++++ Each :py:class:`~.Environment` can have a set of parameters that define the -environment's configuration. These parameters can be _constant_ (i.e., immutable from one trial -run to the next) or _tunable_ (i.e., suggested by the optimizer or provided by the user). The +environment's configuration. These parameters can be *constant* (i.e., immutable from one trial +run to the next) or *tunable* (i.e., suggested by the optimizer or provided by the user). The following clauses in the environment configuration are used to declare these parameters: - ``tunable_params``: @@ -50,10 +50,10 @@ values of these parameters from the outside (e.g., from the optimizer). - ``const_args``: - A dictionary of _constant_ parameters along with their values. + A dictionary of *constant* parameters along with their values. - ``required_args``: - A list of _constant_ parameters supplied to the environment externally + A list of *constant* parameters supplied to the environment externally (i.e., from a parent environment, global config file, or command line). Again, tunable parameters change on every trial, while constant parameters stay fixed for the @@ -122,7 +122,7 @@ parameter is present in the parent (:py:class:`~.CompositeEnv`) Environment, it will replace the corresponding parameter in the child, and so on. -Note that the parameter _must_ appear in the child Environment ``const_args`` or +Note that the parameter **must** appear in the child Environment ``const_args`` or ``required_args`` section; if a parameter is not present in one of these placeholders of the child Environment config, it will not be propagated. This hierarchy allows MLOS users to have small immutable Environment configurations From aad03815b02e56a27688be54400e34ca6587140a Mon Sep 17 00:00:00 2001 From: Brian Kroth Date: Thu, 13 Mar 2025 14:38:04 -0500 Subject: [PATCH 11/35] more cross refs --- mlos_bench/mlos_bench/environments/__init__.py | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/mlos_bench/mlos_bench/environments/__init__.py b/mlos_bench/mlos_bench/environments/__init__.py index cbf3fc788a..eb51b142ab 100644 --- a/mlos_bench/mlos_bench/environments/__init__.py +++ b/mlos_bench/mlos_bench/environments/__init__.py @@ -46,8 +46,13 @@ following clauses in the environment configuration are used to declare these parameters: - ``tunable_params``: - A list of tunable parameters' _groups_. At each trial, the environment will obtain the new - values of these parameters from the outside (e.g., from the optimizer). + A list of :py:mod:`tunable ` parameters' (covariant) *groups*. + At each trial, the Environment will obtain the new values of these parameters + from the outside (e.g., from the :py:mod:`Optimizer `). + + Typically, this is set using variable expansion via the special + ``tunable_params_map`` key in the `globals config + <../config/index.html#globals-and-variable-substitution>`_. - ``const_args``: A dictionary of *constant* parameters along with their values. From 397c6eeec6d9f96deb7f542a909f8ac52bcfa4f6 Mon Sep 17 00:00:00 2001 From: Brian Kroth Date: Thu, 13 Mar 2025 14:45:39 -0500 Subject: [PATCH 12/35] tweaks --- mlos_bench/mlos_bench/environments/__init__.py | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/mlos_bench/mlos_bench/environments/__init__.py b/mlos_bench/mlos_bench/environments/__init__.py index eb51b142ab..be09ec54d0 100644 --- a/mlos_bench/mlos_bench/environments/__init__.py +++ b/mlos_bench/mlos_bench/environments/__init__.py @@ -64,13 +64,16 @@ Again, tunable parameters change on every trial, while constant parameters stay fixed for the entire experiment. -During the setup and run phases, MLOS will combine the constant and tunable parameters and their -values into a single dictionary and pass it to the corresponding method. - -Values of constant parameters defined in the environment config can be overridden with the values -from the command line and/or external config files. That allows MLOS users to have reusable -immutable environment configurations and move all experiment-specific or sensitive data outside of -the version-controlled files. We discuss the parameter propagation mechanism in the section below. +During the ``setup`` and ``run`` phases, MLOS will combine the constant and +tunable parameters and their values into a single dictionary and pass it to the +corresponding method. + +Values of constant parameters defined in the Environment config can be +overridden with the values from the command line and/or external config files. +That allows MLOS users to have reusable immutable environment configurations and +move all experiment-specific or sensitive data outside of the version-controlled +files. We discuss the `variable propagation <#variable-propagation>`_ mechanism +in the section below. Environment Tunables ++++++++++++++++++++ From cb5888598647005f72187548642da0198494ceb8 Mon Sep 17 00:00:00 2001 From: Brian Kroth Date: Thu, 13 Mar 2025 14:48:20 -0500 Subject: [PATCH 13/35] tweaks --- mlos_bench/mlos_bench/environments/__init__.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/mlos_bench/mlos_bench/environments/__init__.py b/mlos_bench/mlos_bench/environments/__init__.py index be09ec54d0..1035e3ac2d 100644 --- a/mlos_bench/mlos_bench/environments/__init__.py +++ b/mlos_bench/mlos_bench/environments/__init__.py @@ -125,6 +125,9 @@ The same parameter can be present in both ``const_args`` and ``required_args`` sections. ``required_args`` is just a way to emphasize the importance of the parameter and create a placeholder for it when no default value can be specified the ``const_args`` section. +If a ``required_args`` parameter is not present in the ``const_args`` section, +and can't be resolved from the ``globals`` this allows MLOS to fail fast and +return an error to the user indicating an incomplete config. Variable replacement happens in the bottom-up manner. That is, if a certain parameter is present in the parent (:py:class:`~.CompositeEnv`) Environment, it @@ -146,8 +149,9 @@ 1. Child environment will only get the parameters defined in its ``const_args`` or ``required_args`` sections. -2. Value of the parameter defined in the ``const_args`` section of the parent Environment will - override the value of the corresponding parameter in the child environments. +2. The value of the parameter defined in the ``const_args`` section of the + parent Environment will override the value of the corresponding parameter in the + child Environments. 3. Values of the parameters defined in the global config files will override the values of the corresponding parameters in all environments. 4. Values of the command line parameters take precedence over values defined in the global or From eb31797afd777a84ce3276d60b8a0bd72563d019 Mon Sep 17 00:00:00 2001 From: Brian Kroth Date: Thu, 13 Mar 2025 15:23:09 -0500 Subject: [PATCH 14/35] incorrect and incomplete example --- .../mlos_bench/environments/__init__.py | 80 ++++++++++++++++++- 1 file changed, 79 insertions(+), 1 deletion(-) diff --git a/mlos_bench/mlos_bench/environments/__init__.py b/mlos_bench/mlos_bench/environments/__init__.py index 1035e3ac2d..c7e2bfe254 100644 --- a/mlos_bench/mlos_bench/environments/__init__.py +++ b/mlos_bench/mlos_bench/environments/__init__.py @@ -157,6 +157,84 @@ 4. Values of the command line parameters take precedence over values defined in the global or environment configs. +Examples +^^^^^^^^ + +Here's a simple working example of a local environment config (written in Python +instead of JSON for testing) to show how variable propogation works: + + +>>> globals_json = ''' +... { +... "required_arg": "required_arg_from_globals_value", +... "const_arg": "const_arg1_from_globals_value", +... "tunable_params_map": { +... "my_env1_tunables": ["dummy_params"], +... "my_env2_tunables": [/* none */], +... }, +... } +... ''' +>>> composite_env_json = ''' +... { +... "class": "mlos_bench.environments.composite_env.CompositeEnv", +... "name": "parent_env", +... "config": { +... // Must be populated by a global config or command line: +... "required_args": ["required_arg"], +... // Can be populate by variable expansion from a higher level, or else overridden here. +... "const_args": { +... "const1_arg": "$required_arg", +... "const2_arg": "const_arg2_from_parent_value", +... }, +... "children": [ +... { +... "class": "mlos_bench.environments.local.local_env.LocalEnv", +... "name": "child_env1", +... "config": { +... "const_args": { +... "const1_arg": "const_arg1_from_env1_value", +... "const2_arg": "const_arg2_from_env1_value", +... }, +... "required_args": ["required_arg"], +... "include_tunables": [ +... "tunables/dummy-tunables.jsonc", +... ], +... "tunable_params": "$my_env_tunables", +... }, +... }, +... { +... "class": "mlos_bench.environments.local.local_env.LocalEnv", +... "name": "child_env2", +... "config": { +... "required_args": ["required_arg", "const_arg"], +... "const_args": { +... "const_arg": "$const_arg", +... }, +... "include_tunables": [ +... "tunables/dummy-tunables.jsonc", +... ], +... "tunable_params": "$my_env2_tunables", +... }, +... }, +... ], +... } +... } +... ''' +>>> from mlos_bench.services.config_persistence import ConfigPersistenceService +>>> from mlos_bench.config.schemas.config_schemas import ConfigSchema +>>> config_loader_service = ConfigPersistenceService() +>>> globals_config = config_loader_service.load_config(globals_json, ConfigSchema.GLOBALS) +>>> composite_env_config = config_loader_service.load_config(composite_env_json, ConfigSchema.ENVIRONMENT) +>>> composite_env_config = config_loader_service.load_environment(composite_env_config, globals_config) +>>> composite_env_config.const_args["const_arg"] +'const_arg_from_env1_value' +>>> composite_env_config.required_args["required_arg"] +'required_arg_from_env1_value' +>>> composite_env_config.tunable_params["my_env_tunables"].tunable_groups +[0].name +>>> composite_env_config.tunable_params["my_env_tunables"].tunable_groups[0].name +'group1' + Environment Services ++++++++++++++++++++ @@ -219,7 +297,7 @@ Overview of the Services available to the Environments and their configurations. :py:mod:`mlos_bench.tunables` : Overview of the Tunables available to the Environments and their configurations. -""" +""" # pylint: disable=line-too-long # noqa: E501 from mlos_bench.environments.base_environment import Environment from mlos_bench.environments.composite_env import CompositeEnv From f15cb6d74113fdc4b38b2b6cb78c698c2b63b58d Mon Sep 17 00:00:00 2001 From: Brian Kroth Date: Thu, 13 Mar 2025 15:49:29 -0500 Subject: [PATCH 15/35] link fixup --- mlos_bench/mlos_bench/environments/__init__.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mlos_bench/mlos_bench/environments/__init__.py b/mlos_bench/mlos_bench/environments/__init__.py index c7e2bfe254..fdc6ddd3f3 100644 --- a/mlos_bench/mlos_bench/environments/__init__.py +++ b/mlos_bench/mlos_bench/environments/__init__.py @@ -72,7 +72,7 @@ overridden with the values from the command line and/or external config files. That allows MLOS users to have reusable immutable environment configurations and move all experiment-specific or sensitive data outside of the version-controlled -files. We discuss the `variable propagation <#variable-propagation>`_ mechanism +files. We discuss the `variable propagation `_ mechanism in the section below. Environment Tunables From 2d062a28faba03f739509991dd81cc7e66bf4d38 Mon Sep 17 00:00:00 2001 From: Brian Kroth Date: Thu, 13 Mar 2025 15:49:38 -0500 Subject: [PATCH 16/35] cross ref --- mlos_bench/mlos_bench/environments/__init__.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/mlos_bench/mlos_bench/environments/__init__.py b/mlos_bench/mlos_bench/environments/__init__.py index fdc6ddd3f3..ecc6bab036 100644 --- a/mlos_bench/mlos_bench/environments/__init__.py +++ b/mlos_bench/mlos_bench/environments/__init__.py @@ -161,8 +161,11 @@ ^^^^^^^^ Here's a simple working example of a local environment config (written in Python -instead of JSON for testing) to show how variable propogation works: +instead of JSON for testing) to show how variable propagation works: +Note: this references the `dummy-tunables.jsonc +`_ +file for simplicity. >>> globals_json = ''' ... { From 8b6913029b1ef3b84f4f1a8244f358dae14bf798 Mon Sep 17 00:00:00 2001 From: Brian Kroth Date: Thu, 13 Mar 2025 15:49:47 -0500 Subject: [PATCH 17/35] fixups --- .../mlos_bench/environments/__init__.py | 89 ++++++++++++------- 1 file changed, 59 insertions(+), 30 deletions(-) diff --git a/mlos_bench/mlos_bench/environments/__init__.py b/mlos_bench/mlos_bench/environments/__init__.py index ecc6bab036..9651bf0105 100644 --- a/mlos_bench/mlos_bench/environments/__init__.py +++ b/mlos_bench/mlos_bench/environments/__init__.py @@ -183,43 +183,72 @@ ... "name": "parent_env", ... "config": { ... // Must be populated by a global config or command line: -... "required_args": ["required_arg"], +... "required_args": [ +... "required_arg" +... ], ... // Can be populate by variable expansion from a higher level, or else overridden here. ... "const_args": { ... "const1_arg": "$required_arg", -... "const2_arg": "const_arg2_from_parent_value", +... "const2_arg": "const_arg2_from_parent_value" ... }, ... "children": [ ... { -... "class": "mlos_bench.environments.local.local_env.LocalEnv", -... "name": "child_env1", -... "config": { -... "const_args": { -... "const1_arg": "const_arg1_from_env1_value", -... "const2_arg": "const_arg2_from_env1_value", -... }, -... "required_args": ["required_arg"], -... "include_tunables": [ -... "tunables/dummy-tunables.jsonc", -... ], -... "tunable_params": "$my_env_tunables", -... }, -... }, +... "class": "mlos_bench.environments.local.local_env.LocalEnv", +... "name": "child_env1", +... "include_tunables": [ +... "tunables/dummy-tunables.jsonc" +... ], +... "config": { +... "tunable_params": ["$my_env_tunables"], +... "const_args": { +... "const1_arg": "const_arg1_from_env1_value", +... "const2_arg": "const_arg2_from_env1_value" +... }, +... "required_args": [ +... "required_arg" +... ], +... // Expose the args as shell env vars for the child env. +... "shell_env_params": [ +... "required_arg", +... "const1_arg", +... "const2_arg" +... ], +... "run": [ +... "echo 'required_arg: $required_arg'", +... "echo 'const1_arg: $const1_arg'", +... "echo 'const2_arg: $const2_arg'" +... ] +... } +... }, ... { -... "class": "mlos_bench.environments.local.local_env.LocalEnv", -... "name": "child_env2", -... "config": { -... "required_args": ["required_arg", "const_arg"], -... "const_args": { -... "const_arg": "$const_arg", -... }, -... "include_tunables": [ -... "tunables/dummy-tunables.jsonc", -... ], -... "tunable_params": "$my_env2_tunables", -... }, -... }, -... ], +... "class": "mlos_bench.environments.local.local_env.LocalEnv", +... "name": "child_env2", +... "include_tunables": [ +... "tunables/dummy-tunables.jsonc" +... ], +... "config": { +... "tunable_params": ["$my_env2_tunables"], +... "required_args": [ +... "required_arg", +... "const_arg" +... ], +... "const_args": { +... "const_arg": "$const_arg" +... }, +... // Expose the args as shell env vars for the child env. +... "shell_env_params": [ +... "required_arg", +... "const1_arg", +... "const2_arg" +... ], +... "run": [ +... "echo 'required_arg: $required_arg'", +... "echo 'const1_arg: $const1_arg'", +... "echo 'const2_arg: $const2_arg'" +... ] +... } +... } +... ] ... } ... } ... ''' From 1ace92769b0357f46968283a343dd0d09159d4ac Mon Sep 17 00:00:00 2001 From: Brian Kroth Date: Thu, 13 Mar 2025 15:55:52 -0500 Subject: [PATCH 18/35] fixups --- mlos_bench/mlos_bench/environments/__init__.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/mlos_bench/mlos_bench/environments/__init__.py b/mlos_bench/mlos_bench/environments/__init__.py index 9651bf0105..81f5830b55 100644 --- a/mlos_bench/mlos_bench/environments/__init__.py +++ b/mlos_bench/mlos_bench/environments/__init__.py @@ -254,10 +254,11 @@ ... ''' >>> from mlos_bench.services.config_persistence import ConfigPersistenceService >>> from mlos_bench.config.schemas.config_schemas import ConfigSchema +>>> from mlos_bench.tunables import TunableGroups +>>> tunable_groups = TunableGroups() >>> config_loader_service = ConfigPersistenceService() >>> globals_config = config_loader_service.load_config(globals_json, ConfigSchema.GLOBALS) ->>> composite_env_config = config_loader_service.load_config(composite_env_json, ConfigSchema.ENVIRONMENT) ->>> composite_env_config = config_loader_service.load_environment(composite_env_config, globals_config) +>>> composite_env_config = config_loader_service.load_environment(composite_env_json, tunable_groups, globals_config) >>> composite_env_config.const_args["const_arg"] 'const_arg_from_env1_value' >>> composite_env_config.required_args["required_arg"] From 55faf60afd21c54cdf45f6785f08876ef4f81855 Mon Sep 17 00:00:00 2001 From: Brian Kroth Date: Thu, 13 Mar 2025 15:56:11 -0500 Subject: [PATCH 19/35] fixup for service=None --- mlos_bench/mlos_bench/services/config_persistence.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mlos_bench/mlos_bench/services/config_persistence.py b/mlos_bench/mlos_bench/services/config_persistence.py index 3fea50a7e6..a674e6ce75 100644 --- a/mlos_bench/mlos_bench/services/config_persistence.py +++ b/mlos_bench/mlos_bench/services/config_persistence.py @@ -457,6 +457,9 @@ def build_environment( if env_services_path is not None: service = self.load_services(env_services_path, global_config, service) + if service is None: + service = Service(parent=self) + env_tunables_path = config.get("include_tunables") if env_tunables_path is not None: tunables = self.load_tunables(env_tunables_path, tunables) From 8dd9ac4a8a4006b749e0ed41df5c8186e0d30efd Mon Sep 17 00:00:00 2001 From: Brian Kroth Date: Thu, 13 Mar 2025 16:08:16 -0500 Subject: [PATCH 20/35] wip more examples --- .../mlos_bench/environments/__init__.py | 23 ++++++++++++------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/mlos_bench/mlos_bench/environments/__init__.py b/mlos_bench/mlos_bench/environments/__init__.py index 81f5830b55..507ce8eea9 100644 --- a/mlos_bench/mlos_bench/environments/__init__.py +++ b/mlos_bench/mlos_bench/environments/__init__.py @@ -199,7 +199,7 @@ ... "tunables/dummy-tunables.jsonc" ... ], ... "config": { -... "tunable_params": ["$my_env_tunables"], +... "tunable_params": ["$my_env1_tunables"], ... "const_args": { ... "const1_arg": "const_arg1_from_env1_value", ... "const2_arg": "const_arg2_from_env1_value" @@ -230,10 +230,10 @@ ... "tunable_params": ["$my_env2_tunables"], ... "required_args": [ ... "required_arg", -... "const_arg" ... ], ... "const_args": { -... "const_arg": "$const_arg" +... "const_arg1": "$const_arg" +... // const_arg2 should not be propagated to this child ... }, ... // Expose the args as shell env vars for the child env. ... "shell_env_params": [ @@ -252,20 +252,27 @@ ... } ... } ... ''' +>>> # Import modules +>>> from mlos_bench.services.local.local_exec import LocalExecService >>> from mlos_bench.services.config_persistence import ConfigPersistenceService >>> from mlos_bench.config.schemas.config_schemas import ConfigSchema >>> from mlos_bench.tunables import TunableGroups +>>> # Do some basic setup that mlos_bench usually handles for us. >>> tunable_groups = TunableGroups() >>> config_loader_service = ConfigPersistenceService() +>>> service = LocalExecService(parent=config_loader_service) +>>> # Load the globals and environment configs defined above. >>> globals_config = config_loader_service.load_config(globals_json, ConfigSchema.GLOBALS) ->>> composite_env_config = config_loader_service.load_environment(composite_env_json, tunable_groups, globals_config) ->>> composite_env_config.const_args["const_arg"] +>>> composite_env_config = config_loader_service.load_environment(composite_env_json, tunable_groups, globals_config, service=service) +>>> # Now see how the variable propagation works. +>>> # TODO ... +>>> composite_env_config.children[0].parameters["const_arg1"] 'const_arg_from_env1_value' ->>> composite_env_config.required_args["required_arg"] +>>> composite_env_config.children[0].parameters["required_arg"] 'required_arg_from_env1_value' ->>> composite_env_config.tunable_params["my_env_tunables"].tunable_groups +>>> composite_env_config.tunable_params["my_env1_tunables"].tunable_groups [0].name ->>> composite_env_config.tunable_params["my_env_tunables"].tunable_groups[0].name +>>> composite_env_config.tunable_params["my_env1_tunables"].tunable_groups[0].name 'group1' Environment Services From 16ccf9e1a4a331768930d78eadffcb83ebb94333 Mon Sep 17 00:00:00 2001 From: Brian Kroth Date: Thu, 13 Mar 2025 16:21:06 -0500 Subject: [PATCH 21/35] tweaks --- .../mlos_bench/environments/__init__.py | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/mlos_bench/mlos_bench/environments/__init__.py b/mlos_bench/mlos_bench/environments/__init__.py index 507ce8eea9..8f5107fbc3 100644 --- a/mlos_bench/mlos_bench/environments/__init__.py +++ b/mlos_bench/mlos_bench/environments/__init__.py @@ -252,7 +252,9 @@ ... } ... } ... ''' ->>> # Import modules + +>>> # Load the globals and environment configs defined above. +>>> # Import the necessary modules to run the example. >>> from mlos_bench.services.local.local_exec import LocalExecService >>> from mlos_bench.services.config_persistence import ConfigPersistenceService >>> from mlos_bench.config.schemas.config_schemas import ConfigSchema @@ -261,18 +263,21 @@ >>> tunable_groups = TunableGroups() >>> config_loader_service = ConfigPersistenceService() >>> service = LocalExecService(parent=config_loader_service) ->>> # Load the globals and environment configs defined above. >>> globals_config = config_loader_service.load_config(globals_json, ConfigSchema.GLOBALS) >>> composite_env_config = config_loader_service.load_environment(composite_env_json, tunable_groups, globals_config, service=service) +>>> child_env1 = composite_env_config.children[0] +>>> assert child_env1.name == "child_env1" +>>> child_env2 = composite_env_config.children[1] +>>> assert child_env2.name == "child_env2" + >>> # Now see how the variable propagation works. ->>> # TODO ... ->>> composite_env_config.children[0].parameters["const_arg1"] +>>> child_env1.parameters["const_arg1"] 'const_arg_from_env1_value' ->>> composite_env_config.children[0].parameters["required_arg"] +>>> child_env1.parameters["required_arg"] 'required_arg_from_env1_value' ->>> composite_env_config.tunable_params["my_env1_tunables"].tunable_groups +>>> child_env1.tunable_params["my_env1_tunables"].tunable_groups [0].name ->>> composite_env_config.tunable_params["my_env1_tunables"].tunable_groups[0].name +>>> child_env1.tunable_params["my_env1_tunables"].tunable_groups[0].name 'group1' Environment Services From b5d77982f1b94c452b63c4dedeb66c70631fb756 Mon Sep 17 00:00:00 2001 From: Brian Kroth Date: Fri, 14 Mar 2025 17:02:57 -0500 Subject: [PATCH 22/35] wip: more docs --- .../mlos_bench/environments/__init__.py | 168 +++++++++++++----- 1 file changed, 119 insertions(+), 49 deletions(-) diff --git a/mlos_bench/mlos_bench/environments/__init__.py b/mlos_bench/mlos_bench/environments/__init__.py index 8f5107fbc3..79e4f00800 100644 --- a/mlos_bench/mlos_bench/environments/__init__.py +++ b/mlos_bench/mlos_bench/environments/__init__.py @@ -167,16 +167,32 @@ `_ file for simplicity. +>>> # globals.jsonc >>> globals_json = ''' ... { -... "required_arg": "required_arg_from_globals_value", -... "const_arg": "const_arg1_from_globals_value", +... "experiment_id": "VariablePropagationExample", +... +... // Required arguments must have their value set from globals, cli args, or shell env. +... "required_arg_from_globals": "required_arg_from_globals_val", +... "required_arg_from_cli": "require_arg_from_globals_NOT_CLI", // will be replaced by cli invocation +... "required_arg_from_shell_env": "$REQUIRED_ARG_FROM_SHELL_ENV", +... +... // Const args have a default value if not set, but can be overridden by +... // the globals, cli args, shell env, or parent env. +... +... "const_arg_from_globals": "const_arg_from_globals_val", +... "const_arg_from_shell_env": "$CONST_ARG_FROM_SHELL_ENV", +... "const_arg_from_child2_env2": "FROM@GLOBALS!", +... // special map of tunable_params_name to their set of enabled covariant tunable groups +... ... "tunable_params_map": { ... "my_env1_tunables": ["dummy_params"], ... "my_env2_tunables": [/* none */], ... }, ... } ... ''' + +>>> # composite_env.jsonc >>> composite_env_json = ''' ... { ... "class": "mlos_bench.environments.composite_env.CompositeEnv", @@ -184,12 +200,21 @@ ... "config": { ... // Must be populated by a global config or command line: ... "required_args": [ -... "required_arg" +... "required_arg_from_globals", +... "required_arg_from_cli", +... //"required_arg_from_shell_env", # TEST ME: does this need to be here to flow down? ... ], -... // Can be populate by variable expansion from a higher level, or else overridden here. +... // Can be populate by variable expansion from a higher level, or else defaulted to here. ... "const_args": { -... "const1_arg": "$required_arg", -... "const2_arg": "const_arg2_from_parent_value" +... "const_arg_from_globals": "const_arg_from_globals_parent_val", +... "const_arg_from_shell_env": "const_arg_from_shell_parent_val", +... "const_arg_from_cli": "const_arg_from_cli_parent_val", +... "const_arg_from_parent_env": "const_arg_from_parent_env_val", +... "const_arg_from_local_env": "const_arg_from_local_env_parent_val", +... // const_arg defaults can also use variable expansion to refer +... // to another variable previously defined (even another const_arg) in order to +... // allow for variable renaming +... "const_arg_from_required": "$required_arg_from_globals", ... }, ... "children": [ ... { @@ -200,23 +225,32 @@ ... ], ... "config": { ... "tunable_params": ["$my_env1_tunables"], -... "const_args": { -... "const1_arg": "const_arg1_from_env1_value", -... "const2_arg": "const_arg2_from_env1_value" -... }, ... "required_args": [ -... "required_arg" -... ], -... // Expose the args as shell env vars for the child env. -... "shell_env_params": [ -... "required_arg", -... "const1_arg", -... "const2_arg" +... "required_arg_from_globals", +... "required_arg_from_cli", +... "required_arg_from_shell_env", +... // Here, we can simply declare a required_arg as +... // required, but let it inherit a value from a higher level environment. +... "const_arg_from_required", +... "const_arg_from_parent_env", ... ], +... "const_args": { +... // Here we provide defaults, though all of these should be overridden by higher levels. +... "const_arg_from_globals": "const_arg_from_globals_child1_val", +... "const_arg_from_shell_env": "const_arg_from_shell_child1_val", +... "const_arg_from_cli": "const_arg_from_cli_child1_val", +... "const_arg_from_local_env": "const_arg_from_local_env_child1_val", +... }, ... "run": [ -... "echo 'required_arg: $required_arg'", -... "echo 'const1_arg: $const1_arg'", -... "echo 'const2_arg: $const2_arg'" +... "echo 'child1: required_arg_from_globals: $required_arg_from_globals'", +... "echo 'child1: required_arg_from_cli: $required_arg_from_cli'", +... "echo 'child1: required_arg_from_shell_env: $required_arg_from_shell_env'", +... "echo 'child1: const_arg_from_required: $const_arg_from_required'", +... "echo 'child1: const_arg_from_globals: $const_arg_from_globals'", +... "echo 'child1: const_arg_from_shell_env: $const_arg_from_shell_env'", +... "echo 'child1: const_arg_from_cli: $const_arg_from_cli'", +... "echo 'child1: const_arg_from_local_env: $const_arg_from_local_env'", +... "echo 'child1: const_arg_from_parent_env: $const_arg_from_parent_env'", ... ] ... } ... }, @@ -229,22 +263,43 @@ ... "config": { ... "tunable_params": ["$my_env2_tunables"], ... "required_args": [ -... "required_arg", +... "required_arg_from_globals", +... "required_arg_from_cli", +... "required_arg_from_shell_env", +... // Here, we can simply declare a required_arg as +... // required, but let it inherit a value from a higher level environment. +... "const_arg_from_required", +... "const_arg_from_parent_env", ... ], ... "const_args": { -... "const_arg1": "$const_arg" -... // const_arg2 should not be propagated to this child +... // Here we provide defaults, though all of these should be overridden by higher levels. +... "const_arg_from_globals": "const_arg_from_globals_child2_val", +... "const_arg_from_shell_env": "const_arg_from_shell_child2_val", +... "const_arg_from_cli": "const_arg_from_cli_child2_val", +... "const_arg_from_local_env": "const_arg_from_local_env_child2_val", +... "const_arg_from_child2_env1": "const_arg_from_child2_val", +... "const_arg_from_child2_env2": "const_arg_from_child2_val", ... }, ... // Expose the args as shell env vars for the child env. ... "shell_env_params": [ -... "required_arg", -... "const1_arg", -... "const2_arg" +... "required_arg_from_globals", +... "const_arg_from_globals", ... ], ... "run": [ -... "echo 'required_arg: $required_arg'", -... "echo 'const1_arg: $const1_arg'", -... "echo 'const2_arg: $const2_arg'" +... // Each of these commands undergoes variable replacement prior to being executed. +... "echo 'child2: required_arg_from_globals: $required_arg_from_globals'", +... "echo 'child2: required_arg_from_cli: $required_arg_from_cli'", +... "echo 'child2: required_arg_from_shell_env: $required_arg_from_shell_env'", +... "echo 'child2: const_arg_from_required: $const_arg_from_required'", +... "echo 'child2: const_arg_from_globals: $const_arg_from_globals'", +... "echo 'child2: const_arg_from_shell_env: $const_arg_from_shell_env'", +... "echo 'child2: const_arg_from_cli: $const_arg_from_cli'", +... "echo 'child2: const_arg_from_local_env: $const_arg_from_local_env'", +... "echo 'child2: const_arg_from_child2_env1: $const_arg_from_child2_env1'", +... "echo 'child2: const_arg_from_child2_env2: $const_arg_from_child2_env2'", +... "echo 'child2: const_arg_from_parent_env: $const_arg_from_parent_env'", +... // Only some of those parameters are actually exposed as shell env vars though. +... "printenv | grep _arg_from_", ... ] ... } ... } @@ -253,33 +308,48 @@ ... } ... ''' ->>> # Load the globals and environment configs defined above. ->>> # Import the necessary modules to run the example. ->>> from mlos_bench.services.local.local_exec import LocalExecService ->>> from mlos_bench.services.config_persistence import ConfigPersistenceService ->>> from mlos_bench.config.schemas.config_schemas import ConfigSchema ->>> from mlos_bench.tunables import TunableGroups ->>> # Do some basic setup that mlos_bench usually handles for us. ->>> tunable_groups = TunableGroups() ->>> config_loader_service = ConfigPersistenceService() ->>> service = LocalExecService(parent=config_loader_service) ->>> globals_config = config_loader_service.load_config(globals_json, ConfigSchema.GLOBALS) ->>> composite_env_config = config_loader_service.load_environment(composite_env_json, tunable_groups, globals_config, service=service) ->>> child_env1 = composite_env_config.children[0] +>>> # Setup the shell env as if bash used an "export VAR='val'" +>>> import os +>>> os.environ["REQUIRED_ARG_FROM_SHELL_ENV"] = "required_arg_from_shell_env_val" +>>> os.environ["CONST_ARG_FROM_SHELL_ENV"] = "const_arg_from_shell_env_val" +>>> # Load the globals and environment configs defined above via the Launcher as +>>> # if we were calling `mlos_bench` directly on the CLI. +>>> from mlos_bench.launcher import Launcher +>>> argv = [ +... "--log-level=WARNING", +... "--globals", globals_json, +... "--environment", composite_env_json, +... # Override some values via CLI directly: +... "--required_arg_from_cli", "required_arg_from_cli_val", +... "--const_arg_from_cli", "const_arg_from_cli_val", +... ] +>>> launcher = Launcher("sample_launcher", argv=argv) +>>> composite_env = launcher.root_environment +>>> child_env1 = composite_env.children[0] >>> assert child_env1.name == "child_env1" ->>> child_env2 = composite_env_config.children[1] +>>> child_env2 = composite_env.children[1] >>> assert child_env2.name == "child_env2" ->>> # Now see how the variable propagation works. ->>> child_env1.parameters["const_arg1"] -'const_arg_from_env1_value' ->>> child_env1.parameters["required_arg"] -'required_arg_from_env1_value' ->>> child_env1.tunable_params["my_env1_tunables"].tunable_groups +>>> # Demonstrate how tunable parameters are selected. +>>> child_env1.tunable_params +>>> child_env1.tunable_params["dummy_params"] [0].name >>> child_env1.tunable_params["my_env1_tunables"].tunable_groups[0].name 'group1' +>>> # Now see how the variable propagation works. +>>> child_env1.parameters["required_arg_from_globals"] +'required_arg_from_globals_val' +>>> child_env1.parameters["required_arg_from_cli"] +'required_arg_from_cli_val' +>>> child_env1.parameters["required_arg_from_shell_env"] +'required_arg_from_shell_env_val' +>>> # TODO: More + +>>> # Simulate running the environment to see its output: +>>> with child_env1: +... child_env1.run() + Environment Services ++++++++++++++++++++ From 514e11c52fb2bdbcafef9d963e0cc341b37445fb Mon Sep 17 00:00:00 2001 From: Brian Kroth Date: Fri, 14 Mar 2025 17:29:07 -0500 Subject: [PATCH 23/35] wip --- .../mlos_bench/environments/__init__.py | 47 +++++++++++++------ 1 file changed, 33 insertions(+), 14 deletions(-) diff --git a/mlos_bench/mlos_bench/environments/__init__.py b/mlos_bench/mlos_bench/environments/__init__.py index 79e4f00800..c7b82b37a8 100644 --- a/mlos_bench/mlos_bench/environments/__init__.py +++ b/mlos_bench/mlos_bench/environments/__init__.py @@ -242,7 +242,7 @@ ... "const_arg_from_local_env": "const_arg_from_local_env_child1_val", ... }, ... "run": [ -... "echo 'child1: required_arg_from_globals: $required_arg_from_globals'", +... "echo 'child1: required_arg_from_globals: ${required_arg_from_globals}'", ... "echo 'child1: required_arg_from_cli: $required_arg_from_cli'", ... "echo 'child1: required_arg_from_shell_env: $required_arg_from_shell_env'", ... "echo 'child1: const_arg_from_required: $const_arg_from_required'", @@ -251,7 +251,8 @@ ... "echo 'child1: const_arg_from_cli: $const_arg_from_cli'", ... "echo 'child1: const_arg_from_local_env: $const_arg_from_local_env'", ... "echo 'child1: const_arg_from_parent_env: $const_arg_from_parent_env'", -... ] +... ], +... "results_stdout_pattern": "(?:child[0-9]: )([a-zA-Z0-9_]+): (.+)", ... } ... }, ... { @@ -299,8 +300,9 @@ ... "echo 'child2: const_arg_from_child2_env2: $const_arg_from_child2_env2'", ... "echo 'child2: const_arg_from_parent_env: $const_arg_from_parent_env'", ... // Only some of those parameters are actually exposed as shell env vars though. -... "printenv | grep _arg_from_", -... ] +... "printenv | grep _arg_from_ | sed -e 's/^/child2: /' -e 's/=/: /'", +... ], +... "results_stdout_pattern": "(?:child[0-9]: )([a-zA-Z0-9_]+): (.+)", ... } ... } ... ] @@ -316,7 +318,7 @@ >>> # if we were calling `mlos_bench` directly on the CLI. >>> from mlos_bench.launcher import Launcher >>> argv = [ -... "--log-level=WARNING", +... "--log-level=DEBUG", # WARNING ... "--globals", globals_json, ... "--environment", composite_env_json, ... # Override some values via CLI directly: @@ -331,11 +333,10 @@ >>> assert child_env2.name == "child_env2" >>> # Demonstrate how tunable parameters are selected. ->>> child_env1.tunable_params ->>> child_env1.tunable_params["dummy_params"] -[0].name ->>> child_env1.tunable_params["my_env1_tunables"].tunable_groups[0].name -'group1' +>>> child_env1.tunable_params.get_param_values() +{'dummy_param': 'dummy'} +>>> child_env2.tunable_params.get_param_values() +{} >>> # Now see how the variable propagation works. >>> child_env1.parameters["required_arg_from_globals"] @@ -344,11 +345,29 @@ 'required_arg_from_cli_val' >>> child_env1.parameters["required_arg_from_shell_env"] 'required_arg_from_shell_env_val' ->>> # TODO: More - ->>> # Simulate running the environment to see its output: +>>> # Note that the default value in the local child env is overridden: +>>> child_env1.parameters["const_arg_from_globals"] +'const_arg_from_globals_val' +>>> child_env1.parameters["const_arg_from_shell_env"] +'const_arg_from_shell_env_val' +>>> child_env1.parameters["const_arg_from_cli"] +'const_arg_from_cli_val' +>>> # This is treated as a required_arg and inherited from the parent. +>>> child_env1.parameters["const_arg_from_parent_env"] +'const_arg_from_parent_env_val' + +>>> # TODO: child2 + +>>> # TODO: Simulate running the environment to see its output: +>>> from mlos_bench.environments.status import Status >>> with child_env1: -... child_env1.run() +... assert child_env1.setup(child_env1.tunable_params) +... (status, ts, result) = child_env1.run() +... assert status == Status.SUCCEEDED +... child_env1.teardown() +>>> # TODO: check output + +>>> # TODO: child2 Environment Services ++++++++++++++++++++ From bc4043cb7d68a8cad65f74a3f5e3c8359f06951e Mon Sep 17 00:00:00 2001 From: Sergiy Matusevych Date: Mon, 17 Mar 2025 16:01:50 -0700 Subject: [PATCH 24/35] add a section documenting the tunable_params_map --- .../mlos_bench/environments/__init__.py | 55 +++++++++++++++---- 1 file changed, 44 insertions(+), 11 deletions(-) diff --git a/mlos_bench/mlos_bench/environments/__init__.py b/mlos_bench/mlos_bench/environments/__init__.py index c7b82b37a8..edad03eeeb 100644 --- a/mlos_bench/mlos_bench/environments/__init__.py +++ b/mlos_bench/mlos_bench/environments/__init__.py @@ -101,14 +101,48 @@ json files when the :py:class:`~mlos_bench.launcher.Launcher` processes the initial set of config files. -The ``tunable_params`` setting in the ``config`` section of the Environment config -can also be used to limit *which* of the ``TunableGroups`` should be used for the -Environment. - -Since :py:mod:`json configs ` also support ``$variable`` -substitution in the values using the `globals` mechanism, this setting can used to -dynamically change the set of active TunableGroups for a given Experiment using only -`globals`, allowing for configs to be more modular and composable. +The ``tunable_params`` setting in the ``config`` section of the Environment config can then be +used to limit *which* of the ``TunableGroups`` should be used for the Environment. + +Tunable Parameters Map +^^^^^^^^^^^^^^^^^^^^^^ + +Although the full set of tunable parameters (and groups) of each Environment is always known in +advance, in practice we often want to limit it to a smaller subset for a given experiment. This +can be done by adding an extra level of indirection and specifying the ``tunable_params_map`` in +the global config. ``tunable_params_map`` associates a variable name with a list of +:py:class:`~mlos_bench.tunables.tunable_groups.TunableGroups` names, e.g., + + .. code-block:: json + + // experiment-globals.mlos.jsonc + { + "tunable_params_map": { + "tunables_ref1": ["tunable_group1", "tunable_group2"], + "tunables_ref2": [] // Useful to disable all tunables. + } + } + +Later, in the Environment config, we can use these variable names to refer to the +tunable groups we want to use for that Environment: + + .. code-block:: json + + // environment.mlos.jsonc + { + // ... + "config": { + "tunable_params": [ + "$tunables_ref1", // Will be replaced with "tunable_group1", "tunable_group2" + "$tunables_ref2", // A no-op + "tunable_group3" // Can still refer to a group directly. + ], + // ... etc. + +Using such ``"$tunables_ref"`` variables in the Environment config allows us to dynamically +change the set of active ``TunableGroups`` for a given Environment using the global config +without modifying the Environment configuration files for each experiment, thus making them +more modular and composable. Variable Propagation ++++++++++++++++++++ @@ -179,14 +213,13 @@ ... ... // Const args have a default value if not set, but can be overridden by ... // the globals, cli args, shell env, or parent env. -... ... "const_arg_from_globals": "const_arg_from_globals_val", ... "const_arg_from_shell_env": "$CONST_ARG_FROM_SHELL_ENV", ... "const_arg_from_child2_env2": "FROM@GLOBALS!", -... // special map of tunable_params_name to their set of enabled covariant tunable groups ... +... // special map of tunable_params_name to their set of enabled covariant tunable groups ... "tunable_params_map": { -... "my_env1_tunables": ["dummy_params"], +... "my_env1_tunables": ["tunable_group1", "tunable_group2"], ... "my_env2_tunables": [/* none */], ... }, ... } From 27fa353c7a99eb646f8e1e3662f160fe278d772b Mon Sep 17 00:00:00 2001 From: Sergiy Matusevych Date: Mon, 17 Mar 2025 17:53:56 -0700 Subject: [PATCH 25/35] a simpler version of the variable propagation example (WIP) --- .../network/virtual-network-environment.jsonc | 2 +- .../config/tunables/dummy-tunables.jsonc | 32 ++- .../mlos_bench/environments/__init__.py | 203 +++++------------- 3 files changed, 88 insertions(+), 149 deletions(-) diff --git a/mlos_bench/mlos_bench/config/environments/network/virtual-network-environment.jsonc b/mlos_bench/mlos_bench/config/environments/network/virtual-network-environment.jsonc index 2a9aa7691a..bab6308459 100644 --- a/mlos_bench/mlos_bench/config/environments/network/virtual-network-environment.jsonc +++ b/mlos_bench/mlos_bench/config/environments/network/virtual-network-environment.jsonc @@ -18,7 +18,7 @@ "config": { // FIXME: There aren't currently any tunable params for the network environment. // But one is required to workaround a bug (#613) in the config storage layer. - "tunable_params": ["dummy_params"], + "tunable_params": ["dummy_params_group1"], // Typically don't want to deprovision the network environment on teardown // since other experiments in this same RG might be using it. diff --git a/mlos_bench/mlos_bench/config/tunables/dummy-tunables.jsonc b/mlos_bench/mlos_bench/config/tunables/dummy-tunables.jsonc index 6fcfa3e6f8..028ee93011 100644 --- a/mlos_bench/mlos_bench/config/tunables/dummy-tunables.jsonc +++ b/mlos_bench/mlos_bench/config/tunables/dummy-tunables.jsonc @@ -1,6 +1,6 @@ // FIXME: A workaround to a bug (#613) in the config system that requires non-empty configs. { - "dummy_params": { + "dummy_params_group1": { "cost": 0, "description": "Dummy parameter group to allow a non-empty config for testing creating singleton resources that have no tunable params.", "params": { @@ -11,5 +11,35 @@ "default": "dummy" } } + }, + "dummy_params_group2": { + "cost": 1, + "description": "Dummy parameter group for testing", + "params": { + "dummy_param_int": { + "description": "An integer dummy parameter.", + "type": "integer", + "range": [0, 100], + "default": 0 + }, + "dummy_param_float": { + "description": "A float dummy parameter.", + "type": "float", + "range": [0, 1], + "default": 0.5 + } + } + }, + "dummy_params_group3": { + "cost": 1, + "description": "Another dummy parameter group for testing", + "params": { + "dummy_param3": { + "description": "One more dummy parameter.", + "type": "float", + "range": [-1, 1], + "default": 0.0 + } + } } } diff --git a/mlos_bench/mlos_bench/environments/__init__.py b/mlos_bench/mlos_bench/environments/__init__.py index edad03eeeb..9ddea2fe1d 100644 --- a/mlos_bench/mlos_bench/environments/__init__.py +++ b/mlos_bench/mlos_bench/environments/__init__.py @@ -139,6 +139,10 @@ ], // ... etc. +Note: this references the `dummy-tunables.jsonc +`_ +file for simplicity. + Using such ``"$tunables_ref"`` variables in the Environment config allows us to dynamically change the set of active ``TunableGroups`` for a given Environment using the global config without modifying the Environment configuration files for each experiment, thus making them @@ -147,14 +151,10 @@ Variable Propagation ++++++++++++++++++++ -Parameters declared in the ``const_args`` or ``required_args`` sections of the -Environment config can be overridden with values of the corresponding parameters -of the parent Environment or specified in the external config files or the -command line. - -In fact, ``const_args`` or ``required_args`` sections can be viewed as -placeholders for the parameters that are being pushed to the environment from -the outside. +Parameters declared in the ``const_args`` or ``required_args`` sections of the Environment +config can be overridden with values specified in the external config files or the command +line. In fact, ``const_args`` or ``required_args`` sections can be viewed as placeholders +for the parameters that are being pushed to the environment from the outside. The same parameter can be present in both ``const_args`` and ``required_args`` sections. ``required_args`` is just a way to emphasize the importance of the parameter and create a @@ -163,32 +163,25 @@ and can't be resolved from the ``globals`` this allows MLOS to fail fast and return an error to the user indicating an incomplete config. -Variable replacement happens in the bottom-up manner. That is, if a certain -parameter is present in the parent (:py:class:`~.CompositeEnv`) Environment, it -will replace the corresponding parameter in the child, and so on. - Note that the parameter **must** appear in the child Environment ``const_args`` or ``required_args`` section; if a parameter is not present in one of these -placeholders of the child Environment config, it will not be propagated. This -hierarchy allows MLOS users to have small immutable Environment configurations -at the lower levels and combine and parameterize them at the higher levels. +placeholders of the Environment config, it will not be propagated. This allows MLOS +users to have small immutable Environment configurations and combine and parameterize +them with external (global) configs. Taking it to the next level outside of the Environment configs, the parameters can be defined in the external key-value JSON config files (usually referred to -as `global config files +as global config files <../config/index.html#globals-and-variable-substitution>`_ in MLOS lingo). See :py:mod:`mlos_bench.config` for more details. We can summarize the parameter propagation rules as follows: -1. Child environment will only get the parameters defined in its ``const_args`` or +1. An environment will only get the parameters defined in its ``const_args`` or ``required_args`` sections. -2. The value of the parameter defined in the ``const_args`` section of the - parent Environment will override the value of the corresponding parameter in the - child Environments. -3. Values of the parameters defined in the global config files will override the values of the +2. Values of the parameters defined in the global config files will override the values of the corresponding parameters in all environments. -4. Values of the command line parameters take precedence over values defined in the global or +3. Values of the command line parameters take precedence over values defined in the global or environment configs. Examples @@ -204,141 +197,57 @@ >>> # globals.jsonc >>> globals_json = ''' ... { -... "experiment_id": "VariablePropagationExample", -... -... // Required arguments must have their value set from globals, cli args, or shell env. -... "required_arg_from_globals": "required_arg_from_globals_val", -... "required_arg_from_cli": "require_arg_from_globals_NOT_CLI", // will be replaced by cli invocation -... "required_arg_from_shell_env": "$REQUIRED_ARG_FROM_SHELL_ENV", +... "const_arg_from_globals_1": "Substituted from globals", ... -... // Const args have a default value if not set, but can be overridden by -... // the globals, cli args, shell env, or parent env. -... "const_arg_from_globals": "const_arg_from_globals_val", -... "const_arg_from_shell_env": "$CONST_ARG_FROM_SHELL_ENV", -... "const_arg_from_child2_env2": "FROM@GLOBALS!", -... -... // special map of tunable_params_name to their set of enabled covariant tunable groups +... // Define reference names to represent tunable groups in the Environment configs. ... "tunable_params_map": { -... "my_env1_tunables": ["tunable_group1", "tunable_group2"], -... "my_env2_tunables": [/* none */], -... }, +... "tunables_ref2": ["dummy_params_group1", "dummy_params_group2"], +... "tunables_ref2": [], // Useful to disable all tunables for the Environment. +... } ... } ... ''' ->>> # composite_env.jsonc ->>> composite_env_json = ''' +>>> # environment.jsonc +>>> environment_json = ''' ... { -... "class": "mlos_bench.environments.composite_env.CompositeEnv", -... "name": "parent_env", +... "class": "mlos_bench.environments.local.local_env.LocalEnv", +... "name": "test_env1", +... "include_tunables": [ +... "tunables/dummy-tunables.jsonc" // For simplicity, include all tunables available. +... ], ... "config": { -... // Must be populated by a global config or command line: -... "required_args": [ -... "required_arg_from_globals", -... "required_arg_from_cli", -... //"required_arg_from_shell_env", # TEST ME: does this need to be here to flow down? +... "tunable_params": [ +... "$tunables_ref1", // Includes "dummy_params_group1", "dummy_params_group2" +... "$tunables_ref2", // A no-op +... "dummy_params_group3" // Can still refer to a group directly. ... ], -... // Can be populate by variable expansion from a higher level, or else defaulted to here. ... "const_args": { -... "const_arg_from_globals": "const_arg_from_globals_parent_val", -... "const_arg_from_shell_env": "const_arg_from_shell_parent_val", -... "const_arg_from_cli": "const_arg_from_cli_parent_val", -... "const_arg_from_parent_env": "const_arg_from_parent_env_val", -... "const_arg_from_local_env": "const_arg_from_local_env_parent_val", -... // const_arg defaults can also use variable expansion to refer -... // to another variable previously defined (even another const_arg) in order to -... // allow for variable renaming -... "const_arg_from_required": "$required_arg_from_globals", +... // Environment-specific non-tunable constant parameters: +... "const_arg_1": "Default value of const_arg_1", +... "const_arg_from_globals_1": "To be replaced from global config", +... "const_arg_from_shell_env": "To be replaced from shell env", +... "const_arg_from_cli_1": "To be replaced from CLI" ... }, -... "children": [ -... { -... "class": "mlos_bench.environments.local.local_env.LocalEnv", -... "name": "child_env1", -... "include_tunables": [ -... "tunables/dummy-tunables.jsonc" -... ], -... "config": { -... "tunable_params": ["$my_env1_tunables"], -... "required_args": [ -... "required_arg_from_globals", -... "required_arg_from_cli", -... "required_arg_from_shell_env", -... // Here, we can simply declare a required_arg as -... // required, but let it inherit a value from a higher level environment. -... "const_arg_from_required", -... "const_arg_from_parent_env", -... ], -... "const_args": { -... // Here we provide defaults, though all of these should be overridden by higher levels. -... "const_arg_from_globals": "const_arg_from_globals_child1_val", -... "const_arg_from_shell_env": "const_arg_from_shell_child1_val", -... "const_arg_from_cli": "const_arg_from_cli_child1_val", -... "const_arg_from_local_env": "const_arg_from_local_env_child1_val", -... }, -... "run": [ -... "echo 'child1: required_arg_from_globals: ${required_arg_from_globals}'", -... "echo 'child1: required_arg_from_cli: $required_arg_from_cli'", -... "echo 'child1: required_arg_from_shell_env: $required_arg_from_shell_env'", -... "echo 'child1: const_arg_from_required: $const_arg_from_required'", -... "echo 'child1: const_arg_from_globals: $const_arg_from_globals'", -... "echo 'child1: const_arg_from_shell_env: $const_arg_from_shell_env'", -... "echo 'child1: const_arg_from_cli: $const_arg_from_cli'", -... "echo 'child1: const_arg_from_local_env: $const_arg_from_local_env'", -... "echo 'child1: const_arg_from_parent_env: $const_arg_from_parent_env'", -... ], -... "results_stdout_pattern": "(?:child[0-9]: )([a-zA-Z0-9_]+): (.+)", -... } -... }, -... { -... "class": "mlos_bench.environments.local.local_env.LocalEnv", -... "name": "child_env2", -... "include_tunables": [ -... "tunables/dummy-tunables.jsonc" -... ], -... "config": { -... "tunable_params": ["$my_env2_tunables"], -... "required_args": [ -... "required_arg_from_globals", -... "required_arg_from_cli", -... "required_arg_from_shell_env", -... // Here, we can simply declare a required_arg as -... // required, but let it inherit a value from a higher level environment. -... "const_arg_from_required", -... "const_arg_from_parent_env", -... ], -... "const_args": { -... // Here we provide defaults, though all of these should be overridden by higher levels. -... "const_arg_from_globals": "const_arg_from_globals_child2_val", -... "const_arg_from_shell_env": "const_arg_from_shell_child2_val", -... "const_arg_from_cli": "const_arg_from_cli_child2_val", -... "const_arg_from_local_env": "const_arg_from_local_env_child2_val", -... "const_arg_from_child2_env1": "const_arg_from_child2_val", -... "const_arg_from_child2_env2": "const_arg_from_child2_val", -... }, -... // Expose the args as shell env vars for the child env. -... "shell_env_params": [ -... "required_arg_from_globals", -... "const_arg_from_globals", -... ], -... "run": [ -... // Each of these commands undergoes variable replacement prior to being executed. -... "echo 'child2: required_arg_from_globals: $required_arg_from_globals'", -... "echo 'child2: required_arg_from_cli: $required_arg_from_cli'", -... "echo 'child2: required_arg_from_shell_env: $required_arg_from_shell_env'", -... "echo 'child2: const_arg_from_required: $const_arg_from_required'", -... "echo 'child2: const_arg_from_globals: $const_arg_from_globals'", -... "echo 'child2: const_arg_from_shell_env: $const_arg_from_shell_env'", -... "echo 'child2: const_arg_from_cli: $const_arg_from_cli'", -... "echo 'child2: const_arg_from_local_env: $const_arg_from_local_env'", -... "echo 'child2: const_arg_from_child2_env1: $const_arg_from_child2_env1'", -... "echo 'child2: const_arg_from_child2_env2: $const_arg_from_child2_env2'", -... "echo 'child2: const_arg_from_parent_env: $const_arg_from_parent_env'", -... // Only some of those parameters are actually exposed as shell env vars though. -... "printenv | grep _arg_from_ | sed -e 's/^/child2: /' -e 's/=/: /'", -... ], -... "results_stdout_pattern": "(?:child[0-9]: )([a-zA-Z0-9_]+): (.+)", -... } -... } -... ] +... "required_args": [ +... // These parameters always come from elsewhere: +... "const_arg_from_globals_2", +... "const_arg_from_cli_2", +... // We already define these parameters in "const_args" section above; +... // mentioning them here is optional, but can be used for clarity: +... "const_arg_from_globals_1", +... "const_arg_from_cli_1" +... ], +... "run": [ +... "echo const_arg_1: $const_arg_1", +... "echo const_arg_from_globals_1: $const_arg_from_globals_1", +... "echo const_arg_from_globals_2: $const_arg_from_globals_2", +... "echo const_arg_from_shell_env: $const_arg_from_shell_env", +... "echo const_arg_from_cli_1: $const_arg_from_cli_1", +... "echo const_arg_from_cli_2: $const_arg_from_cli_2", +... "echo dummy_param: $dummy_param", // A tunable from dummy_params_group1 +... "echo dummy_param3: $dummy_param3" // A tunable from dummy_params_group3 +... ], +... "results_stdout_pattern": "([a-zA-Z0-9_]+): (.+)" ... } ... } ... ''' From 9d7f491e3bc51a0540d5da172efb2c82efe6c787 Mon Sep 17 00:00:00 2001 From: Sergiy Matusevych Date: Tue, 18 Mar 2025 11:37:45 -0700 Subject: [PATCH 26/35] fixing the executable examples in the environments docstrings (WIP) --- .../config/tunables/dummy-tunables.jsonc | 2 +- .../mlos_bench/environments/__init__.py | 31 +++++++------------ 2 files changed, 12 insertions(+), 21 deletions(-) diff --git a/mlos_bench/mlos_bench/config/tunables/dummy-tunables.jsonc b/mlos_bench/mlos_bench/config/tunables/dummy-tunables.jsonc index 028ee93011..9f812e0a15 100644 --- a/mlos_bench/mlos_bench/config/tunables/dummy-tunables.jsonc +++ b/mlos_bench/mlos_bench/config/tunables/dummy-tunables.jsonc @@ -18,7 +18,7 @@ "params": { "dummy_param_int": { "description": "An integer dummy parameter.", - "type": "integer", + "type": "int", "range": [0, 100], "default": 0 }, diff --git a/mlos_bench/mlos_bench/environments/__init__.py b/mlos_bench/mlos_bench/environments/__init__.py index 9ddea2fe1d..50cba28761 100644 --- a/mlos_bench/mlos_bench/environments/__init__.py +++ b/mlos_bench/mlos_bench/environments/__init__.py @@ -197,11 +197,12 @@ >>> # globals.jsonc >>> globals_json = ''' ... { -... "const_arg_from_globals_1": "Substituted from globals", +... "const_arg_from_globals_1": "Substituted from globals - 1", +... "const_arg_from_globals_2": "Substituted from globals - 2", ... ... // Define reference names to represent tunable groups in the Environment configs. ... "tunable_params_map": { -... "tunables_ref2": ["dummy_params_group1", "dummy_params_group2"], +... "tunables_ref1": ["dummy_params_group1", "dummy_params_group2"], ... "tunables_ref2": [], // Useful to disable all tunables for the Environment. ... } ... } @@ -251,35 +252,25 @@ ... } ... } ... ''' - ->>> # Setup the shell env as if bash used an "export VAR='val'" ->>> import os ->>> os.environ["REQUIRED_ARG_FROM_SHELL_ENV"] = "required_arg_from_shell_env_val" ->>> os.environ["CONST_ARG_FROM_SHELL_ENV"] = "const_arg_from_shell_env_val" +... >>> # Load the globals and environment configs defined above via the Launcher as >>> # if we were calling `mlos_bench` directly on the CLI. >>> from mlos_bench.launcher import Launcher >>> argv = [ ... "--log-level=DEBUG", # WARNING ... "--globals", globals_json, -... "--environment", composite_env_json, +... "--environment", environment_json, ... # Override some values via CLI directly: -... "--required_arg_from_cli", "required_arg_from_cli_val", -... "--const_arg_from_cli", "const_arg_from_cli_val", +... "--const_arg_from_cli_1", "const_arg_from_cli_val1", +... "--const_arg_from_cli_2", "const_arg_from_cli_val2", ... ] >>> launcher = Launcher("sample_launcher", argv=argv) ->>> composite_env = launcher.root_environment ->>> child_env1 = composite_env.children[0] ->>> assert child_env1.name == "child_env1" ->>> child_env2 = composite_env.children[1] ->>> assert child_env2.name == "child_env2" - +>>> env = launcher.root_environment +>>> env.name +'test_env1' >>> # Demonstrate how tunable parameters are selected. ->>> child_env1.tunable_params.get_param_values() +>>> env.tunable_params.get_param_values() {'dummy_param': 'dummy'} ->>> child_env2.tunable_params.get_param_values() -{} - >>> # Now see how the variable propagation works. >>> child_env1.parameters["required_arg_from_globals"] 'required_arg_from_globals_val' From e27fdd2a5d83c1505688fd77e07ec3ef1fb08245 Mon Sep 17 00:00:00 2001 From: Sergiy Matusevych Date: Tue, 18 Mar 2025 14:07:15 -0700 Subject: [PATCH 27/35] first complete version of the variable propagation example. --- .../mlos_bench/environments/__init__.py | 142 ++++++++++++------ mlos_bench/mlos_bench/launcher.py | 2 +- 2 files changed, 93 insertions(+), 51 deletions(-) diff --git a/mlos_bench/mlos_bench/environments/__init__.py b/mlos_bench/mlos_bench/environments/__init__.py index 50cba28761..9a6215f271 100644 --- a/mlos_bench/mlos_bench/environments/__init__.py +++ b/mlos_bench/mlos_bench/environments/__init__.py @@ -197,9 +197,13 @@ >>> # globals.jsonc >>> globals_json = ''' ... { +... "experiment_id": "test_experiment", +... ... "const_arg_from_globals_1": "Substituted from globals - 1", ... "const_arg_from_globals_2": "Substituted from globals - 2", ... +... "const_arg_from_cli_1": "Will be overridden from CLI", +... ... // Define reference names to represent tunable groups in the Environment configs. ... "tunable_params_map": { ... "tunables_ref1": ["dummy_params_group1", "dummy_params_group2"], @@ -226,7 +230,6 @@ ... // Environment-specific non-tunable constant parameters: ... "const_arg_1": "Default value of const_arg_1", ... "const_arg_from_globals_1": "To be replaced from global config", -... "const_arg_from_shell_env": "To be replaced from shell env", ... "const_arg_from_cli_1": "To be replaced from CLI" ... }, ... "required_args": [ @@ -239,68 +242,107 @@ ... "const_arg_from_cli_1" ... ], ... "run": [ -... "echo const_arg_1: $const_arg_1", -... "echo const_arg_from_globals_1: $const_arg_from_globals_1", -... "echo const_arg_from_globals_2: $const_arg_from_globals_2", -... "echo const_arg_from_shell_env: $const_arg_from_shell_env", -... "echo const_arg_from_cli_1: $const_arg_from_cli_1", -... "echo const_arg_from_cli_2: $const_arg_from_cli_2", -... "echo dummy_param: $dummy_param", // A tunable from dummy_params_group1 -... "echo dummy_param3: $dummy_param3" // A tunable from dummy_params_group3 -... ], -... "results_stdout_pattern": "([a-zA-Z0-9_]+): (.+)" +... "echo Hello world" +... ] ... } ... } ... ''' -... + +Now that we have our environment and global configurations, we can instantiate the +:py:class:`~.Environment` and inspect it. In this example we will simulate the command line execution to demonstrate how CLI parameters propagate to the environment. + >>> # Load the globals and environment configs defined above via the Launcher as >>> # if we were calling `mlos_bench` directly on the CLI. >>> from mlos_bench.launcher import Launcher >>> argv = [ -... "--log-level=DEBUG", # WARNING -... "--globals", globals_json, ... "--environment", environment_json, +... "--globals", globals_json, ... # Override some values via CLI directly: -... "--const_arg_from_cli_1", "const_arg_from_cli_val1", -... "--const_arg_from_cli_2", "const_arg_from_cli_val2", +... "--const_arg_from_cli_1", "Substituted from CLI - 1", +... "--const_arg_from_cli_2", "Substituted from CLI - 2", ... ] >>> launcher = Launcher("sample_launcher", argv=argv) >>> env = launcher.root_environment ->>> env.name -'test_env1' ->>> # Demonstrate how tunable parameters are selected. +>>> assert env.name == "test_env1" + +``env`` is an instance of :py:class:`~.Environment` class that we can use to setup, run, and tear +down the environment. It also has a set of properties and methods that we can use to access the +object's parameters. This way we can check the actual runtime configuration of the environment. + +First, let's take a look at the tunable parameters: + >>> env.tunable_params.get_param_values() -{'dummy_param': 'dummy'} ->>> # Now see how the variable propagation works. ->>> child_env1.parameters["required_arg_from_globals"] -'required_arg_from_globals_val' ->>> child_env1.parameters["required_arg_from_cli"] -'required_arg_from_cli_val' ->>> child_env1.parameters["required_arg_from_shell_env"] -'required_arg_from_shell_env_val' ->>> # Note that the default value in the local child env is overridden: ->>> child_env1.parameters["const_arg_from_globals"] -'const_arg_from_globals_val' ->>> child_env1.parameters["const_arg_from_shell_env"] -'const_arg_from_shell_env_val' ->>> child_env1.parameters["const_arg_from_cli"] -'const_arg_from_cli_val' ->>> # This is treated as a required_arg and inherited from the parent. ->>> child_env1.parameters["const_arg_from_parent_env"] -'const_arg_from_parent_env_val' - ->>> # TODO: child2 - ->>> # TODO: Simulate running the environment to see its output: ->>> from mlos_bench.environments.status import Status ->>> with child_env1: -... assert child_env1.setup(child_env1.tunable_params) -... (status, ts, result) = child_env1.run() -... assert status == Status.SUCCEEDED -... child_env1.teardown() ->>> # TODO: check output - ->>> # TODO: child2 +{'dummy_param': 'dummy', + 'dummy_param_int': 0, + 'dummy_param_float': 0.5, + 'dummy_param3': 0.0} + +We can see the tunables from ``dummy_params_group1`` and ``dummy_params_group2`` groups specified +via ``$tunables_ref1``, as well as the tunables from ``dummy_params_group3`` that we specified +directly in the Environment config. All tunables are initialized to their default values. + +Now let's see how the variable propagation works. + +>>> env.const_args["const_arg_1"] +'Default value of const_arg_1' + +``const_arg_1`` has the value we have assigned in the ``"const_args"`` section of the +Environment config. No surprises here. + +>>> env.const_args["const_arg_from_globals_1"] +'Substituted from globals - 1' +>>> env.const_args["const_arg_from_globals_2"] +'Substituted from globals - 2' + +``const_arg_from_globals_1`` and ``const_arg_from_globals_2`` were declared in the Environment's +``const_args`` and ``required_args`` sections, respectively. Their values were overridden by the +values from the global config. + +>>> env.const_args["const_arg_from_cli_1"] +'Substituted from CLI - 1' +>>> env.const_args["const_arg_from_cli_2"] +'Substituted from CLI - 2' + +Likewise, ``const_arg_from_cli_1`` and ``const_arg_from_cli_2`` got their values from the +command line. Note that for ``const_arg_from_cli_1`` the value from the command line takes +precedence over the values specified in the Environment's ``const_args`` section **and** the one +in the global config. + +Now let's set up the environment and see how the constant and tunable parameters get combined. +We'll also assign some non-default values to the tunables, as the optimizer would do on each +trial. + +>>> env.tunable_params["dummy_param_int"] = 99 +>>> env.tunable_params["dummy_param3"] = 0.999 +>>> with env: +... assert env.setup(env.tunable_params) +... env.parameters +{'const_arg_1': 'Default value of const_arg_1', + 'const_arg_from_globals_1': 'Substituted from globals', + 'const_arg_from_cli_1': 'const_arg_from_cli_val1', + 'const_arg_from_globals_2': 'Substituted from globals', + 'trial_id': 1, + 'const_arg_from_cli_2': 'const_arg_from_cli_val2', + 'trial_runner_id': 1, + 'experiment_id': 'test_experiment', + 'dummy_param': 'dummy', + 'dummy_param_int': 99, + 'dummy_param_float': 0.5, + 'dummy_param': 0.999} + +These are the values visible to the implementations of the ``setup``, ``run``, and ``teardown`` +methods. We can see both the constant and tunable parameters combined into a single dictionary +with proper values assigned to each of them on each iteration. + +A few "Magic" parameters like ``trial_id`` and ``trial_runner_id`` are added by the Scheduler and +used for trials parallelization and storage of the results. It is sometimes useful to add them, +for example, to the paths used by the Environment, as in, e.g., +``"/storage/$experiment_id/$trial_id/data/"``, to prevent conflicts when running multiple +experiments and trials in parallel. + +We will discuss passing the parameters to external scripts and using them in referencing files +and directories in local and shared storage in the documentation of the concrete +:py:class:`~.Environment` implementations. Environment Services ++++++++++++++++++++ diff --git a/mlos_bench/mlos_bench/launcher.py b/mlos_bench/mlos_bench/launcher.py index d64045c823..c728ed7fb2 100644 --- a/mlos_bench/mlos_bench/launcher.py +++ b/mlos_bench/mlos_bench/launcher.py @@ -609,7 +609,7 @@ def _load_scheduler(self, args_scheduler: str | None) -> Scheduler: return SyncScheduler( # All config values can be overridden from global config config={ - "experiment_id": "UNDEFINED - override from global config", + "experiment_id": "DEFAULT_EXPERIMENT_ID", "trial_id": 0, "config_id": -1, "trial_config_repeat_count": 1, From bbbe0b9cc5e21444f4906792c9e313d232b82c5b Mon Sep 17 00:00:00 2001 From: Sergiy Matusevych Date: Tue, 18 Mar 2025 14:34:43 -0700 Subject: [PATCH 28/35] make docstring tests pass --- .../mlos_bench/environments/__init__.py | 43 ++++++++++--------- 1 file changed, 23 insertions(+), 20 deletions(-) diff --git a/mlos_bench/mlos_bench/environments/__init__.py b/mlos_bench/mlos_bench/environments/__init__.py index 9a6215f271..5b817023fd 100644 --- a/mlos_bench/mlos_bench/environments/__init__.py +++ b/mlos_bench/mlos_bench/environments/__init__.py @@ -263,19 +263,21 @@ ... ] >>> launcher = Launcher("sample_launcher", argv=argv) >>> env = launcher.root_environment ->>> assert env.name == "test_env1" +>>> env.name +'test_env1' ``env`` is an instance of :py:class:`~.Environment` class that we can use to setup, run, and tear down the environment. It also has a set of properties and methods that we can use to access the object's parameters. This way we can check the actual runtime configuration of the environment. -First, let's take a look at the tunable parameters: +First, let's check the tunable parameters: ->>> env.tunable_params.get_param_values() -{'dummy_param': 'dummy', - 'dummy_param_int': 0, - 'dummy_param_float': 0.5, - 'dummy_param3': 0.0} +>>> assert env.tunable_params.get_param_values() == { +... "dummy_param": "dummy", +... "dummy_param_int": 0, +... "dummy_param_float": 0.5, +... "dummy_param3": 0.0 +... } We can see the tunables from ``dummy_params_group1`` and ``dummy_params_group2`` groups specified via ``$tunables_ref1``, as well as the tunables from ``dummy_params_group3`` that we specified @@ -316,19 +318,20 @@ >>> env.tunable_params["dummy_param3"] = 0.999 >>> with env: ... assert env.setup(env.tunable_params) -... env.parameters -{'const_arg_1': 'Default value of const_arg_1', - 'const_arg_from_globals_1': 'Substituted from globals', - 'const_arg_from_cli_1': 'const_arg_from_cli_val1', - 'const_arg_from_globals_2': 'Substituted from globals', - 'trial_id': 1, - 'const_arg_from_cli_2': 'const_arg_from_cli_val2', - 'trial_runner_id': 1, - 'experiment_id': 'test_experiment', - 'dummy_param': 'dummy', - 'dummy_param_int': 99, - 'dummy_param_float': 0.5, - 'dummy_param': 0.999} +... assert env.parameters == { +... "const_arg_1": "Default value of const_arg_1", +... "const_arg_from_globals_1": "Substituted from globals - 1", +... "const_arg_from_globals_2": "Substituted from globals - 2", +... "const_arg_from_cli_1": "Substituted from CLI - 1", +... "const_arg_from_cli_2": "Substituted from CLI - 2", +... "trial_id": 1, +... "trial_runner_id": 1, +... "experiment_id": "test_experiment", +... "dummy_param": "dummy", +... "dummy_param_int": 99, +... "dummy_param_float": 0.5, +... "dummy_param3": 0.999 +... } These are the values visible to the implementations of the ``setup``, ``run``, and ``teardown`` methods. We can see both the constant and tunable parameters combined into a single dictionary From ff89593d8d86c5f3ec396bdf7aa7a4612594b77c Mon Sep 17 00:00:00 2001 From: Sergiy Matusevych Date: Tue, 18 Mar 2025 14:45:07 -0700 Subject: [PATCH 29/35] trying to please pydocstyle --- mlos_bench/mlos_bench/environments/__init__.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/mlos_bench/mlos_bench/environments/__init__.py b/mlos_bench/mlos_bench/environments/__init__.py index 5b817023fd..e0c8351d18 100644 --- a/mlos_bench/mlos_bench/environments/__init__.py +++ b/mlos_bench/mlos_bench/environments/__init__.py @@ -185,7 +185,7 @@ environment configs. Examples -^^^^^^^^ +-------- Here's a simple working example of a local environment config (written in Python instead of JSON for testing) to show how variable propagation works: @@ -380,6 +380,7 @@ Examples -------- + While this documentation is generated from the source code and is intended to be a useful reference on the internal details, most users will be more interested in generating json configs to be used with the ``mlos_bench`` command line tool. @@ -394,6 +395,7 @@ Notes ----- + - See `mlos_bench/environments/README.md `_ for additional documentation in the source tree. @@ -403,6 +405,7 @@ See Also -------- + :py:mod:`mlos_bench.config` : Overview of the configuration system. :py:mod:`mlos_bench.services` : From 393e0388b3f56e234277172094a28a5f76c8e9bd Mon Sep 17 00:00:00 2001 From: Sergiy Matusevych Date: Tue, 18 Mar 2025 14:49:13 -0700 Subject: [PATCH 30/35] remove blank lines after Examples, Notes, and See Also sections to please the linter --- mlos_bench/mlos_bench/environments/__init__.py | 4 ---- 1 file changed, 4 deletions(-) diff --git a/mlos_bench/mlos_bench/environments/__init__.py b/mlos_bench/mlos_bench/environments/__init__.py index e0c8351d18..c4f8976858 100644 --- a/mlos_bench/mlos_bench/environments/__init__.py +++ b/mlos_bench/mlos_bench/environments/__init__.py @@ -186,7 +186,6 @@ Examples -------- - Here's a simple working example of a local environment config (written in Python instead of JSON for testing) to show how variable propagation works: @@ -380,7 +379,6 @@ Examples -------- - While this documentation is generated from the source code and is intended to be a useful reference on the internal details, most users will be more interested in generating json configs to be used with the ``mlos_bench`` command line tool. @@ -395,7 +393,6 @@ Notes ----- - - See `mlos_bench/environments/README.md `_ for additional documentation in the source tree. @@ -405,7 +402,6 @@ See Also -------- - :py:mod:`mlos_bench.config` : Overview of the configuration system. :py:mod:`mlos_bench.services` : From eecabf9058f2e2bc8ec0579f263ac66935637cb2 Mon Sep 17 00:00:00 2001 From: Brian Kroth Date: Thu, 20 Mar 2025 13:55:34 -0500 Subject: [PATCH 31/35] fix a few links --- mlos_bench/mlos_bench/environments/__init__.py | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/mlos_bench/mlos_bench/environments/__init__.py b/mlos_bench/mlos_bench/environments/__init__.py index c4f8976858..fb000d56c3 100644 --- a/mlos_bench/mlos_bench/environments/__init__.py +++ b/mlos_bench/mlos_bench/environments/__init__.py @@ -171,7 +171,7 @@ Taking it to the next level outside of the Environment configs, the parameters can be defined in the external key-value JSON config files (usually referred to -as global config files +as `global config files <../config/index.html#globals-and-variable-substitution>`_ in MLOS lingo). See :py:mod:`mlos_bench.config` for more details. @@ -336,11 +336,13 @@ methods. We can see both the constant and tunable parameters combined into a single dictionary with proper values assigned to each of them on each iteration. -A few "Magic" parameters like ``trial_id`` and ``trial_runner_id`` are added by the Scheduler and -used for trials parallelization and storage of the results. It is sometimes useful to add them, -for example, to the paths used by the Environment, as in, e.g., -``"/storage/$experiment_id/$trial_id/data/"``, to prevent conflicts when running multiple -experiments and trials in parallel. +A few `Well Known Parameters <../config/index.html#well-known-variables>`_ +parameters like ``trial_id`` and ``trial_runner_id`` are added by the +:py:mod:`Scheduler ` and used for trials parallelization +and storage of the results. It is sometimes useful to add them, for example, to +the paths used by the Environment, as in, e.g., +``"/storage/$experiment_id/$trial_id/data/"``, to prevent conflicts when running +multiple Experiments and Trials in parallel. We will discuss passing the parameters to external scripts and using them in referencing files and directories in local and shared storage in the documentation of the concrete From 475189e6709dc9cee79ee6535200ed1ffe783576 Mon Sep 17 00:00:00 2001 From: Brian Kroth Date: Thu, 20 Mar 2025 14:04:48 -0500 Subject: [PATCH 32/35] Add a rule to look for malformed RST links --- Makefile | 2 ++ 1 file changed, 2 insertions(+) diff --git a/Makefile b/Makefile index 94cfabda45..6172491fad 100644 --- a/Makefile +++ b/Makefile @@ -480,6 +480,8 @@ build/check-doc.build-stamp: doc/build/html/index.html doc/build/html/htmlcov/in test -s doc/build/html/autoapi/mlos_viz/index.html test -s doc/build/html/autoapi/mlos_viz/dabl/index.html grep -q -e '--config CONFIG' doc/build/html//mlos_bench.run.usage.html + # Look for malformed rst links: `LinkName `_ + if find doc/build/html -name '*.html' -print0 | xargs -0 grep -m1 '>`_' | grep -m1 .; then echo "Bad links found"; false; fi # Check doc logs for errors (but skip over some known ones) ... @cat doc/build/log.txt \ | egrep -C1 -e WARNING -e CRITICAL -e ERROR \ From cf31fec04c737d03702bf413c1d48dd81f4f6d6f Mon Sep 17 00:00:00 2001 From: Sergiy Matusevych Date: Thu, 20 Mar 2025 13:27:46 -0700 Subject: [PATCH 33/35] update the Environment docstring explaining the `.parameters` property and its usage in the child classes --- mlos_bench/mlos_bench/environments/__init__.py | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/mlos_bench/mlos_bench/environments/__init__.py b/mlos_bench/mlos_bench/environments/__init__.py index fb000d56c3..78855b499b 100644 --- a/mlos_bench/mlos_bench/environments/__init__.py +++ b/mlos_bench/mlos_bench/environments/__init__.py @@ -332,9 +332,17 @@ ... "dummy_param3": 0.999 ... } -These are the values visible to the implementations of the ``setup``, ``run``, and ``teardown`` -methods. We can see both the constant and tunable parameters combined into a single dictionary -with proper values assigned to each of them on each iteration. +These are the values visible to the implementations of the :py:meth:`~.Environment.setup`, +:py:meth:`~.Environment.run`, and :py:meth:`~.Environment.teardown` methods. We can see both +the constant and tunable parameters combined into a single dictionary +:py:attr:`~.Environment.parameters` with proper values assigned to each of them on each iteration. +When implementing a new :py:class:`~.Environment`-derived class, developers can rely on the +:py:attr:`~.Environment.parameters` data in their versions of :py:meth:`~.Environment.setup` and +other methods. For example, :py:class:`~mlos_bench.environments.remote.vm_env.VMEnv` would then +pass the :py:attr:`~.Environment.parameters` into an ARM template when provisioning a new VM, +and :py:class:`~.LocalEnv` can dump them into a JSON file specified in the ``dump_params_file`` +config property, or/and cherry-pick some of these values and make them shell variables with the +``shell_env_params``. A few `Well Known Parameters <../config/index.html#well-known-variables>`_ parameters like ``trial_id`` and ``trial_runner_id`` are added by the From 1f31ca0ef77343ddb61770fd9418518759054a64 Mon Sep 17 00:00:00 2001 From: Sergiy Matusevych Date: Thu, 20 Mar 2025 13:33:59 -0700 Subject: [PATCH 34/35] Update mlos_bench/mlos_bench/environments/__init__.py Co-authored-by: Brian Kroth --- mlos_bench/mlos_bench/environments/__init__.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mlos_bench/mlos_bench/environments/__init__.py b/mlos_bench/mlos_bench/environments/__init__.py index 78855b499b..04675be943 100644 --- a/mlos_bench/mlos_bench/environments/__init__.py +++ b/mlos_bench/mlos_bench/environments/__init__.py @@ -354,7 +354,7 @@ We will discuss passing the parameters to external scripts and using them in referencing files and directories in local and shared storage in the documentation of the concrete -:py:class:`~.Environment` implementations. +:py:class:`~.Environment` implementations, especially :py:class:`~mlos_bench.environments.script_env.ScriptEnv` and :py:class:`~mlos_bench.environments.local.local_env.LocalEnv`. Environment Services ++++++++++++++++++++ From 308dff53e05831408b160cfb0c0bc24f961819af Mon Sep 17 00:00:00 2001 From: Sergiy Matusevych Date: Thu, 20 Mar 2025 13:35:23 -0700 Subject: [PATCH 35/35] merge in PR suggestions from Brian --- mlos_bench/mlos_bench/environments/__init__.py | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/mlos_bench/mlos_bench/environments/__init__.py b/mlos_bench/mlos_bench/environments/__init__.py index 04675be943..e8519556da 100644 --- a/mlos_bench/mlos_bench/environments/__init__.py +++ b/mlos_bench/mlos_bench/environments/__init__.py @@ -340,9 +340,9 @@ :py:attr:`~.Environment.parameters` data in their versions of :py:meth:`~.Environment.setup` and other methods. For example, :py:class:`~mlos_bench.environments.remote.vm_env.VMEnv` would then pass the :py:attr:`~.Environment.parameters` into an ARM template when provisioning a new VM, -and :py:class:`~.LocalEnv` can dump them into a JSON file specified in the ``dump_params_file`` -config property, or/and cherry-pick some of these values and make them shell variables with the -``shell_env_params``. +and :py:class:`~mlos_bench.environments.local.local_env.LocalEnv` can dump them into a JSON file +specified in the ``dump_params_file`` config property, or/and cherry-pick some of these values +and make them shell variables with the ``shell_env_params``. A few `Well Known Parameters <../config/index.html#well-known-variables>`_ parameters like ``trial_id`` and ``trial_runner_id`` are added by the @@ -354,7 +354,9 @@ We will discuss passing the parameters to external scripts and using them in referencing files and directories in local and shared storage in the documentation of the concrete -:py:class:`~.Environment` implementations, especially :py:class:`~mlos_bench.environments.script_env.ScriptEnv` and :py:class:`~mlos_bench.environments.local.local_env.LocalEnv`. +:py:class:`~.Environment` implementations, especially +:py:class:`~mlos_bench.environments.script_env.ScriptEnv` and +:py:class:`~mlos_bench.environments.local.local_env.LocalEnv`. Environment Services ++++++++++++++++++++