Skip to content

Conversation

@NParsonsMO
Copy link
Collaborator

@NParsonsMO NParsonsMO commented Jan 9, 2026

Closes #294.

PR creation checklist for the developer

  • Has <issue_number> above ☝️ been replaced with the issue number?
  • Has main been selected as the base branch?
  • Does the feature branch name follow the format <issue_number>_<short_description_of_feature>?
  • Does the text of the PR title exactly match with the text (not including the issue number) of the issue title?
  • Have appropriate reviewers been added to the PR (once it is ready for review)?
  • Has the PR been assigned to the developer(s)?
  • Have the same labels as on the issue (except for the good first issue label) been added to the PR?
  • Has the Climate Model Evaluation Workflow (CMEW) project been added to the PR?
  • Has the appropriate milestone been added to the PR?

Definition of Done for the developer

  • Does the change in this PR address the above issue / have all acceptance criteria been met?
  • Does the change in this PR follow the requirements in the wiki: Developer Guide (including copyrights)?
  • Have new tests related to the change been added?
  • Do all the GitHub workflow checks pass?
  • Do all the tests run locally and pass? (Note: the tests are not run by the GitHub workflow, see wiki: Run the tests locally)
  • Has the API documentation (e.g. docstrings in Python modules) related to the change been updated appropriately?
  • Has the user documentation (i.e. everything in the doc directory) related to the change been updated appropriately, including the Quick Start section?
  • Do the HTML pages render correctly? (See wiki: Build the documentation locally)

PR creation checklist for the reviewer

  • Has <issue_number> above ☝️ been replaced with the issue number?
  • Has main been selected as the base branch?
  • Does the feature branch name follow the format <issue_number>_<short_description_of_feature>?
  • Does the text of the PR title exactly match with the text (not including the issue number) of the issue title?
  • Have appropriate reviewers been added to the PR (once it is ready for review)?
  • Has the PR been assigned to the developer(s)?
  • Have the same labels as on the issue (except for the good first issue label) been added to the PR?
  • Has the Climate Model Evaluation Workflow (CMEW) project been added to the PR?
  • Has the appropriate milestone been added to the PR?

Definition of Done for the reviewer

  • Does the change in this PR address the above issue / have all acceptance criteria been met?
  • Does the change in this PR follow the requirements in the wiki: Developer Guide (including copyrights)?
  • Have new tests related to the change been added? N/A
  • Do all the GitHub workflow checks pass?
  • Do all the tests run locally and pass? (Note: the tests are not run by the GitHub workflow, see wiki: Run the tests locally)
  • Has the API documentation (e.g. docstrings in Python modules) related to the change been updated appropriately? N/A
  • Has the user documentation (i.e. everything in the doc directory) related to the change been updated appropriately, including the Quick Start section? N/A
  • Do the HTML pages render correctly? (See wiki: Build the documentation locally)

@NParsonsMO NParsonsMO linked an issue Jan 9, 2026 that may be closed by this pull request
@NParsonsMO NParsonsMO changed the title Moving config_user.yml to its own directory and removing flag from co… Get CMEW to work with new-style ESMValTool configuration files Jan 9, 2026
@NParsonsMO NParsonsMO self-assigned this Jan 9, 2026
@NParsonsMO NParsonsMO added the configure Anything related to configuration label Jan 9, 2026
@NParsonsMO NParsonsMO marked this pull request as ready for review January 9, 2026 15:14
@NParsonsMO NParsonsMO requested a review from ehogan January 9, 2026 15:14
@mo-nikosbaltas mo-nikosbaltas self-requested a review January 12, 2026 10:10
Copy link
Collaborator

@mo-nikosbaltas mo-nikosbaltas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

run_recipe_radiation_budget failed.

mo-nikosbaltas
mo-nikosbaltas previously approved these changes Jan 12, 2026
Copy link
Collaborator

@mo-nikosbaltas mo-nikosbaltas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to debug and I think the changes you made are incorrect, unless I am using an old version of ESMValTool.
In the CMEW/app/run_recipe/rose-app.conf , you stopped passing the CMEW-generated user config to ESMValTool and instead you set the env ESMVALTOOL_CONFIG_DIR
As a result, ESMValTool falls back to its default configuration search behaviour. The job.err proves that:
Reading configuration files from:
.../esmvalcore/config/.../defaults (defaults)
/home/users/nikolaos.baltas/.esmvaltool/config-user.yml (single configuration file
So ESMValTool is not reading the config you generated under:
/home/users/nikolaos.baltas/cylc-run/CMEW/run19/share/etc/user_config/config-user.yml
More detailed job.err extract.

2026-01-12 21:11:29,612 UTC [2912639] INFO    Reading configuration files from:
/data/apps/sss/environments/esmvaltool-2.12.0/lib/python3.12/site-packages/esmvalcore/config/configurations/defaults (defaults)
/home/users/nikolaos.baltas/.esmvaltool/config-user.yml (single configuration file [deprecated])
2026-01-12 21:11:29,612 UTC [2912639] INFO    Writing program log files to:
/home/users/nikolaos.baltas/esmvaltool_output/recipe_radiation_budget_20260112_211129/run/main_log.txt
/home/users/nikolaos.baltas/esmvaltool_output/recipe_radiation_budget_20260112_211129/run/main_log_debug.txt
/home/users/nikolaos.baltas/esmvaltool_output/recipe_radiation_budget_20260112_211129/run/cmor_log.txt

The question is whether EsmValTool 2.12.0 supports ESMVALTOOL_CONFIG_DIR. It does not seem to. Do we need to use a different version?

@mo-nikosbaltas mo-nikosbaltas self-requested a review January 13, 2026 08:52
@NParsonsMO
Copy link
Collaborator Author

For me it is running! I get this in the logs:
image

So I'll see if I can work out why it isn't for you. Does it still happen if you don't load the esmvaltool community module independently @mo-nikosbaltas, and just cylc vip -O metoffice in a clean terminal?

@mo-nikosbaltas
Copy link
Collaborator

I tried on a clean terminal using your PR branch, and still got the same error.
git branch
282-enable-switching-off-cdds-extract
286-run-two-model-runs-through-standardise_model_data
287-feed-model_id-and-variant_label-to-recipe
main

  • pr-324
    [nikolaos.baltas@cazldf0000FI CMEW]$ cylc vip -O metoffice

run_recipe_radiation_budget/01/job.err

/data/apps/sss/environments/esmvaltool-2.12.0/lib/python3.12/site-packages/esmvalcore/config/_config_object.py:639: ESMValCoreDeprecationWarning: Usage of the single configuration file ~/.esmvaltool/config-user.yml or specifying it via CLI argument `--config_file` has been deprecated in ESMValCore version 2.12.0 and is scheduled for removal in version 2.14.0. Please run `mkdir -p ~/.config/esmvaltool && mv /home/users/nikolaos.baltas/.esmvaltool/config-user.yml ~/.config/esmvaltool` (or alternatively use a custom `--config_dir`) and omit `--config_file`.
  config_obj.reload()
/data/apps/sss/environments/esmvaltool-2.12.0/lib/python3.12/site-packages/esmvalcore/config/_config_validators.py:405: ESMValCoreDeprecationWarning: The configuration option or command line argument `config_file` has been deprecated in ESMValCore version 2.12.0 and is scheduled for removal in version 2.14.0. Please use the option `config_dir` instead.
  _handle_deprecation(option, deprecated_version, remove_version, more_info)
/data/apps/sss/environments/esmvaltool-2.12.0/lib/python3.12/site-packages/esmvalcore/config/_config_validators.py:405: ESMValCoreDeprecationWarning: The configuration option or command line argument `config_file` has been deprecated in ESMValCore version 2.12.0 and is scheduled for removal in version 2.14.0. Please use the option `config_dir` instead.
  _handle_deprecation(option, deprecated_version, remove_version, more_info)
[FAIL] cmew-esmvaltool-env esmvaltool run ${RECIPE_PATH} # return-code=1
2026-01-13T11:12:27Z CRITICAL - failed/ERR

@mo-nikosbaltas
Copy link
Collaborator

Further investigation on the failure I was getting.
I managed to diagnose the error I was getting when running run_recipe_radiation_budget. I had a config file (stale) ~/.esmvaltool/config-user.yml which probably was not compatible and ESMValTool was picking this up. When I removed this file the run was successful. worked. Just to confirm that this was the only reason of this failure, it would be good for @NParsonsMO to check whether there was this ~/.esmvaltool/config-user.yml file in her environment. Getting this confirmation I will approve this PR.

@mo-nikosbaltas
Copy link
Collaborator

mo-nikosbaltas commented Jan 14, 2026

@NParsonsMO check #336 for further explanation.
In order to use CMEW's configuration and not the ESMValtool default you need to make the following changes:
In CMEW/app/run_recipe/rose-app.conf

[command]
-default=cmew-esmvaltool-env esmvaltool run ${RECIPE_PATH}
+default=cmew-esmvaltool-env esmvaltool run ${RECIPE_PATH} --config_dir=${ESMVALTOOL_CONFIG_DIR}

Also in:

CMEW/app/configure_recipe/bin/configure_recipe.py
         "auxiliary_data_dir": "",
-        "config_file": values.get("USER_CONFIG_PATH", None),
+        # "config_file": values.get("USER_CONFIG_PATH", None),
         "config_developer_file": config_developer_file,

This line "config_file": values.get("USER_CONFIG_PATH", None) should be removed. I think it was added for testing. I commented out for my tests. It is not needed and seems confusing and problematic.
• config_file is explicitly flagged as deprecated (see the warning in the logs), and it is scheduled for removal.
• It is conceptually odd: a config file containing a field that points to itself. With --config_dir, this is not needed.
• Relying on deprecated keys increases the risk of future breakage.

I made the changes myself and the runs succeeded. From the logs I can confirm that we are now using CMEW's configuration.
Here is a snippet from /home/users/nikolaos.baltas/cylc-run/CMEW/run28/log/job/1/run_recipe_radiation_budget/01/job.out:

2026-01-14 14:43:10,272 UTC [623729] INFO    Package versions
2026-01-14 14:43:10,272 UTC [623729] INFO    ----------------
2026-01-14 14:43:10,272 UTC [623729] INFO    ESMValCore: 2.12.0
2026-01-14 14:43:10,272 UTC [623729] INFO    ESMValTool: 2.12.0
2026-01-14 14:43:10,272 UTC [623729] INFO    ----------------
2026-01-14 14:43:10,274 UTC [623729] INFO    Reading configuration files from:
/data/apps/sss/environments/esmvaltool-2.12.0/lib/python3.12/site-packages/esmvalcore/config/configurations/defaults (defaults)
/home/users/nikolaos.baltas/cylc-run/CMEW/run28/share/etc/user_config (ESMVALTOOL_CONFIG_DIR environment variable)
/home/users/nikolaos.baltas/cylc-run/CMEW/run28/share/etc/user_config (command line argument)
2026-01-14 14:43:10,274 UTC [623729] INFO    Writing program log files to:
......
2026-01-14 14:43:39,243 UTC [623729] INFO    Wrote recipe output to:
file:///home/users/nikolaos.baltas/cylc-run/CMEW/run28/share/cycle/1/recipe_radiation_budget_20260114_144310/index.html
2026-01-14 14:43:39,244 UTC [623729] INFO    Ending the Earth System Model Evaluation Tool at time: 2026-01-14 14:43:39 UTC
2026-01-14 14:43:39,244 UTC [623729] INFO    Time for running the recipe was: 0:00:26.369468
2026-01-14 14:43:39,815 UTC [623729] INFO    Maximum memory used (estimate): 2.7 GB
2026-01-14 14:43:39,816 UTC [623729] INFO    Sampled every second. It may be inaccurate if short but high spikes in memory consumption occur.
2026-01-14 14:43:39,818 UTC [623729] WARNING Input data is not (fully) CMOR-compliant, see /home/users/nikolaos.baltas/cylc-run/CMEW/run28/share/cycle/1/recipe_radiation_budget_20260114_144310/run/cmor_log.txt for details
2026-01-14 14:43:39,819 UTC [623729] INFO    Run was successful
2026-01-14T14:43:41Z INFO - succeeded

Plots were created:

cd /home/users/nikolaos.baltas/cylc-run/CMEW/run28/share/cycle/1/recipe_radiation_budget_20260114_144310
gio open index.html

Copy link
Collaborator

@mo-nikosbaltas mo-nikosbaltas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comments above. I run the pr-324 branch with the changes mentioned.

@ehogan
Copy link
Member

ehogan commented Jan 15, 2026

Thanks for investigating the problem @mo-nikosbaltas! 🎉

In order to use CMEW's configuration and not the ESMValtool default you need to make the following changes: In CMEW/app/run_recipe/rose-app.conf

[command]
-default=cmew-esmvaltool-env esmvaltool run ${RECIPE_PATH}
+default=cmew-esmvaltool-env esmvaltool run ${RECIPE_PATH} --config_dir=${ESMVALTOOL_CONFIG_DIR}

As described in #294 (comment), the ESMVALTOOL_CONFIG_DIR environment variable must be used, not the --config-dir argument.

Also in:

CMEW/app/configure_recipe/bin/configure_recipe.py
         "auxiliary_data_dir": "",
-        "config_file": values.get("USER_CONFIG_PATH", None),
+        # "config_file": values.get("USER_CONFIG_PATH", None),
         "config_developer_file": config_developer_file,

This line "config_file": values.get("USER_CONFIG_PATH", None) should be removed. I think it was added for testing. I commented out for my tests. It is not needed and seems confusing and problematic.

This was needed in the old style configuration file. I expect this is the reason why the directory provided by the ESMVALTOOL_CONFIG_DIR environment variable was not getting used. I hope this change alone will solve the problem 👍

@NParsonsMO
Copy link
Collaborator Author

Further investigation on the failure I was getting. I managed to diagnose the error I was getting when running run_recipe_radiation_budget. I had a config file (stale) ~/.esmvaltool/config-user.yml which probably was not compatible and ESMValTool was picking this up. When I removed this file the run was successful. worked. Just to confirm that this was the only reason of this failure, it would be good for @NParsonsMO to check whether there was this ~/.esmvaltool/config-user.yml file in her environment. Getting this confirmation I will approve this PR.

Mine is not at ~/.esmvaltool/config-user.yml, it is at ~/.config/esmvaltool/config-user.yml.

However, I was under the impression (from the conversation with @ehogan after stand-up a few days ago) that we were not meant to use --config_dir=$... as this adds to the places that ESMValTool searches, whereas we want to confine them. It is possible that I have misunderstood.

@NParsonsMO
Copy link
Collaborator Author

Removing the line "config_file": values.get("USER_CONFIG_PATH", None), seems to have fixed it.

I hadn't spotted that line, thank-you!

image

@NParsonsMO
Copy link
Collaborator Author

@mo-nikosbaltas or @ehogan , it would be really good if at least one of you could double check my last commit removing the test,
7a32dbd
as this didn't look like any test I have seen before and it's possible that I've now stopped it checking for other things that we do still expect to be there.

mo-nikosbaltas
mo-nikosbaltas previously approved these changes Jan 15, 2026
Copy link
Collaborator

@mo-nikosbaltas mo-nikosbaltas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All looks good. Tests passed. Approved.

Copy link
Member

@ehogan ehogan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @NParsonsMO 🥳

# Write the updated configuration values to the file defined by
# 'USER_CONFIG_PATH'.
user_config_path = values["USER_CONFIG_PATH"]
os.makedirs(os.path.dirname(user_config_path), exist_ok=True)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not necessary. The directory is made by the configure_recipe/rose-app.conf file.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I literally comment out the line, then the run fails.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Conclusion:

Once the changes below were implemented then this line could be removed.

CMEW/flow.cylc Outdated
Comment on lines 39 to 40
ESMVALTOOL_CONFIG_DIR = ${USER_CONFIG_DIR}/user_config
USER_CONFIG_PATH = ${ESMVALTOOL_CONFIG_DIR}/config-user.yml
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thinking about how the configuration now works in ESMValTool, I wonder whether we need to drop the term USER_CONFIG_DIR, since it's now one big configuration directory (USER_CONFIG_PATH is ok for now to reference the file itself, but that might change in the future when we look at the formatting / contents of that file).

Using ESMVALTOOL_CONFIG_DIR in favour of USER_CONFIG_DIR would minimise the use of variables here, and should solve the directory creation issue (as mentioned in my previous comment).

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image

Sorry, I am not following (Monday morning!).
Which directory should be called what?

@NParsonsMO
Copy link
Collaborator Author

Following conversation this morning, I have tidied the directories to the following structure within etc
image

Note that I haven't also moved config-developer.yml, as this PR didn't initially, and I think it is probably more aligned with issue #212

Copy link
Collaborator

@mo-nikosbaltas mo-nikosbaltas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Failed unittest: cylc vip -O metoffice -O unittest

@NParsonsMO
Copy link
Collaborator Author

Failed unittest: cylc vip -O metoffice -O unittest

The [9 remaining] unit tests run fine for me locally with pytest and also successfully with cylc vip -O metoffice -O test, so I'm really confused.

Maybe @ehogan can help?!

@mo-nikosbaltas
Copy link
Collaborator

When I run : cylc vip -O metoffice -O test then all unit tests passed. Based on these results I approve.

mo-nikosbaltas
mo-nikosbaltas previously approved these changes Jan 19, 2026
@ehogan
Copy link
Member

ehogan commented Jan 20, 2026

Failed unittest: cylc vip -O metoffice -O unittest

The [9 remaining] unit tests run fine for me locally with pytest and also successfully with cylc vip -O metoffice -O test, so I'm really confused.

Maybe @ehogan can help?!

The error when running cylc vip -O metoffice -O unittest is:

==================================== ERRORS ====================================
___ ERROR collecting configure_standardise/bin/test_create_variables_file.py ___
configure_standardise/bin/test_create_variables_file.py:7: in <module>
    from create_variables_file import parse_variables_from_recipe
configure_standardise/bin/create_variables_file.py:8: in <module>
    from esmvalcore.experimental.recipe import Recipe
/esmvaltool-2.12.0/lib/python3.12/site-packages/esmvalcore/experimental/__init__.py:6: in <module>
    from esmvalcore.config import CFG
/esmvaltool-2.12.0/lib/python3.12/site-packages/esmvalcore/config/__init__.py:16: in <module>
    from ._config_object import CFG, Config, Session
/esmvaltool-2.12.0/lib/python3.12/site-packages/esmvalcore/config/_config_object.py:63: in <module>
    USER_CONFIG_DIR = _get_user_config_dir()
                      ^^^^^^^^^^^^^^^^^^^^^^
/esmvaltool-2.12.0/lib/python3.12/site-packages/esmvalcore/config/_config_object.py:46: in _get_user_config_dir
    raise NotADirectoryError(
E   NotADirectoryError: Invalid configuration directory specified via ESMVALTOOL_CONFIG_DIR environment variable: ${HOME}/cylc-run/CMEW/run20/share/etc/esmvaltool_config is not an existing directory
=========================== short test summary info ============================
ERROR configure_standardise/bin/test_create_variables_file.py - NotADirectory...
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 4.83s ===============================

This error occurs because the configure_recipe task (which is where the ESMVALTOOL_CONFIG_DIR directory is created) isn't run when -O unittest is run.

To solve this, we could move the creation of the ESMVALTOOL_CONFIG_DIR to the install_env_file task? 🤔

@NParsonsMO
Copy link
Collaborator Author

This error occurs because the configure_recipe task (which is where the ESMVALTOOL_CONFIG_DIR directory is created) isn't run when -O unittest is run.

To solve this, we could move the creation of the ESMVALTOOL_CONFIG_DIR to the install_env_file task? 🤔

I think the creation of this directory is really part of configure_recipe rather than part of install_env_file, so this seems like changing something to fit a slightly incorrect metric? The unit tests should really be picking up stuff that has gone wrong, but actually nothing has gone wrong here - it has been created by the point that we need it, so I wonder if there is instead someway to improve the test?

@ehogan
Copy link
Member

ehogan commented Jan 21, 2026

I think the creation of this directory is really part of configure_recipe rather than part of install_env_file, so this seems like changing something to fit a slightly incorrect metric? The unit tests should really be picking up stuff that has gone wrong, but actually nothing has gone wrong here - it has been created by the point that we need it, so I wonder if there is instead someway to improve the test?

Great question! 🥳 The traceback shows that ESMValTool is checking that the directory defined by ESMVALTOOL_CONFIG_DIR exists when the Recipe class is imported, which means the directory needs to exist before the Recipe class is imported. So I'm not sure there's anything we can do to improve the test, we just need to made sure the directory exists before importing the Recipe class. Any ideas on how best to do this? 🤔

@NParsonsMO
Copy link
Collaborator Author

NParsonsMO commented Jan 21, 2026

Great question! 🥳 The traceback shows that ESMValTool is checking that the directory defined by ESMVALTOOL_CONFIG_DIR exists when the Recipe class is imported, which means the directory needs to exist before the Recipe class is imported. So I'm not sure there's anything we can do to improve the test, we just need to made sure the directory exists before importing the Recipe class. Any ideas on how best to do this? 🤔

Well I can get around the unit tests by adding lines to CMEW/app/unittest/rose-app.conf, but it seems like cheating (and then has duplication within the repo).
image

@ehogan
Copy link
Member

ehogan commented Jan 21, 2026

I just had a thought; is it because ESMVALTOOL_CONFIG_DIR is defined in the root environment? That means it's defined for each task. What if we only defined it for the config run_recipe task (by only having it in the rose-app.conf file)? Maybe if it's not defined in the environment for the other tasks then ESMValTool won't try to check if the directory exists? 🤔

@NParsonsMO
Copy link
Collaborator Author

I just had a thought; is it because ESMVALTOOL_CONFIG_DIR is defined in the root environment? That means it's defined for each task. What if we only defined it for the config task (by only having it in the rose-app.conf file)? Maybe if it's not defined in the environment for the other tasks then ESMValTool won't try to check if the directory exists? 🤔

We need in available to the run_recipe task, to solve the main issue.

I tried define it (with and without the associated USER_CONFIG_PATH) under the [[RECIPE]] section, as I thought all the apps needing it would inherit that, but I haven't yet found anywhere else that runs...

Do you want me to keep looking, or are you happy to create it in the install_env task, even though it may not be obvious in future why it's there?

Or, alternatively, have the mode=mkdir twice, once in configure_recipe and once in unittest?

Copy link
Member

@ehogan ehogan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost there! 🥳

"input_key, output_key, expected",
[
(None, "remove_preproc_dir", False),
("USER_CONFIG_PATH", "config_file", "userpath"),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The copyright in this file needs updating 😊

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, done.

@NParsonsMO NParsonsMO requested a review from ehogan January 23, 2026 09:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

configure Anything related to configuration

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Get CMEW to work with new-style ESMValTool configuration files

4 participants