Skip to content

Update rose-stem to extract MOCI and UM-related repos from Git #29

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ericaneininger wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Update rose-stem to extract MOCI and UM-related repos from Git #29

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a0075be
Extract MOCI and UM-related repos from Git
ericaneininger Feb 4, 2026
b753aec
Update contributors file
ericaneininger Feb 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CONTRIBUTORS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,4 @@
| ----------- | --------- | ----------- | ---- |
| james-bruten-mo | James Bruten | Met Office | 2025-12-09 |
| Pierre-siddall | Pierre Siddall | Met Office | 2025-01-19 |
| ericaneininger | Erica Neininger | Met Office | 2026-02-04 |
47 changes: 47 additions & 0 deletions dependencies.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
##############################################################################
# (c) Crown copyright 2026 Met Office. All rights reserved.
# The file LICENCE, distributed with this code, contains details of the terms
# under which the code may be used.
##############################################################################

# This file contains a list of all linked external repositories containing code
# and data required to build and run the moci rose-stem
#
# Each section is the name of the repository
# source: Is either the url to the repository (the upstream or
# a fork) or a path to a local clone with format `host:/path/to/clone`
# ref: Is a valid git reference. This can either be a commit hash, a tag or a
# branch name
#
# The entry for this repository will have blank values for both source and ref
# by default, which will cause the test suite to use the source code from the
# local clone. These can be filled to set a different source if desired.

moci:
source:
ref:

casim:
source: git@github.com:MetOffice/casim.git
ref: um13.1

jules:
source: git@github.com:MetOffice/jules-internal.git
ref: um13.1

shumlib:
source: git@github.com:MetOffice/shumlib.git
ref: um13.1

socrates:
source: git@github.com:MetOffice/socrates.git
ref: um13.1

ukca:
source: git@github.com:MetOffice/ukca.git
ref: um13.1

um:
source: git@github.com:MetOffice/um.git
ref: vn13.1

8 changes: 8 additions & 0 deletions rose-stem/app/extract_source/rose-app.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
[command]
default=python rose_stem_extract_source.py

[file:get_git_sources.py]
source=git:https://github.com/MetOffice/SimSys_Scripts.git::github_scripts/get_git_sources.py::main

[file:rose_stem_extract_source.py]
source=git:https://github.com/MetOffice/SimSys_Scripts.git::github_scripts/rose_stem_extract_source.py::main
13 changes: 7 additions & 6 deletions rose-stem/app/fcm_make_um/file/fcm-make_main.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,10 @@ $ldflags_coupling = -L$prism_path/lib -lpsmile.MPI1 -lmct -lmpeu -lscrip

include = $config_root_path/fcm-make/$platform_config_dir/um-$config_type-$optimisation_level.cfg$config_revision

extract.location{diff}[um] = $um_sources
extract.location{diff}[ukca] = $ukca_sources
extract.location{diff}[shumlib] = $shumlib_sources
extract.location{diff}[casim] = $casim_sources
extract.location{diff}[jules] = $jules_sources
extract.location{diff}[socrates] = $socrates_sources
extract.location{diff}[um] = $source_directory_um
extract.location{diff}[casim] = $source_directory_casim
extract.location{diff}[jules] = $source_directory_jules
extract.location{diff}[socrates] = $source_directory_socrates
extract.location{diff}[shumlib] = $source_directory_shumlib
extract.location{diff}[ukca] = $source_directory_ukca

12 changes: 6 additions & 6 deletions rose-stem/app/fcm_make_um/file/fcm-make_slash.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ $ldflags_coupling = -L\$prism_path/lib -lpsmile.MPI1 -lmct -lmpeu -lscrip

include = $config_root_path/fcm-make/$platform_config_dir/um-$config_type-$optimisation_level.cfg$config_revision

extract.location{diff}[um] = $um_sources
extract.location{diff}[ukca] = $ukca_sources
extract.location{diff}[shumlib] = $shumlib_sources
extract.location{diff}[casim] = $casim_sources
extract.location{diff}[jules] = $jules_sources
extract.location{diff}[socrates] = $socrates_sources
extract.location{diff}[um] = $source_directory_um
extract.location{diff}[casim] = $source_directory_casim
extract.location{diff}[jules] = $source_directory_jules
extract.location{diff}[socrates] = $source_directory_socrates
extract.location{diff}[shumlib] = $source_directory_shumlib
extract.location{diff}[ukca] = $source_directory_ukca
27 changes: 9 additions & 18 deletions rose-stem/app/fcm_make_um/rose-app.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,6 @@ meta=um-fcm-make/vn13.1
[env]
COUPLER=oasis3_mct
DR_HOOK=false
casim_rev=um13.1
casim_sources=
compile_atmos=preprocess-atmos build-atmos
!!compile_createbc=preprocess-createbc build-createbc
!!compile_crmstyle_coarse_grid=preprocess-crmstyle_coarse_grid build-crmstyle_coarse_grid
Expand All @@ -13,16 +11,14 @@ compile_recon=preprocess-recon build-recon
!!compile_scm=preprocess-scm build-scm
!!compile_sstpert_lib=preprocess-sstpert_lib build-sstpert_lib
!!compile_wafccb_lib=preprocess-wafccb_lib build-wafccb_lib
config_revision=@vn13.1
config_root_path=fcm:um.xm_tr
config_revision=
config_root_path=$SOURCE_DIRECTORY/um
config_type=atmos
eccodes=false
extract=extract
fcflags_overrides=
gcom_root_path=/common/internal/umdir/lib/cce-15.0.0/gcom/gcom-7.9.2
gwd_ussp_precision=double
jules_rev=um13.1
jules_sources=
land_surface_model=jules
ldflags_overrides_prefix=
ldflags_overrides_suffix=
Expand All @@ -39,17 +35,12 @@ platform_config_dir=$CONFIG
portio_version=2A
prebuild=
!!recon_mpi=parallel
shumlib_rev=um13.1
shumlib_sources=
socrates_rev=um13.1
socrates_sources=
stash_version=1A
thread_utils=false
timer_version=3A
ukca_rev=um13.1
ukca_sources=
um_rev=vn13.1
um_sources=
source_directory_casim=$SOURCE_DIRECTORY/casim
source_directory_jules=$SOURCE_DIRECTORY/jules
source_directory_shumlib=$SOURCE_DIRECTORY/shumlib
source_directory_socrates=$SOURCE_DIRECTORY/socrates
source_directory_ukca=$SOURCE_DIRECTORY/ukca
source_directory_um=$SOURCE_DIRECTORY/um

[file:fcm-make.cfg]
source=$ROSE_SUITE_DIR/app/fcm_make_um/file/fcm-make_${FCM_MAKE_FILE}.cfg
source=$CYLC_WORKFLOW_RUN_DIR/app/fcm_make_um/file/fcm-make_${FCM_MAKE_FILE}.cfg
57 changes: 55 additions & 2 deletions rose-stem/flow.cylc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,29 @@
#!jinja2
{% from "cylc.flow" import LOG %}
{% from "utils" import get_site %}
{% from "read_sources" import read_sources %}

{% if SITE is not defined %}
{% set SITE = get_site() %}
{% endif %}
{% do LOG.info("Site: " + SITE) %}

{% set SOURCE_DIRECTORY = ROSE_ORIG_HOST~":"~CYLC_WORKFLOW_SRC_DIR~"/.." %}
{# Include log message for suite_report #}
{% do LOG.info("MOCI SOURCE CLONE="~SOURCE_DIRECTORY) %}

{# Read the sources of dependencies from the dependencies.yaml file #}
{% set dependencies = read_sources(SOURCE_DIRECTORY, "moci", USE_HEADS) %}

{# Read in groups to run #}
{% if group is defined %}
{% set RUN_NAMES = group %}
{% elif g is defined %}
{% set RUN_NAMES = g %}
{% endif %}
{% do LOG.info("Running groups: "~RUN_NAMES|join('
')) %}

{% set TASK_RUN = "rose task-run --verbose -O '(" + SITE + ")'"%}
{% set PP_CYCLES = range(110,113)|list + range(201,213)|list + [301,]|list %}
{% set PP_UNICICLES_CYCLES = [112,203,206,209,212,]|list %}
Expand Down Expand Up @@ -33,6 +58,8 @@
[scheduler]
UTC mode = True
[[events]]
shutdown handlers = "suite_report_git.py -S $CYLC_WORKFLOW_RUN_DIR"
stall handlers = "suite_report_git.py -S $CYLC_WORKFLOW_RUN_DIR"
stall timeout handlers = "echo %worklfow%s %event%s"
stall timeout = PT6H
abort on stall timeout = true
Expand Down Expand Up @@ -69,11 +96,37 @@
submission timeout = PT12H # 12 hours
execution timeout = PT3H # 3 hours
[[[environment]]]
SOURCE_MOCI_BASE = {{ HOST_SOURCE_MOCI_BASE }}

SOURCE_DIRECTORY = $CYLC_WORKFLOW_SHARE_DIR/source
SOURCE_MOCI_BASE = $SOURCE_DIRECTORY/moci

[[default_task]]
script = "echo [ERROR] Please select a valid group within the test suite to test the component that has been modified >&2; exit 1"

[[SOURCE_EXTRACTION]]
post-script = """
cp $SOURCE_DIRECTORY/moci/dependencies.yaml $CYLC_WORKFLOW_RUN_DIR
cp $CYLC_WORKFLOW_RUN_DIR/bin/github_scripts/suite_report_git.py $CYLC_WORKFLOW_RUN_DIR/bin
cp $CYLC_WORKFLOW_RUN_DIR/bin/github_scripts/suite_data.py $CYLC_WORKFLOW_RUN_DIR/bin
cp $CYLC_WORKFLOW_RUN_DIR/bin/github_scripts/git_bdiff.py $CYLC_WORKFLOW_RUN_DIR/bin
cp $CYLC_WORKFLOW_RUN_DIR/bin/github_scripts/get_git_sources.py $CYLC_WORKFLOW_RUN_DIR/bin
"""
[[[environment]]]
ROSE_TASK_APP = extract_source
DEPENDENCIES = {{dependencies}}
USE_MIRRORS = {{USE_MIRRORS}}
USE_TOKENS = false
{% if USE_MIRRORS %}
{% if GIT_MIRROR_LOC %}
GIT_MIRROR_LOC = {{GIT_MIRROR_LOC}}
{% else %}
{{ raise(
"Trying to run with mirrors without setting a mirror location. "
"Ensure 'GIT_MIRROR_LOC' set."
) }}
{% endif %}
{% endif %}


[[HOUSEKEEPING]]
{% if HOUSEKEEPING == true %}
script = "rose task-run --verbose"
Expand Down
14 changes: 7 additions & 7 deletions rose-stem/graph-drivers.cylc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
{# Setup graph required to run postproc tasks #}
{% set SPACER = " " %}
{%- do name_graphs.update({
"test_drivers_unit" : SPACER + "fcm_make_drivers => drivers_unittests & mct_validate_unittests" + (" => drivers_local_housekeeping" if HOUSEKEEPING else ""),
"test_drivers_misc" : SPACER + "fcm_make_drivers => drivers_check_python_tabs" + (" => drivers_local_housekeeping" if HOUSEKEEPING else ""),
"test_drivers_unit" : SPACER + "extract_source => fcm_make_drivers => drivers_unittests & mct_validate_unittests" + (" => drivers_local_housekeeping" if HOUSEKEEPING else ""),
"test_drivers_misc" : SPACER + "extract_source => fcm_make_drivers => drivers_check_python_tabs" + (" => drivers_local_housekeeping" if HOUSEKEEPING else ""),
"test_drivers_run" : """
fcm_make_drivers => fcm_make2_drivers => coupled
fcm_make_um => recon => coupled
fcm_make_ocean => fcm_make2_ocean => recon
extract_source => fcm_make_drivers => fcm_make2_drivers => coupled
extract_source => extract_source_hpc => fcm_make_um => recon => coupled
extract_source => fcm_make_ocean => fcm_make2_ocean => recon
install_ancil => recon
recon => coupled => check_coupled" + (" => drivers_hpc_housekeeping" if HOUSEKEEPING else "")+"
""",
"test_drivers_verify_metrics" : SPACER + "test_verify_metrics" + (" => drivers_local_housekeeping" if HOUSEKEEPING else ""),
"test_drivers_plot_cpmip" : SPACER + "fcm_make_drivers => test_plot_cpmip_summary" + (" => drivers_local_housekeeping" if HOUSEKEEPING else ""),
"test_drivers_verify_metrics" : SPACER + "extract_source => test_verify_metrics" + (" => drivers_local_housekeeping" if HOUSEKEEPING else ""),
"test_drivers_plot_cpmip" : SPACER + "extract_source => fcm_make_drivers => test_plot_cpmip_summary" + (" => drivers_local_housekeeping" if HOUSEKEEPING else ""),
}) %}

{# Update groups with drivers tasks. -#}
Expand Down
4 changes: 2 additions & 2 deletions rose-stem/graph-postproc.cylc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
{# Setup graph required to run postproc tasks #}

{% if SITE in MULTISTEP_FCM %}
{% set FCMPP_GRAPH = 'fcm_make_pp => fcm_make2_pp ' %}
{% set FCMPP_GRAPH = 'extract_source => fcm_make_pp => fcm_make2_pp' %}
{% else %}
{% set FCMPP_GRAPH = 'fcm_make_pp' %}
{% set FCMPP_GRAPH = 'extract_source => fcm_make_pp' %}
{% endif %}


Expand Down
66 changes: 66 additions & 0 deletions rose-stem/lib/python/read_sources.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
##############################################################################
# (c) Crown copyright 2026 Met Office. All rights reserved.
# The file LICENCE, distributed with this code, contains details of the terms
# under which the code may be used.
##############################################################################

import yaml
import tempfile
import os
from subprocess import run
from shutil import rmtree

def get_dependencies_file(wc_loc):
"""
Copy the dependencies file to a temporary directory on the local machine that can be
read.
"""

tempdir = tempfile.mkdtemp()

try:
host, path = wc_loc.split(":", 1)
path = os.path.join(path, "dependencies.yaml")
copy_command = f"scp -o StrictHostKeyChecking=no {host}:"
except ValueError:
path = os.path.join(wc_loc, "dependencies.yaml")
copy_command = "cp "
copy_command += f"{path} {tempdir}"

result = run(
copy_command.split(), capture_output=True, text=True, timeout=120
)

# Raise an error if the returncode is positive
if result.returncode:
raise RuntimeError(
f"An error occured while running the command '{copy_command}' "
"in order to read the dependencies file. The error message is:\n\n"
f"'{result.stderr}'"
)

return tempdir

def read_sources(clone_source, repo, use_heads):
"""
Load the dependencies.yaml file as a dictionary
"""

dependencies_file = get_dependencies_file(clone_source)

with open(os.path.join(dependencies_file, "dependencies.yaml")) as stream:
dependencies = yaml.safe_load(stream)

if not dependencies[repo]["source"]:
dependencies[repo]["source"] = clone_source

# Populate parent, assume MetOffice is owner if not set
for dependency, values in dependencies.items():
if "parent" not in values:
dependencies[dependency]["parent"] = f"MetOffice/{dependency}.git"
if use_heads:
dependencies[dependency]["ref"] = ""

rmtree(dependencies_file)

return dependencies
20 changes: 20 additions & 0 deletions rose-stem/lib/python/utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
##############################################################################
# (c) Crown copyright 2025 Met Office. All rights reserved.
# The file LICENCE, distributed with this code, contains details of the terms
# under which the code may be used.
##############################################################################

from subprocess import Popen, PIPE


def get_site():
proc = Popen(
["rose", "config", "rose-stem", "automatic-options"], stdout=PIPE, text=True
)
out, _ = proc.communicate()
if proc.returncode or "SITE" not in out:
raise Exception('Could not determine the rose-stem "SITE"')
# At some sites there may be many variables that are returned by rose config rose-stem
# Try to just grab the thing after SITE= and then ignore anything afterwards
site = out.split("SITE=")[1].split(' ')[0].strip()
return site
7 changes: 5 additions & 2 deletions rose-stem/rose-suite.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
ROSE_STEM_VERSION=1
[file:bin/github_scripts]
#mode=mkdir
source=git:https://github.com/MetOffice/SimSys_Scripts.git::github_scripts/::main

[template variables]
ATM_PROCX=8
Expand All @@ -15,11 +17,12 @@ IOS_NPROC=0
NEMO_IPROC=15
NEMO_JPROC=17
POSTPROC_HOST='$ROSE_ORIG_HOST'
#PYTHON3=true
RCF_PROCX=4
RCF_PROCY=8
SPLIT_PP=true
SYSTEM_NAME='meto_azspice'
USE_LINUX_SERVER=true
VERBOSE=true
XIOS_NPROC=6
USE_HEADS=false
USE_MIRRORS=false
Loading