Performance comparison and performance tests

[mo-marqh]

PR Summary

Sci/Tech Reviewer: @iboutle
Code Reviewer: @mo-alistairp

This branch first adds a performance configuration for a C896 test run triggered multiple times, to enable cross run variability to be analysed. this test is not run regularly, but is derived from the weekly C896 test so should remain safe to use.

Following the adoption of #82 & #176 it is possible to compare timer based output from December's Performance analysis package branch (https://code.metoffice.gov.uk/trac/lfric_apps/browser/main/branches/pkg/Share/r15393_kpi_benchmark) with the current main.

We cannot run with complicated decompositions across panels until #138. But, we can run with 4704 32x32 ranks.

Using a 720 timestep C896 run with 4704 ranks then a run can be analysed with the svn package branch and the main.

Comparing test results for the legacy package branch and the current main shows a significant discrepancy in the timestepping time, within predominantly fast_physics

which is fixed by setting
["namelist:physics", "configure_segments"], ".true."
and allowing values of 0 for all segments (mis-configured in rose metadata)
mo-marqh@1e58ea2

this requires a macro, the results of the macro running on the code base are detailed in:
https://github.com/mo-marqh/lfric_apps/tree/performanceComparePostMacro
mo-marqh@16041eb

PKG branch from svn performance analysis Dec25 *1880s*

Routine	mintime(s)	meantime(s)	maxtime(s)	No.calls	%time	timepercall(s)
lfric_atm	1879.62	1879.72	1882.76	1	100.0000	1879.7165
gungho_initialise	53.66	53.70	53.78	1	2.8568	53.6994
fs.constructor	0.04	0.04	0.05	71	0.0022	0.0006
field.initialise	3.39	3.96	4.78	1488265	0.2105	0.0000
field.halo_ex_1	101.57	175.41	290.81	298450	9.3318	0.0006
halo_routing_creation	13.12	16.55	18.10	375709	0.8806	0.0000
xios_close_context_definition	8.87	8.91	8.98	1	0.4739	8.9072
finalise_xios_context	12.26	12.31	15.37	2	0.6547	6.1532
field.read	17.12	17.24	17.32	66	0.9174	0.2613
lfric_xios_read_fldg	17.12	17.24	17.32	66	0.9174	0.2613
field.halo_ex_9	0.00	0.00	0.01	7	0.0002	0.0006
runtime_constants.geometric	0.41	0.45	0.73	23	0.0237	0.0194
mappings.set_wind	17.87	62.31	156.70	2881	3.3150	0.0216
runtime_constants.fem	1.58	2.62	3.55	31	0.1392	0.0844
runtime_constants.mapping	0.15	0.16	0.27	1	0.0083	0.1564
runtime_constants.dycore	19.19	20.33	23.06	13	1.0816	1.5639
mass_matrix_solver_alg	61.86	66.56	70.86	3637	3.5407	0.0183
mappings.map_physics_fields	48.38	52.06	54.96	721	2.7697	0.0722
gungho_diagnostics_driver	11.00	11.94	13.72	721	0.6350	0.0166
field.write	3.36	3.67	4.19	274566	0.1952	0.0000
lfric_xios_write_fldg	2.24	2.56	3.06	12173	0.1361	0.0002
gungho_driver_first_timestep	35.57	35.66	35.70	1	1.8969	35.6562
gungho_driver_timestep	1811.50	1811.69	1811.78	720	96.3811	2.5162
semi_implicit_timestep	1799.85	1801.49	1802.17	720	95.8382	2.5021
dynamics.compute_si_operators	21.58	21.87	22.24	52	1.1633	0.4205
runtime_constants.solver	0.02	0.03	0.04	17	0.0015	0.0016
field.halo_ex_2	0.83	1.51	3.93	3808	0.0803	0.0004
slow_physics	373.65	375.78	379.94	720	19.9915	0.5219
cloud.fsd_condensate	1.81	2.61	4.15	720	0.1387	0.0036
locate_tropopause	0.09	0.13	0.18	720	0.0071	0.0002
diags.locate_tropopause	0.01	0.01	0.01	720	0.0005	0.0000
aerosol.glomap_clim	19.60	20.53	23.65	720	1.0920	0.0285
diags.glomap_clim	0.10	0.12	0.20	720	0.0064	0.0002
aerosol.stratosphere	0.03	0.09	0.27	720	0.0050	0.0001
microphysics.wb	27.86	40.85	61.15	720	2.1734	0.0567
diags.mphys	0.31	0.38	0.53	1440	0.0204	0.0003
radiation.illuminate	0.62	0.74	1.43	720	0.0396	0.0010
diags.illuminate	0.02	0.02	0.03	720	0.0013	0.0000
aerosol.radaer	46.12	61.43	164.63	48	3.2683	1.2799
diags.radaer	0.03	0.04	0.11	96	0.0022	0.0004
jules.radiation	0.40	0.82	1.18	720	0.0435	0.0011
radiation_diags	0.94	1.02	1.14	1440	0.0540	0.0007
radiation.sw	2.80	11.34	25.78	720	0.6032	0.0157
radiation.lw	11.34	22.48	48.88	720	1.1962	0.0312
cloud.pc2_radiation	1.96	2.53	3.90	720	0.1348	0.0035
diags.pc2_radiation	0.02	0.02	0.04	720	0.0013	0.0000
spectral_gwd	5.01	6.58	9.40	720	0.3498	0.0091
diags.spectral_gwd	0.06	0.07	0.10	1440	0.0039	0.0001
orographic_drag	0.39	3.00	11.94	720	0.1595	0.0042
diags.orog_gwd	0.07	0.08	0.11	1440	0.0044	0.0001
methane_oxidation	0.03	0.04	0.07	720	0.0020	0.0001
diags.methox	0.00	0.00	0.01	720	0.0003	0.0000
jules.explicit	10.43	36.89	111.17	720	1.9628	0.0512
diags.jules_exp	0.10	0.12	0.18	1440	0.0066	0.0001
bl.explicit	27.16	56.92	133.36	1440	3.0281	0.0395
runtime_constants.physics	0.00	0.02	0.08	1	0.0010	0.0185
diags.bl_exp	0.09	0.12	0.20	1440	0.0062	0.0001
cloud.bm_tau	4.69	6.08	7.46	720	0.3235	0.0084
cloud.pc2_checks	2.17	2.52	3.61	720	0.1339	0.0035
diags.pc2_checks	0.02	0.03	0.05	720	0.0016	0.0000
dynamics.rhs_alg	70.16	73.01	77.19	3600	3.8843	0.0203
dynamics.transport_predictors	20.23	21.83	23.57	720	1.1613	0.0303
dynamics.transport	257.44	262.08	266.98	1440	13.9424	0.1820
transport.controller_init	14.48	16.40	18.71	1440	0.8726	0.0114
runtime_constants.transport	0.01	0.01	0.02	6	0.0007	0.0021
transport.flux_precomp_init	2.41	2.57	2.74	11520	0.1369	0.0002
transport.flux_precomp	15.29	17.51	20.65	33840	0.9315	0.0005
transport.hori_dep_pts	7.08	9.17	12.26	4320	0.4876	0.0021
field.halo_ex_7	0.91	1.14	1.62	1440	0.0608	0.0008
transport.ffsl_vertical	49.38	53.45	57.16	44640	2.8434	0.0012
transport.ffsl_horizontal	110.12	116.71	129.05	13680	6.2089	0.0085
field.halo_ex_5	12.54	25.99	33.86	41044	1.3826	0.0006
field.halo_ex_3	3.62	5.74	8.09	13680	0.3055	0.0004
transport.wind_to_comps	3.37	3.62	3.99	1440	0.1927	0.0025
transport.sl_horizontal	26.68	28.48	30.73	8640	1.5152	0.0033
field.halo_ex_4	4.38	5.94	7.70	17280	0.3160	0.0003
transport.wind_from_comps	5.20	6.61	7.61	1440	0.3515	0.0046
transport.wind_precomp_final	0.01	0.02	0.02	8640	0.0009	0.0000
dynamics.phys_predictors	55.55	60.61	64.65	1440	3.2244	0.0421
fast_physics	398.52	403.16	406.68	1440	21.4477	0.2800
convection.gr	59.62	117.65	238.44	1440	6.2587	0.0817
diags.conv	1.00	1.05	1.14	2160	0.0558	0.0005
cloud.pc2_conv	4.49	5.54	7.12	1440	0.2945	0.0038
diags.pc2_conv	0.04	0.05	0.08	1440	0.0027	0.0000
bl.implicit	92.24	101.65	117.09	2880	5.4080	0.0353
diags.bl_imp	0.51	0.55	0.67	2160	0.0295	0.0003
jules.implicit	1.68	3.15	7.80	2880	0.1675	0.0011
dynamics.solver	408.73	410.50	412.56	2880	21.8382	0.1425
solver.mixed_solver	385.66	388.62	390.92	2880	20.6745	0.1349
mixed_solver.schur_precon	158.25	167.84	174.04	12242	8.9292	0.0137
schur_precon.rhs	40.68	44.85	47.67	12242	2.3859	0.0037
schur_precon.pressure_solver	78.30	83.59	90.35	12242	4.4471	0.0068
pressure_solver.helmholtz_lhs	36.75	42.20	47.75	183630	2.2451	0.0002
schur_precon.back_sub	36.13	39.17	44.36	12242	2.0836	0.0032
mixed_solver.operator	140.23	144.77	149.38	12242	7.7018	0.0118
diags.jules_imp	0.08	0.10	0.13	1440	0.0053	0.0001
jules.extra	0.14	1.57	9.61	720	0.0837	0.0022
diags.jules_snow	0.04	0.05	0.08	1440	0.0027	0.0000
diags.jules_soil	0.06	0.07	0.10	1440	0.0038	0.0000
diags.jules_seaice	0.02	0.03	0.06	720	0.0014	0.0000
cloud	23.78	26.23	29.95	720	1.3952	0.0364
diags.cloud	1.04	1.42	1.83	1440	0.0754	0.0010
cloud.pc2_pressure	2.03	2.50	3.62	720	0.1331	0.0035
diags.pc2_pressure	0.02	0.03	0.04	720	0.0013	0.0000
cloud.pc2_initiation	19.83	22.39	25.82	720	1.1914	0.0311
diags.pc2_initiation	0.03	0.04	0.05	720	0.0019	0.0000
ukca	56.14	61.81	64.15	720	3.2881	0.0858
diags.ukca	0.17	0.19	0.21	720	0.0102	0.0003
diags.dynamics	0.23	0.25	0.32	720	0.0135	0.0004
diags.pmsl	1.56	1.85	2.55	720	0.0984	0.0026
diags.pressure_lev	0.19	0.20	0.21	720	0.0104	0.0003
gungho_finalise	12.28	12.34	15.40	1	0.6565	12.3400

This branch, *2006s*

Routine	mintime(s)	meantime(s)	maxtime(s)	No.calls	%time	timepercall(s)
lfric_atm	2005.99	2006.02	2006.07	1	100.0000	2006.0218
setup	50.93	50.97	51.00	1	2.5409	50.9716
gungho_driver.initialise	50.93	50.97	51.00	1	2.5409	50.9704
fs.constructor	0.04	0.04	0.06	71	0.0021	0.0006
field.initialise	3.27	3.79	4.57	1487993	0.1891	0.0000
lfric_xios.init_context	5.05	5.40	5.66	1	0.2693	5.4016
field.halo_ex_1	103.44	183.94	324.35	344034	9.1694	0.0005
halo_routing_creation	22.75	28.72	30.78	421277	1.4318	0.0001
xios.close_context_definition	4.64	4.67	4.72	1	0.2328	4.6692
lfric_xios.finalise_context	11.56	11.58	11.63	2	0.5772	5.7896
field.read	15.81	15.94	16.01	66	0.7944	0.2415
lfric_xios.read_fldg	15.81	15.94	16.01	66	0.7944	0.2415
field.halo_ex_9	0.00	0.00	0.01	7	0.0002	0.0006
runtime_constants.geometric	0.48	0.61	0.82	23	0.0302	0.0263
mappings.set_wind	23.21	65.67	183.61	2881	3.2738	0.0228
runtime_constants.fem	1.65	6.85	7.93	31	0.3413	0.2208
runtime_constants.mapping	0.15	0.16	0.23	1	0.0078	0.1566
runtime_constants.dycore	19.27	20.37	26.74	13	1.0154	1.5668
mass_matrix_solver_alg	62.62	67.03	71.06	3637	3.3416	0.0184
mappings.map_physics_fields	50.28	54.21	57.09	721	2.7022	0.0752
xios.update_calendar	1.81	1.89	2.03	722	0.0940	0.0026
gungho_diagnostics_driver	13.35	14.60	15.35	721	0.7278	0.0203
field.write	3.40	3.77	4.28	274566	0.1881	0.0000
lfric_xios.write_fldg	2.29	2.60	3.06	12173	0.1297	0.0002
gungho_driver.first_timestep	41.37	41.39	41.42	1	2.0633	41.3905
gungho_timestep	1928.07	1928.72	1929.56	720	96.1467	2.6788
semi_implicit_timestep	1928.02	1928.67	1929.52	720	96.1442	2.6787
dynamics.compute_si_operators	26.30	27.73	28.54	48	1.3822	0.5776
runtime_constants.solver	0.02	0.03	0.03	17	0.0013	0.0016
field.halo_ex_2	0.81	1.63	4.73	3792	0.0812	0.0004
slow_physics	351.60	353.34	356.38	720	17.6138	0.4907
cloud.fsd_condensate	2.09	2.79	3.58	720	0.1393	0.0039
locate_tropopause	0.09	0.13	0.18	720	0.0065	0.0002
diags.locate_tropopause	0.01	0.01	0.01	720	0.0005	0.0000
aerosol.glomap_clim	19.48	20.47	28.00	720	1.0206	0.0284
diags.glomap_clim	0.10	0.13	0.21	720	0.0063	0.0002
aerosol.stratosphere	0.03	0.09	0.25	720	0.0044	0.0001
microphysics.wb	28.33	41.12	61.66	720	2.0496	0.0571
diags.mphys	0.30	0.39	0.55	1440	0.0192	0.0003
radiation.illuminate	0.59	0.70	1.06	720	0.0347	0.0010
diags.illuminate	0.02	0.02	0.03	720	0.0012	0.0000
aerosol.radaer	46.06	61.63	78.66	48	3.0721	1.2839
diags.radaer	0.03	0.04	0.10	144	0.0021	0.0003
jules.radiation	0.45	0.79	1.18	720	0.0394	0.0011
diags.radiation	0.94	1.02	1.16	1440	0.0510	0.0007
radiation.sw	2.80	11.30	25.10	720	0.5634	0.0157
radiation.lw	11.88	22.84	49.25	720	1.1384	0.0317
cloud.pc2_radiation	1.97	2.55	3.64	720	0.1271	0.0035
diags.pc2_radiation	0.02	0.02	0.04	720	0.0012	0.0000
spectral_gwd	5.00	6.58	9.23	720	0.3278	0.0091
diags.spectral_gwd	0.06	0.07	0.09	1440	0.0036	0.0001
orographic_drag	0.39	2.99	11.88	720	0.1492	0.0042
diags.orog_gwd	0.07	0.08	0.12	1440	0.0042	0.0001
methane_oxidation	0.03	0.04	0.07	720	0.0019	0.0001
diags.methox	0.00	0.00	0.01	720	0.0002	0.0000
jules.explicit	7.15	35.16	80.83	720	1.7527	0.0488
diags.jules_exp	0.10	0.12	0.17	1440	0.0061	0.0001
bl.explicit	27.59	55.96	98.31	1440	2.7897	0.0389
runtime_constants.physics	0.00	0.02	0.08	1	0.0009	0.0190
diags.bl_exp	0.10	0.12	0.20	1440	0.0058	0.0001
cloud.bm_tau	4.65	6.07	7.52	720	0.3028	0.0084
cloud.pc2_checks	2.17	2.54	3.46	720	0.1265	0.0035
diags.pc2_checks	0.02	0.03	0.05	720	0.0015	0.0000
dynamics.rhs_alg	69.61	72.69	77.24	3600	3.6235	0.0202
dynamics.transport_predictors	20.05	21.99	23.53	720	1.0960	0.0305
dynamics.transport	259.60	262.97	268.07	1440	13.1092	0.1826
transport.controller_init	14.58	16.43	19.03	1440	0.8191	0.0114
runtime_constants.transport	0.01	0.01	0.02	6	0.0007	0.0022
transport.flux_precomp_init	2.42	2.58	2.76	11520	0.1287	0.0002
transport.flux_precomp	15.91	18.08	20.89	33840	0.9012	0.0005
transport.hori_dep_pts	7.76	9.72	12.23	4320	0.4844	0.0022
field.halo_ex_7	1.18	1.40	1.86	1440	0.0698	0.0010
transport.ffsl_vertical	50.27	53.57	57.99	44640	2.6702	0.0012
transport.ffsl_horizontal	111.88	117.08	127.06	13680	5.8364	0.0086
field.halo_ex_5	12.69	26.21	33.54	41044	1.3066	0.0006
field.halo_ex_3	4.31	6.13	8.49	13680	0.3056	0.0004
transport.wind_to_comps	3.40	3.62	4.07	1440	0.1806	0.0025
transport.sl_horizontal	26.92	28.84	31.37	8640	1.4377	0.0033
field.halo_ex_4	4.66	6.17	7.82	17280	0.3076	0.0004
transport.wind_from_comps	5.12	6.58	7.58	1440	0.3281	0.0046
transport.wind_precomp_final	0.01	0.02	0.02	8640	0.0008	0.0000
dynamics.phys_predictors	56.72	60.92	69.40	1440	3.0369	0.0423
fast_physics	550.59	557.02	561.45	1440	27.7674	0.3868
convection.gr	180.54	288.85	421.35	1440	14.3991	0.2006
diags.conv	1.04	1.20	1.48	2160	0.0600	0.0006
cloud.pc2_conv	5.13	6.98	12.10	1440	0.3481	0.0048
diags.pc2_conv	0.05	0.07	0.13	1440	0.0037	0.0001
bl.implicit	91.48	104.94	132.35	2880	5.2311	0.0364
diags.bl_imp	0.52	0.61	0.77	2160	0.0306	0.0003
jules.implicit	1.55	3.38	10.47	2880	0.1687	0.0012
dynamics.solver	406.78	408.44	410.38	2880	20.3605	0.1418
solver.mixed_solver	383.78	386.61	388.94	2880	19.2723	0.1342
mixed_solver.schur_precon	161.08	167.88	173.38	12242	8.3688	0.0137
schur_precon.rhs	40.99	45.02	47.17	12242	2.2444	0.0037
schur_precon.pressure_solver	78.24	83.48	89.70	12242	4.1617	0.0068
pressure_solver.helmholtz_lhs	37.08	42.13	47.43	183630	2.1002	0.0002
schur_precon.back_sub	36.25	39.14	44.08	12242	1.9509	0.0032
mixed_solver.operator	141.03	144.64	148.03	12242	7.2102	0.0118
diags.jules_imp	0.08	0.10	0.15	1440	0.0050	0.0001
jules.extra	0.13	1.68	9.47	720	0.0837	0.0023
diags.jules_snow	0.04	0.05	0.08	1440	0.0026	0.0000
diags.jules_soil	0.06	0.07	0.09	1440	0.0036	0.0001
diags.jules_seaice	0.02	0.03	0.06	720	0.0013	0.0000
cloud	23.36	26.09	30.37	720	1.3008	0.0362
diags.cloud	1.09	1.45	1.87	1440	0.0723	0.0010
cloud.pc2_pressure	2.04	2.51	3.54	720	0.1252	0.0035
diags.pc2_pressure	0.02	0.03	0.04	720	0.0013	0.0000
cloud.pc2_initiation	20.09	22.45	26.32	720	1.1191	0.0312
diags.pc2_initiation	0.03	0.04	0.05	720	0.0018	0.0001
ukca	48.46	50.71	53.39	720	2.5277	0.0704
ukca.chemistry_alg	48.01	50.15	52.77	720	2.4999	0.0697
diags.ukca	0.17	0.19	0.22	720	0.0094	0.0003
diags.dynamics	0.22	0.26	0.34	720	0.0130	0.0004
diags.pmsl	4.09	4.60	4.98	720	0.2292	0.0064
diags.pressure_lev	0.19	0.19	0.21	720	0.0097	0.0003
gungho_driver.timestep	1900.02	1900.15	1900.23	719	94.7221	2.6428
gungho_driver.finalise	11.60	11.61	11.66	1	0.5788	11.6106
xios.context_finalize	8.57	8.58	8.59	1	0.4279	8.5833

This branch, with configure segments 16 *1807s*

Routine	mintime(s)	meantime(s)	maxtime(s)	No.calls	%time	timepercall(s)
lfric_atm	1807.23	1807.27	1808.22	1	100.0000	1807.2722
setup	54.08	54.11	54.14	1	2.9939	54.1088
gungho_driver.initialise	54.08	54.11	54.13	1	2.9939	54.1078
fs.constructor	0.04	0.04	0.05	71	0.0023	0.0006
field.initialise	3.27	3.77	4.54	1487945	0.2087	0.0000
lfric_xios.init_context	7.33	7.72	8.02	1	0.4272	7.7203
field.halo_ex_1	97.09	164.83	242.69	344034	9.1204	0.0005
halo_routing_creation	23.17	24.93	26.49	421277	1.3792	0.0001
xios.close_context_definition	6.81	6.84	6.88	1	0.3783	6.8375
lfric_xios.finalise_context	12.38	12.40	13.34	2	0.6861	6.1996
field.read	16.75	16.86	16.94	66	0.9330	0.2555
lfric_xios.read_fldg	16.75	16.86	16.94	66	0.9330	0.2555
field.halo_ex_9	0.00	0.00	0.01	7	0.0002	0.0006
runtime_constants.geometric	0.48	0.53	0.69	23	0.0293	0.0230
mappings.set_wind	22.23	48.20	97.40	2881	2.6672	0.0167
runtime_constants.fem	1.66	3.21	4.20	31	0.1777	0.1036
runtime_constants.mapping	0.15	0.16	0.18	1	0.0087	0.1567
runtime_constants.dycore	19.22	20.37	23.18	13	1.1269	1.5667
mass_matrix_solver_alg	62.31	66.99	71.20	3637	3.7068	0.0184
mappings.map_physics_fields	51.38	55.35	58.64	721	3.0624	0.0768
xios.update_calendar	0.36	1.63	2.03	722	0.0904	0.0023
gungho_diagnostics_driver	14.07	15.23	16.00	721	0.8428	0.0211
field.write	3.34	3.68	4.13	274566	0.2037	0.0000
lfric_xios.write_fldg	2.25	2.54	2.97	12173	0.1404	0.0002
gungho_driver.first_timestep	37.19	37.22	37.25	1	2.0593	37.2173
gungho_timestep	1724.67	1725.66	1727.49	720	95.4840	2.3967
semi_implicit_timestep	1724.63	1725.60	1727.44	720	95.4812	2.3967
dynamics.compute_si_operators	22.46	23.94	24.83	48	1.3247	0.4988
runtime_constants.solver	0.02	0.03	0.03	17	0.0014	0.0015
field.halo_ex_2	0.86	1.67	4.67	3792	0.0926	0.0004
slow_physics	351.32	353.22	357.10	720	19.5442	0.4906
cloud.fsd_condensate	1.97	2.72	7.41	720	0.1507	0.0038
locate_tropopause	0.09	0.13	0.18	720	0.0072	0.0002
diags.locate_tropopause	0.01	0.01	0.01	720	0.0005	0.0000
aerosol.glomap_clim	19.50	20.47	23.15	720	1.1325	0.0284
diags.glomap_clim	0.09	0.13	0.22	720	0.0070	0.0002
aerosol.stratosphere	0.03	0.09	0.27	720	0.0048	0.0001
microphysics.wb	27.84	40.90	61.60	720	2.2629	0.0568
diags.mphys	0.31	0.39	0.59	1440	0.0213	0.0003
radiation.illuminate	0.59	0.69	0.99	720	0.0385	0.0010
diags.illuminate	0.02	0.02	0.03	720	0.0013	0.0000
aerosol.radaer	46.29	61.64	80.02	48	3.4108	1.2842
diags.radaer	0.03	0.04	0.10	144	0.0023	0.0003
jules.radiation	0.40	0.78	1.39	720	0.0432	0.0011
diags.radiation	0.94	1.02	1.16	1440	0.0565	0.0007
radiation.sw	2.81	11.28	25.22	720	0.6243	0.0157
radiation.lw	11.99	22.80	49.27	720	1.2618	0.0317
cloud.pc2_radiation	1.96	2.53	3.52	720	0.1403	0.0035
diags.pc2_radiation	0.02	0.02	0.04	720	0.0014	0.0000
spectral_gwd	4.91	6.57	9.34	720	0.3634	0.0091
diags.spectral_gwd	0.06	0.07	0.09	1440	0.0040	0.0001
orographic_drag	0.39	2.99	11.92	720	0.1657	0.0042
diags.orog_gwd	0.07	0.08	0.12	1440	0.0046	0.0001
methane_oxidation	0.03	0.04	0.07	720	0.0021	0.0001
diags.methox	0.00	0.00	0.01	720	0.0003	0.0000
jules.explicit	6.91	35.42	82.13	720	1.9601	0.0492
diags.jules_exp	0.10	0.12	0.18	1440	0.0068	0.0001
bl.explicit	27.24	55.98	98.21	1440	3.0975	0.0389
runtime_constants.physics	0.00	0.02	0.09	1	0.0011	0.0194
diags.bl_exp	0.09	0.12	0.19	1440	0.0064	0.0001
cloud.bm_tau	4.71	6.09	7.49	720	0.3370	0.0085
cloud.pc2_checks	2.17	2.53	3.30	720	0.1398	0.0035
diags.pc2_checks	0.02	0.03	0.05	720	0.0017	0.0000
dynamics.rhs_alg	69.77	72.70	77.28	3600	4.0229	0.0202
dynamics.transport_predictors	19.94	21.94	23.58	720	1.2138	0.0305
dynamics.transport	261.52	265.17	270.39	1440	14.6724	0.1841
transport.controller_init	14.58	16.52	19.19	1440	0.9139	0.0115
runtime_constants.transport	0.01	0.01	0.03	6	0.0007	0.0022
transport.flux_precomp_init	2.43	2.58	2.78	11520	0.1426	0.0002
transport.flux_precomp	16.11	18.10	21.07	33840	1.0015	0.0005
transport.hori_dep_pts	7.82	9.76	12.66	4320	0.5403	0.0023
field.halo_ex_7	1.16	1.41	1.86	1440	0.0780	0.0010
transport.ffsl_vertical	50.19	53.47	58.74	44640	2.9586	0.0012
transport.ffsl_horizontal	110.82	117.41	128.19	13680	6.4964	0.0086
field.halo_ex_5	12.06	26.36	35.20	41044	1.4586	0.0006
field.halo_ex_3	4.15	6.14	8.39	13680	0.3398	0.0004
transport.wind_to_comps	3.38	3.62	4.04	1440	0.2004	0.0025
transport.sl_horizontal	27.22	28.95	31.47	8640	1.6020	0.0034
field.halo_ex_4	4.68	6.29	7.91	17280	0.3481	0.0004
transport.wind_from_comps	5.34	6.61	7.83	1440	0.3659	0.0046
transport.wind_precomp_final	0.01	0.02	0.02	8640	0.0009	0.0000
dynamics.phys_predictors	56.50	60.87	63.96	1440	3.3678	0.0423
fast_physics	348.16	353.99	358.32	1440	19.5872	0.2458
convection.gr	115.66	158.39	210.06	1440	8.7642	0.1100
diags.conv	1.04	1.12	1.33	2160	0.0619	0.0005
cloud.pc2_conv	5.17	6.51	10.98	1440	0.3603	0.0045
diags.pc2_conv	0.05	0.06	0.09	1440	0.0035	0.0000
bl.implicit	98.22	106.30	120.55	2880	5.8819	0.0369
diags.bl_imp	0.53	0.59	0.71	2160	0.0328	0.0003
jules.implicit	1.90	3.45	9.35	2880	0.1907	0.0012
dynamics.solver	404.12	408.73	411.13	2880	22.6160	0.1419
solver.mixed_solver	381.07	386.94	389.52	2880	21.4103	0.1344
mixed_solver.schur_precon	160.23	167.78	173.26	12242	9.2834	0.0137
schur_precon.rhs	40.84	44.89	47.19	12242	2.4840	0.0037
schur_precon.pressure_solver	78.74	83.54	89.94	12242	4.6223	0.0068
pressure_solver.helmholtz_lhs	37.67	42.15	47.47	183630	2.3324	0.0002
schur_precon.back_sub	36.16	39.11	43.97	12242	2.1641	0.0032
mixed_solver.operator	140.32	144.45	148.22	12242	7.9926	0.0118
diags.jules_imp	0.09	0.10	0.15	1440	0.0056	0.0001
jules.extra	0.14	1.72	9.66	720	0.0952	0.0024
diags.jules_snow	0.04	0.05	0.08	1440	0.0029	0.0000
diags.jules_soil	0.06	0.07	0.10	1440	0.0041	0.0001
diags.jules_seaice	0.02	0.03	0.07	720	0.0015	0.0000
cloud	22.82	26.09	30.29	720	1.4434	0.0362
diags.cloud	1.07	1.45	1.90	1440	0.0802	0.0010
cloud.pc2_pressure	2.04	2.51	3.67	720	0.1389	0.0035
diags.pc2_pressure	0.02	0.03	0.04	720	0.0014	0.0000
cloud.pc2_initiation	19.61	22.44	26.13	720	1.2418	0.0312
diags.pc2_initiation	0.03	0.04	0.05	720	0.0020	0.0001
ukca	48.44	50.68	53.31	720	2.8041	0.0704
ukca.chemistry_alg	47.95	50.12	52.74	720	2.7732	0.0696
diags.ukca	0.17	0.19	0.21	720	0.0105	0.0003
diags.dynamics	0.23	0.26	0.34	720	0.0144	0.0004
diags.pmsl	4.70	5.21	5.59	720	0.2885	0.0072
diags.pressure_lev	0.18	0.19	0.21	720	0.0107	0.0003
gungho_driver.timestep	1701.47	1701.86	1703.14	719	94.1675	2.3670
gungho_driver.finalise	12.41	12.43	13.37	1	0.6879	12.4315
xios.context_finalize	9.38	9.39	9.39	1	0.5196	9.3906

Code Quality Checklist

• I have performed a self-review of my own code
• My code follows the project's style guidelines
• Comments have been included that aid understanding and enhance the readability of the code
• My changes generate no new warnings
• All automated checks in the CI pipeline have completed successfully

Testing

• I have tested this change locally, using the LFRic Apps rose-stem suite
• If any tests fail (rose-stem or CI) the reason is understood and acceptable (e.g. kgo changes)
• I have added tests to cover new functionality as appropriate (e.g. system tests, unit tests, etc.)
• Any new tests have been assigned an appropriate amount of compute resource and have been allocated to an appropriate testing group (i.e. the developer tests are for jobs which use a small amount of compute resource and complete in a matter of minutes)

trac.log

Security Considerations

• I have reviewed my changes for potential security issues
• Sensitive data is properly handled (if applicable)
• Authentication and authorisation are properly implemented (if applicable)

Performance Impact

• Performance of the code has been considered and, if applicable, suitable performance measurements have been conducted

AI Assistance and Attribution

• Some of the content of this change has been produced with the assistance of Generative AI tool name (e.g., Met Office Github Copilot Enterprise, Github Copilot Personal, ChatGPT GPT-4, etc) and I have followed the Simulation Systems AI policy (including attribution labels)

Documentation

• Where appropriate I have updated documentation related to this change and confirmed that it builds correctly

PSyclone Approval

• If you have edited any PSyclone-related code (e.g. PSyKAl-lite, Kernel interface, optimisation scripts, LFRic data structure code) then please contact the TCD Team

Sci/Tech Review

• I understand this area of code and the changes being added
• The proposed changes correspond to the pull request description
• Documentation is sufficient (do documentation papers need updating)
• Sufficient testing has been completed

(Please alert the code reviewer via a tag when you have approved the SR)

Code Review

• All dependencies have been resolved
• Related Issues have been properly linked and addressed
• CLA compliance has been confirmed
• Code quality standards have been met
• Tests are adequate and have passed
• Documentation is complete and accurate
• Security considerations have been addressed
• Performance impact is acceptable