Streamline workflow #1838
Streamline workflow #1838
Conversation
- Add test_streamlined.smk with 4-step workflow (base → clustered → composed → solved) - Support all three foresight modes: overnight, myopic, perfect - Add robust horizon handling for single values and lists - Update test configs with proper temporal and clustering sections - Add comprehensive test configurations for all foresight modes - Add implementation plan and documentation Features: - Config-driven approach (no wildcard expansion) - Dynamic input resolution based on foresight and horizons - Sequential dependencies for myopic/perfect foresight - Unified rule structure across all modes - Robust parameter handling with defaults Tested workflows: - Overnight: 6 jobs, single horizon optimization - Myopic: 10 jobs, sequential horizon optimization with brownfield - Perfect: 8 jobs, multi-period simultaneous optimization Refs: GitHub Discussion #1529
- Add Status Update section documenting completed proof of concept - Update timeline to reflect completed Phase 0 (POC) and current focus - Document validated implementation patterns from testing experience - Add robust parameter handling and dynamic input resolution examples - Update success criteria to distinguish completed vs remaining goals - Reorganize open questions to focus on remaining implementation details - Document created test files and artifacts Key achievements validated: - 4-step workflow architecture working across all foresight modes - Config-driven approach with temporal section replacing scenario wildcards - Dynamic dependencies with proper sequential chaining - Robust error handling for missing config sections - Flexible horizon handling for single values and lists Next phase: Implement actual scripts with real PyPSA network operations
- Create scripts/compose_network.py that combines all network building steps
in one big main section without additional function definitions
- Implement network concatenation logic for perfect foresight mode
- Add brownfield constraints for myopic optimization
- Create production rules/compose.smk with dynamic dependencies
- Update Snakefile to include compose rules when streamlined_workflow is enabled
- Add temporal configuration section to default config
- Create example configuration file demonstrating the new workflow
The new workflow follows the 4-step structure:
base → clustered → composed_{horizon} → solved_{horizon}
All configuration is now driven by config sections rather than wildcards,
simplifying the workflow and making it more maintainable.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
CRITICAL FIXES:
- Fix snapshot handling bug: prevent overwriting multi-period snapshots in perfect foresight
- Fix function signature error in load_and_aggregate_powerplants call
- Remove resource path conflict by using existing path provider system
- Simplify overly complex perfect foresight concatenation logic
- Fix parameter access patterns throughout the script
- Improve error handling for missing config sections
RESOURCE PATH UPDATES:
- Update all resource references to use existing path patterns with {clusters} wildcards
- Remove custom resources() function that conflicted with existing system
- Align file paths with existing workflow conventions
ROBUSTNESS IMPROVEMENTS:
- Add proper error handling for missing inputs
- Simplify carbon budget handling to avoid complex logic failures
- Use safer parameter access patterns with fallbacks
These fixes address the most serious issues that would have prevented
the workflow from functioning correctly.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Fix critical wildcard mismatch: resource paths used {clusters} but rules didn't have this wildcard
- Get cluster count from config and substitute into resource paths
- Add missing cluster_network rule to generate clustered.nc input
- Fix all resource path references to use config-derived cluster values
- Ensure proper dependency chain: base → clustered → composed → solved
This resolves the fundamental mismatch between wildcard-based resource
paths and the new config-driven workflow approach.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
STANDARDIZATION IMPROVEMENTS: - Replace all direct config access with config_provider function calls - Ensure consistent use of resources() function from existing path provider - Follow PyPSA-EUR conventions for parameter handling in Snakemake rules - Use proper wildcard handling patterns from existing codebase KEY CHANGES: - get_compose_inputs: Use config_provider for all config value retrieval - cluster_network rule: Convert all params to use config_provider - compose_network rule: Standardize all parameter definitions - solve_network rule: Use config_provider for solver configuration - Collection rules: Use config_provider for run name retrieval - Validation rule: Standardize parameter access patterns This ensures the streamlined workflow follows the same patterns and conventions as the existing PyPSA-EUR workflow implementation. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
|
Thanks for starting this. Had only a short look so far. I guess it is too early for reviewing. How do you think we can best support?! I have opinions about many of the "critical open questions". |
|
You are totally right, it is too early for reviewing. I guess at this stage, it is best to think about whether I can proceed with this approach or if there is potential red flags, which speak against this strategy in the first place. But perhaps I spend more time on refining this and make a proper report for you guys what are this wider implications. |
I must admit that the section |
…y network creation
…ed workflow - Rename set_line_nom_max to cap_transmission_capacity with clearer parameter names - s_nom_max_set → line_max (AC line capacity limit) - p_nom_max_set → link_max (DC link capacity limit) - s_nom_max_ext → line_max_extension (AC line extension limit) - p_nom_max_ext → link_max_extension (DC link extension limit) - Port cap_transmission_capacity to compose_network.py for network composition - Port enforce_autarky to solve_network.py for pre-solve constraints - Add cap_transmission_capacity config section to config.default.yaml - Remove .get() calls in favor of strict dictionary access for better error messages - All functionality now config-based instead of wildcard-based 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- Add adjust_renewable_capacity_limits() function to compose_network.py for brownfield scenarios in myopic foresight mode - Function subtracts existing renewable capacities from previous horizons from p_nom_max values in current horizon - Uses renewable_carriers list from config (not hardcoded) - Remove redundant add_land_use_constraint() from solve_network.py for myopic mode (functionality now in compose step) - Keep add_land_use_constraint() for perfect foresight (different logic) - Add comprehensive test suite with 8 test cases covering: * Basic capacity subtraction * Multiple existing generators * Cases where existing exceeds potential * Negative value clipping * All renewable carriers * Edge cases (no existing, non-renewables) * Custom carrier lists - Update compose.smk to pass renewable_carriers as param 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
…ctor - Replace separate resolution_elec and resolution_sector settings with single clustering.temporal.resolution key - Update compose_network.py and _helpers.py to use unified key - Update all test configs and documentation
- Remove electricity.co2limit_enable, co2limit, co2base config options - Remove Co2L opts wildcard parsing from _helpers.py - Migrate test configs to use co2_budget with upper/lower bounds - Update release notes with comprehensive PR #1838 breaking changes - Update documentation (electricity.csv, opts.csv)
- Delete opts.csv and sector-opts.csv (wildcards now config-driven) - Remove Co2L and cb* wildcard parsing from sector_opts in _helpers.py - Update release notes to document cb* wildcard removal
- Fix wildcard name mismatch in retrieve_cost_data rule (planning_horizons -> horizon) - Update Snakefile to use dataset_version() for cutout directory - Merge completed for new data versioning system
- Remove deprecated co2limit parameters, use unified co2_budget - Fix memory parameter naming (mem -> mem_mb) - Correct heat_vent structure for sector-coupled configs - Move plotting params to proper section, add missing planning_horizons - Remove unsupported shared_cutouts parameter
fneum
left a comment
fneum
left a comment
There was a problem hiding this comment.
For the future, we really need to break such rearrangements into multiple smaller PRs. This took a very long time to review, and would have been easier if it were broken down into stages.
| # Set cutout directory after dataset_version function is available | ||
| cutout_dir = dataset_version("cutout")["folder"] | ||
| shared_cutouts = run.get("shared_cutouts", False) | ||
| CDIR = Path(cutout_dir).joinpath("" if shared_cutouts else RDIR) |
| planning_horizons: | ||
| - 2050 | ||
| # docs in https://pypsa-eur.readthedocs.io/en/latest/configuration.html#planning-horizons | ||
| planning_horizons: 2050 # Single year or list [2030, 2040, 2050] |
There was a problem hiding this comment.
| planning_horizons: 2050 # Single year or list [2030, 2040, 2050] | |
| planning_horizons: 2050 |
All documentation should be in the corresponding .csv files.
config/config.default.yaml
Outdated
| # Interpretation mode for per-period values (applies to BOTH upper and lower bounds) | ||
| # "fraction": Values are fractions of 1990 baseline (e.g., 0.450 = 45% of 1990) | ||
| # "absolute": Values are Gt CO2/year (e.g., 2.04 Gt CO2/year) | ||
| # Note: When enable: true, only years explicitly specified are constrained. | ||
| # Omitted years have no constraint for that bound type. |
There was a problem hiding this comment.
| # Interpretation mode for per-period values (applies to BOTH upper and lower bounds) | |
| # "fraction": Values are fractions of 1990 baseline (e.g., 0.450 = 45% of 1990) | |
| # "absolute": Values are Gt CO2/year (e.g., 2.04 Gt CO2/year) | |
| # Note: When enable: true, only years explicitly specified are constrained. | |
| # Omitted years have no constraint for that bound type. |
Same here. No docs in YAML.
| 2040: 0.100 # 90% by 2040 | ||
| 2045: 0.050 | ||
| 2050: 0.000 # climate-neutral by 2050 | ||
| emissions_scope: All greenhouse gases - (CO2 equivalent) |
There was a problem hiding this comment.
Is a different emissions scope than pure CO2 actually implemented?
| # "absolute": Values are Gt CO2/year (e.g., 2.04 Gt CO2/year) | ||
| # Note: When enable: true, only years explicitly specified are constrained. | ||
| # Omitted years have no constraint for that bound type. | ||
| values: fraction |
There was a problem hiding this comment.
| values: fraction | |
| values: fraction |
as discussed: could be relative: {true/false}
| The main script portion (concat_networks, if __name__ == "__main__") is DEPRECATED | ||
| and not used in the streamlined workflow, as networks are now composed incrementally | ||
| rather than batch-concatenated. |
There was a problem hiding this comment.
The main block can then be removed as it is not usable anymore.
| This module provides functions for perfect foresight optimization in the | ||
| streamlined workflow. Functions are used by compose_network.py for incremental | ||
| network composition across planning horizons. |
There was a problem hiding this comment.
Refer to rules rather than scripts.
| .. tip:: | ||
| The rule :mod:`prepare_elec_networks` runs | ||
| for all ``scenario`` s in the configuration file | ||
| the rule :mod:`prepare_network`. | ||
| The deprecated rule :mod:`prepare_elec_networks` has been replaced by the | ||
| unified :mod:`compose_network` script which handles network preparation for | ||
| all scenarios. |
| The rule set follows ``base → simplified → clustered → composed | ||
| → solved``. Intermediate networks (`networks/simplified.nc`, | ||
| `networks/clustered.nc`, `networks/composed_{horizon}.nc`) live under | ||
| ``resources/{run}``, while solved artefacts are always written to | ||
| ``results/{run}/networks/solved_{horizon}.nc``. For users upgrading from the | ||
| legacy ``base_s`` naming scheme, see the dedicated :doc:`migration` guide. |
There was a problem hiding this comment.
This is not really understandable for first-time users; perhaps rephrase in a more accessible way for the landing page?
| introduction | ||
| installation | ||
| tutorial | ||
| migration |
There was a problem hiding this comment.
I don't think this is the right place for the migration guide; it should go near the release notes.
This comment was marked as duplicate.
This comment was marked as duplicate.
Sorry, something went wrong.
| Upcoming Release | ||
| ================ | ||
|
|
||
| * Streamlined workflow with simplified configuration (https://github.com/PyPSA/pypsa-eur/pull/1838). See ``doc/migration.rst`` for detailed migration guidance. |
There was a problem hiding this comment.
Reference to migration guide should be a link not file path.
|
|
||
| **Workflow structure (related to streamlining):** | ||
|
|
||
| - The network pipeline now follows a clear 4-stage progression: ``base.nc`` → ``simplified.nc`` → ``clustered.nc`` → ``composed_{horizon}.nc`` → ``solved_{horizon}.nc``. |
There was a problem hiding this comment.
| - The network pipeline now follows a clear 4-stage progression: ``base.nc`` → ``simplified.nc`` → ``clustered.nc`` → ``composed_{horizon}.nc`` → ``solved_{horizon}.nc``. | |
| - The network pipeline now follows a 4-stage progression: ``base.nc`` → ``simplified.nc`` → ``clustered.nc`` → ``composed_{horizon}.nc`` → ``solved_{horizon}.nc``. |
| * Fix parsing in Swiss passenger cars data (https://github.com/PyPSA/pypsa-eur/pull/1934 and https://github.com/PyPSA/pypsa-eur/pull/1936). | ||
|
|
||
| * Fix: ValueError with `cop_heat_pump` in `prepare_sector_network.py` if `tim_dep_hp_cop` is `false`. | ||
|
|
There was a problem hiding this comment.
The removal of lines bloats the diff.
| Upcoming Release | ||
| ================ | ||
|
|
||
| * Streamlined workflow with simplified configuration (https://github.com/PyPSA/pypsa-eur/pull/1838). See ``doc/migration.rst`` for detailed migration guidance. |
There was a problem hiding this comment.
| * Streamlined workflow with simplified configuration (https://github.com/PyPSA/pypsa-eur/pull/1838). See ``doc/migration.rst`` for detailed migration guidance. | |
| * Streamlined workflow (https://github.com/PyPSA/pypsa-eur/pull/1838). See ``doc/migration.rst`` for detailed migration guidance. |
The configuration was not really simplified.
| **Workflow structure (related to streamlining):** | ||
|
|
||
| - The network pipeline now follows a clear 4-stage progression: ``base.nc`` → ``simplified.nc`` → ``clustered.nc`` → ``composed_{horizon}.nc`` → ``solved_{horizon}.nc``. | ||
| - Cryptic filenames like ``elec_s_37_lv1.25_3H_2030.nc`` are replaced with readable names. Scenario parameters (clusters, opts, sector_opts) are now set via configuration rather than filename wildcards. |
There was a problem hiding this comment.
Try more neutral formulations. These filenames were not cryptic, but followed wildcard naming schemes.
Scenario parameters is the wrong word. These were wildcards.
| - **Overnight**: Require a single horizon; ``compose_network`` never looks for | ||
| previous outputs. | ||
| - **Myopic**: ``compose_network`` imports | ||
| ``RESULTS/networks/solved_{previous}.nc`` via the helper |
There was a problem hiding this comment.
| ``RESULTS/networks/solved_{previous}.nc`` via the helper | |
| ``results/{run}/networks/solved_{previous}.nc`` via the helper |
| - **Myopic**: ``compose_network`` imports | ||
| ``RESULTS/networks/solved_{previous}.nc`` via the helper | ||
| ``solved_previous_horizon``. Ensure horizons are sorted ascendingly. | ||
| - **Perfect**: ``compose_network`` uses ``networks/composed_{previous}.nc`` as |
There was a problem hiding this comment.
| - **Perfect**: ``compose_network`` uses ``networks/composed_{previous}.nc`` as | |
| - **Perfect**: ``compose_network`` uses ``resources/{run}/networks/composed_{previous}.nc`` as |
| Release checklist | ||
| ================= | ||
|
|
||
| - Update any bespoke scripts or notebooks to read | ||
| ``results/{run}/networks/solved_{horizon}.nc`` and the new CSV names from | ||
| :mod:`scripts/make_summary`. | ||
| - Configure clustering counts, electricity options, sector options, and | ||
| planning horizons directly in their respective config sections. | ||
| - Verify documentation builds via ``make -C doc html`` and search for outdated | ||
| targets with ``rg -n "base_s" doc build/html``. Older workflows exposed these | ||
| controls via the ``{clusters}``, ``{opts}``, ``{sector_opts}``, and | ||
| ``{planning_horizons}`` wildcards, so scrub those placeholders when migrating | ||
| custom rule code. |
There was a problem hiding this comment.
Double-check that this is complete in final version.
| n.add( | ||
| "GlobalConstraint", | ||
| "CO2Min" + suffix, | ||
| type="co2_limit", |
There was a problem hiding this comment.
This type does not exist in PyPSA and is not handled in solve_network (like co2_atmosphere). Won't it have no effect then (with unknown type)?
| """ | ||
| Compose network by combining all electricity and sector components. | ||
| This script combines functionality from add_electricity.py, prepare_sector_network.py, |
There was a problem hiding this comment.
refer to rules rather than scripts
| else: # overnight | ||
| # Should not reach here for overnight with multiple horizons | ||
| n = pypsa.Network(snakemake.input.clustered) |
There was a problem hiding this comment.
This case should throw an error (means multiple planning horizons for overnight run config).
| return n | ||
|
|
||
|
|
||
| def validate_network_state( |
There was a problem hiding this comment.
Hmm, this function doesn't really do much. Can we remove for now?
| if sector_mode: | ||
| # Remove generators with undefined buses | ||
| valid_buses = set(n.buses.index) | ||
| for c in n.iterate_components(n.one_port_components | n.branch_components): | ||
| if "bus" in c.df.columns: | ||
| invalid = ~c.df["bus"].isin(valid_buses) | ||
| if invalid.any(): | ||
| logger.warning( | ||
| f"Removing {invalid.sum()} {c.name} with undefined buses: {c.df.index[invalid].tolist()}" | ||
| ) | ||
| n.remove(c.name, c.df.index[invalid]) | ||
| if "bus0" in c.df.columns: | ||
| invalid0 = ~c.df["bus0"].isin(valid_buses) | ||
| invalid1 = ~c.df["bus1"].isin(valid_buses) | ||
| invalid = invalid0 | invalid1 | ||
| if invalid.any(): | ||
| logger.warning( | ||
| f"Removing {invalid.sum()} {c.name} with undefined buses: {c.df.index[invalid].tolist()}" | ||
| ) | ||
| n.remove(c.name, c.df.index[invalid]) |
There was a problem hiding this comment.
Why is this necessary? If this happens, it seems to indicate a modelling error, which I wouldn't want to have silently handled.
At least wrap in a function.
| ) | ||
| n.remove(c.name, c.df.index[invalid]) | ||
|
|
||
| # ========== SECTOR COMPONENTS (from prepare_sector_network.py) ========== |
There was a problem hiding this comment.
Are you sure you can swap order of prepare_sector_network and prepare_network?
| if "marginal_cost" in n.generators_t: | ||
| n.generators_t["marginal_cost"] = n.generators_t["marginal_cost"].fillna( | ||
| n.generators.marginal_cost | ||
| ) |
| ) | ||
| # Force synchronization by setting investment_periods to match snapshots | ||
| # Use set_investment_periods() to ensure proper validation and storage | ||
| n.set_investment_periods(snapshot_periods) |
There was a problem hiding this comment.
Is this safe? Or should it rather terminate in an error?
| return counts | ||
|
|
||
|
|
||
| def adjust_renewable_capacity_limits( |
There was a problem hiding this comment.
Shouldn't this be in add_brownfield.py?
| n.generators["p_nom_max"] = n.generators["p_nom_max"].clip(lower=0) | ||
|
|
||
|
|
||
| def adjust_biomass_availability(n: pypsa.Network) -> None: |
There was a problem hiding this comment.
This function should be removed! It's an error in configuration if this happens! It should not automatically be rescaled!
| # Cluster heat buses if enabled | ||
| if params.sector["cluster_heat_buses"]: | ||
| cluster_heat_buses(n) | ||
|
|
There was a problem hiding this comment.
Isn't the function maybe_adjust_costs_and_potentials missing? It's not called at all...
|
Overall, the direction is great, and I concur with the structural changes. As you see from the large amount of comments, there's a lot around the details that needs to be refined. Owing to the amount of changes (and some squeezed in changes beyond mere restructuring), we need to be really certain that this PR reproduces the same results as the current master for a range of different configuration files that exceed the reduced CI tests (e.g. full resolution models in different foresight modes). In particular, I would like to see different emission limits tested. |
|
thank you for the detailed review @fneum; I know the PR is laaarge and I would like have to avoid it. however, since a lot of changes are interlinked and it was not clear how things work out, it would have taken quite some effort to come up with a structural bit-wise approach in the first place. I will work on your comments now and try to make things as backwards compatible as possible |
|
No worries, @FabianHofmann! It worked out :) It's clear for this PR it was very difficult. We just have to be careful since problems might slip through this way. Sorry for the dump of so many review comments, but it will be worth it! |
totally agree! also happy to address the comments, it is good that a reviewer looks at the code in detail |
Closes #1916. (supersedes)
Streamlined and restructured workflow
This PR refactors the workflow in order to streamline and unify optimization approaches and configuration settings. here are some highlights of the PR, otherwise no one will have the motivation to look at this large change. I suggest before going into the code, read this here first, even though it is long...
Highlights
base.nc->clustered.nc->composed_{horizon}.nc->solved_{horizon}.nccsvs/nodal_costs.csv,csvs/capacities.csv...)composerule is the entry point for green-field, brown-field, perfect foresight model for both sector-coupled and electricity-only.and now a more descriptive summary
Overview
base → simplified → clustered → composed → solved) and standardises filenames so scenario information now lives in configuration instead of wildcards.*_s_{clusters}or..._{planning_horizons}artefacts have been renamed; the new naming scheme makes horizons explicit via{horizon}and removes cluster suffixes.Workflow Changes
networks/base.nc→networks/simplified.nc→networks/clustered.nc→networks/composed_{horizon}.nc→RESULTS/networks/solved_{horizon}.nc. Old intermediate targets such asnetworks/base_s_{clusters}_{opts}_{sector_opts}_{planning_horizons}.ncno longer exist.rules/compose.smkencapsulates what used to be multipleprepare_*,add_*, and brownfield rules; it assembles everything needed for a givenhorizonand handles the myopic/perfect brownfield inputs.rules/solve.smkcontains onesolve_networkrule; electricity-only and sector-coupled cases are distinguished by config, not by separate rule files (solve_*smks were deleted).rules/collect.smknow checks four milestones: clustered networks, composed networks, solved networks, and plotting;{horizon}replaces{planning_horizons}wildcards in file names.navigate_config,get_full_config, andget_config(all inrules/common.smk) cache fully merged configs per wildcard set. Any new rules should obtain parameters throughconfig_provider(...)orget_config(w)to stay scenario compatible.RESULTS/networks/solved_{prev}.nc, while perfect foresight reuses the priornetworks/composed_{prev}.nc. Overnight runs must supply a single planning horizon.rules/postprocess.smknow readsRESULTS/networks/solved_{horizon}.ncfor all foresight modes, so every map (power_network_{horizon}.pdf,h2_network_{horizon}.pdf,{carrier}_balance_map_{horizon}.pdf, etc.) and CSV summary is generated with the same naming scheme regardless of foresight setting.Stage Notes
rules/build_electricity.smkwritesregions_onshore_base.geojson/regions_offshore_base.geojsontogether withnetworks/base.nc. Administrative shapes and OSM/TYNDP inputs remain unchanged.simplify_networknow emitsnetworks/simplified.nc,regions_onshore_simplified.geojson,regions_offshore_simplified.geojson, andbusmap_simplified.csv.build_electricity_demand_baseconsumes those files and produceselectricity_demand_simplified.nc.process_cost_datareads per-horizoncosts_{horizon}.csv.cluster_networktakesnetworks/simplified.ncplusbusmap_simplified.csvand emitsnetworks/clustered.nc,regions_onshore.geojson,regions_offshore.geojson,busmap.csv, andlinemap.csv. Cluster counts are configured, not embedded in filenames._s_{clusters}suffix; ie. population layouts collapse topop_layout.csvandpop_layout_simplified.csv; Gas locations becomegas_input_locations.geojson/gas_input_locations_simplified.csv; Powerplant list becomespowerplants.csvetc.compose_network(one rule) flattens all previousadd_existing_baseyear,add_brownfield,prepare_perfect_*, etc. It automatically wires the previous-horizon inputs depending on foresight and provides the final pre-solve network (networks/composed_{horizon}.nc).solve_networkreadsnetworks/composed_{horizon}.nc, writesRESULTS/networks/solved_{horizon}.nc, and stores solver/memory/python logs underRESULTS/logs/solve_network/. Custom extra functionality and solver settings are driven purely by config.maps/base_network.pdfandmaps/clustered_network.pdf; solved outputs areRESULTS/maps/power_network_{horizon}.pdf,.../h2_network_{horizon}.pdf,.../ch4_network_{horizon}.pdf, and.../{carrier}_balance_map_{horizon}.pdf.make_summaryandplot_summaryoperate onRESULTS/networks/solved_{horizon}.ncregardless of foresight.Config Changes
planning_horizonsdirectly under the root of your config (config/config.default.yaml:34-35). The oldscenario.planning_horizonsentry is ignored, and the workflow expectsconfig["planning_horizons"]to exist even when scenarios are disabled. Configuration bundles such asconfig/test/config.scenarios.yaml:17-20already follow this format.scenarioblock removed – The legacyscenario:section (withclusters,opts,sector_opts, etc.) is no longer part ofconfig.default.yaml. Scenario sweeps should now be described viarun.scenariosplus the dedicated scenario YAML file; individual dimensions (e.g. clusters) are configured directly under their respective sections. If you keep ascenarioblock in a local config it will simply be ignored.co2_budgetnow specifies anemissions_scope, avaluesinterpretation flag, and nestedupper/lowerdictionaries with their ownenableswitches (config/config.default.yaml:90-113). Update custom configs accordingly if you previously listed plain year-value pairs.linesusess_nom_max/s_nom_max_extension, andlinksusesp_nom_max/p_nom_max_extension(config/config.default.yaml:320-355). Rename any overrides that still refer tomax_extensionso the new limits are applied.electricity.exclude_carriers(config/config.default.yaml:115-162) lets you strip carriers during clustering; custom busmap logic should now read that list instead of hard-coding exclusions.existing_capacities.enabled(config/config.default.yaml:479-486). Set it totruewhen you wantcompose_networkto import historical assets; otherwise the new workflow assumes a greenfield build.Script Changes
scripts/compose_network.pycombines function calls from multiple preparatory scripts (add_electricity.py,add_existing_baseyear.py,add_brownfield.py,prepare_network.py,prepare_sector_network.py,prepare_perfect_foresight.py). At a later stage a clearer packaging structure should be used here.compose_network.py.scripts/solve_network.pyis the only solver script called from Snakemake;solve_operations_network.py(operations-only runs) is no longer referenced by the workflow.scripts/make_summary.py– the script now loads all horizons viapypsa.NetworkCollectionfor overnight, myopic, and perfect runs, so the dedicated helpersscripts/make_summary_perfect.pyandscripts/make_global_summary.pywere deleted. Any custom tooling should invokemake_summary.pyand read the unified CSV outputs.File Name Mapping
networks/base_s.ncnetworks/simplified.ncregions_onshore_simplified.geojson,regions_offshore_simplified.geojson, andbusmap_simplified.csv.regions_onshore_base_s_{clusters}.geojson/regions_offshore_base_s_{clusters}.geojsonregions_onshore.geojson/regions_offshore.geojsoncluster_network; no{clusters}wildcard in filename.busmap_base_s_{clusters}.csv/linemap_base_s_{clusters}.csvbusmap.csv/linemap.csvpowerplants_s_{clusters}.csvpowerplants_s.csvnetworks/clustered.nc.electricity_demand_base_s.ncelectricity_demand_simplified.ncavailability_matrix_{clusters}_{technology}.ncavailability_matrix_{technology}.ncprofile_{clusters}_{technology}.ncprofile_{technology}.ncregions_by_class_*follow the same pattern.pop_layout_base_s_{clusters}.csv/pop_layout_base_s.csvpop_layout.csv/pop_layout_simplified.csvgas_input_locations_s_{clusters}.geojsongas_input_locations.geojsoncosts_{planning_horizons}.csv/_processed.csvcosts_{horizon}.csv/_processed.csvcollect.smkexpands over{horizon}.networks/base_s_{clusters}_{opts}_{sector_opts}_{planning_horizons}.nc(+_brownfield*)networks/composed_{horizon}.ncRESULTS/networks/base_s_{clusters}_{opts}_{sector_opts}_{planning_horizons}.ncRESULTS/networks/solved_{horizon}.ncmaps/power-network.pdf/maps/power-network-s-{clusters}.pdfmaps/base_network.pdf/maps/clustered_network.pdfRESULTS/maps/base_s_{clusters}_{opts}_{sector_opts}-costs-all_{planning_horizons}.pdfRESULTS/maps/power_network_{horizon}.pdfMigration Checklist
*_base_s*or*_base_s_{clusters}_*targets with the new filenames; consumenetworks/composed_{horizon}.nc/RESULTS/networks/solved_{horizon}.ncinstead of the legacybase_s_*artefacts.composerule incompose.smkand update to the new wildcard convention (only keep{horizon}wildcard)compose_network.pyreference the old scripts to clearly indicate where to add the changes.{horizon}-based filenames and the new CSV outputs frommake_summary.RESULTS/networks/solved_{prev}.nc; perfect foresight helpers should import the previousnetworks/composed_{prev}.nc. Do not craft filenames manually.Checklist
envs/environment.yaml.config/config.default.yaml.doc/configtables/*.csv.doc/data_sources.rst.doc/release_notes.rstis added.