Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
14272fa
chore: Standardize pipeline optimization name, dropping 'resource'
bebosudo Feb 5, 2026
a8292a7
Mention that groundswell is internal name, add pipeline optimization …
bebosudo Feb 5, 2026
8e9d8b0
Merge branch 'master' into chiusle/standardize-pipeline-optimization-…
justinegeffen Feb 5, 2026
8ca8779
Apply suggestion from @github-actions[bot]
justinegeffen Feb 5, 2026
fc81819
Apply suggestion from @github-actions[bot]
justinegeffen Feb 5, 2026
cc91bba
Apply suggestion from @justinegeffen
justinegeffen Feb 5, 2026
63b3a2f
Apply suggestion from @justinegeffen
justinegeffen Feb 5, 2026
314c480
Apply suggestion from @justinegeffen
justinegeffen Feb 5, 2026
f8de339
Apply suggestion from @github-actions[bot]
justinegeffen Feb 5, 2026
894dfa9
Apply suggestion from @justinegeffen
justinegeffen Feb 5, 2026
ad8443b
Apply suggestion from @justinegeffen
justinegeffen Feb 5, 2026
46e9e58
Apply suggestion from @justinegeffen
justinegeffen Feb 5, 2026
fb38a55
Apply suggestion from @justinegeffen
justinegeffen Feb 5, 2026
4c2cf82
Apply suggestion from @justinegeffen
justinegeffen Feb 5, 2026
72359f5
Apply suggestion from @justinegeffen
justinegeffen Feb 5, 2026
58894da
Apply suggestion from @justinegeffen
justinegeffen Feb 5, 2026
f256bf5
Apply suggestion from @justinegeffen
justinegeffen Feb 5, 2026
cd9f947
Apply suggestion from @justinegeffen
justinegeffen Feb 5, 2026
c58e308
Apply suggestion from @justinegeffen
justinegeffen Feb 5, 2026
d3bdf76
Apply suggestion from @justinegeffen
justinegeffen Feb 5, 2026
b210629
Apply suggestion from @justinegeffen
justinegeffen Feb 5, 2026
af36c53
Apply suggestion from @justinegeffen
justinegeffen Feb 5, 2026
a559142
Update platform-enterprise_versioned_docs/version-25.3/enterprise/con…
justinegeffen Feb 5, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions platform-cloud/docs/pipeline-optimization/overview.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
---
title: "Pipeline resource optimization"
title: "Pipeline optimization"
description: "Optimize the resource usage of your pipelines to save time and money."
date: "12 Feb 2024"
tags: [compute, resource, optimization]
---

**Available from version 23.3.0**

Pipeline resource optimization takes the resource usage information from previous workflow runs to optimize subsequent runs.
Pipeline optimization takes the resource usage information from previous workflow runs to optimize subsequent runs.

When a run completes successfully, an _optimized profile_ is created. This profile consists of Nextflow configuration settings for each process and each resource directive (where applicable): `cpus`, `memory`, and `time`. The optimized setting for a given process and resource directive is based on the maximum use of that resource across all tasks in that process.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ services:
retries: 10
restart: always
volumes:
# To use the pipeline resource optimization service, uncomment the lines below to mount the initialization script
# To use the pipeline optimization service, uncomment the lines below to mount the initialization script
# - type: bind
# source: ./init.sql
# target: /docker-entrypoint-initdb.d/init.sql
Expand Down Expand Up @@ -141,7 +141,7 @@ services:
# - 7070:7070
# restart: always

# Uncomment the following section to enable the pipeline resource optimization service. Add TOWER_ENABLE_GROUNDSWELL
# Uncomment the following section to enable the pipeline optimization service. Add TOWER_ENABLE_GROUNDSWELL
# or GROUNDSWELL_SERVER_URL to tower.env environment variables
# groundswell:
# image: cr.seqera.io/private/nf-tower-enterprise/groundswell:0.3.3
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ TOWER_SMTP_PORT=587
TOWER_SMTP_USER=foo
TOWER_SMTP_PASSWORD=foo

# Pipeline resource optimization service. Set TOWER_ENABLE_GROUNDSWELL true to use default service URL (http://groundswell:8090)
# Pipeline optimization service. Set TOWER_ENABLE_GROUNDSWELL true to use default service URL (http://groundswell:8090)
# Use GROUNDSWELL_SERVER_URL to specify a custom service URL (enables the service by default)
TOWER_ENABLE_GROUNDSWELL=false
GROUNDSWELL_SERVER_URL="http://groundswell:8090"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ data:
TOWER_ENABLE_PLATFORMS: "altair-platform,awsbatch-platform,awscloud-platform,azbatch-platform,azcloud-platform,eks-platform,gke-platform,googlebatch-platform,googlecloud-platform,k8s-platform,lsf-platform,moab-platform,slurm-platform,uge-platform"
TOWER_REDIS_URL: "redis://<YOUR REDIS IP>:6379" # For a managed Redis service, specify the IP address or DNS name of the service endpoint. If you use the containerized Redis service, specify "redis" instead of an IP address.

# Pipeline resource optimization service. Set TOWER_ENABLE_GROUNDSWELL true to use default service URL (http://groundswell:8090)
# Pipeline optimization service. Set TOWER_ENABLE_GROUNDSWELL true to use default service URL (http://groundswell:8090)
# Use GROUNDSWELL_SERVER_URL to specify a custom service URL (enables the service by default)
TOWER_ENABLE_GROUNDSWELL: "false"
GROUNDSWELL_SERVER_URL: http://groundswell:8090
Expand Down
Original file line number Diff line number Diff line change
@@ -1,24 +1,25 @@
---
title: "Pipeline optimization"
description: "Configure pipeline resource optimization in your Seqera Enterprise deployment."
description: "Configure pipeline optimization in your Seqera Enterprise deployment."
date: "12 Feb 2024"
tags: [compute, resource, optimization, configuration]
---

[Pipeline resource optimization](../../pipeline-optimization/overview) takes the resource usage information from previous workflow runs to optimize subsequent runs.
[Pipeline optimization](../../pipeline-optimization/overview) takes the resource usage information from previous workflow runs to optimize subsequent runs.

The pipeline resource optimization service requires a separate database schema to store its internal data, but also requires access to the Seqera schema. The Seqera and optimization service schemas can coexist on the same database instance.
The pipeline optimization service requires a separate database schema to store its internal data, but also requires access to the Seqera schema. The Seqera and optimization service schemas can coexist on the same database instance.

## Docker Compose deployment

Docker Compose makes use of a separate container to set up the pipeline resource optimization service during initialization. Configuration steps differ for new and existing deployments.
Docker Compose makes use of a separate container to set up the pipeline optimization service during initialization. Configuration steps differ for new and existing deployments.

### New installation

To use the pipeline resource optimization service in a new Docker Compose installation of Seqera Enterprise, use the following steps:
To use the pipeline optimization service in a new Docker Compose installation of Seqera Enterprise, use the following steps:

1. To run the service from a custom URL, declare the URL with the `GROUNDSWELL_SERVER_URL` environment variable in `tower.env`. A non-zero value for this environment variable activates the optimization service automatically, so `TOWER_ENABLE_GROUNDSWELL` does not need to be set when you declare a custom URL.


2. Set the `TOWER_ENABLE_GROUNDSWELL` environment variable in `tower.env` to `true`. This enables the service at the default service URL `http://groundswell:8090`.

3. In your [docker-compose.yml](../_templates/docker/docker-compose.yml) file, uncomment the `groundswell` section at the bottom.
Expand All @@ -27,11 +28,11 @@ To use the pipeline resource optimization service in a new Docker Compose instal

4. Download the [init.sql](../_templates/docker/init.sql) file. Store this file in the mount path of your `docker-compose.yml` file, else update the `source: ./init.sql` line in your `docker-compose.yml` with the file path.

5. When the pipeline resource optimization service is active, pipelines that can be optimized display a lightbulb icon in your Launchpad. Any pipeline with at least one successful run can be optimized.
5. When the pipeline optimization service is active, pipelines that can be optimized display a lightbulb icon in your Launchpad. Any pipeline with at least one successful run can be optimized.

### Existing installation

To use the pipeline resource optimization service in an existing Docker Compose installation of Seqera Enterprise, use the following steps:
To use the pipeline optimization service in an existing Docker Compose installation of Seqera Enterprise, use the following steps:

1. To run the service from a custom URL, declare the URL with the `GROUNDSWELL_SERVER_URL` environment variable. A non-zero value for this environment variable activates the optimization service automatically, so `TOWER_ENABLE_GROUNDSWELL` does not need to be set when you declare a custom URL.

Expand Down Expand Up @@ -69,11 +70,11 @@ To use the pipeline resource optimization service in an existing Docker Compose
# SWELL_DB_URL=mysql://db1.abcdefghijkl.us-east-1.rds.amazonaws.com:3306/swell
```

7. When the pipeline resource optimization service is active, pipelines that can be optimized display a lightbulb icon in your Launchpad. Any pipeline with at least one successful run can be optimized.
7. When the pipeline optimization service is active, pipelines that can be optimized display a lightbulb icon in your Launchpad. Any pipeline with at least one successful run can be optimized.

## Kubernetes deployment

Kubernetes deployments use an [initContainer](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/) that runs during pod initialization to set up the pipeline resource optimization service. To use the service in new or existing Kubernetes installations of Seqera Enterprise, do the following:
Kubernetes deployments use an [initContainer](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/) that runs during pod initialization to set up the pipeline optimization service. To use the service in new or existing Kubernetes installations of Seqera Enterprise, do the following:

1. Download the [groundswell manifest](../_templates/k8s/groundswell.yml).

Expand Down Expand Up @@ -101,9 +102,10 @@ Kubernetes deployments use an [initContainer](https://kubernetes.io/docs/concept
FLUSH PRIVILEGES;
```

The initContainers process will wait until both the Seqera and pipeline resource optimization service databases are ready before starting the migration in the Seqera database and finally starting the optimization container.
The `initContainers` process waits until both the Seqera and pipeline optimization service databases are ready before starting the migration in the Seqera database, and finally starting the optimization container.


When the pipeline resource optimization service is active, pipelines that can be optimized display a lightbulb icon in your Launchpad. Any pipeline with at least one successful run can be optimized.
When the pipeline optimization service is active, pipelines that can be optimized display a lightbulb icon in your Launchpad. Any pipeline with at least one successful run can be optimized.


[platform-k8s]: ../platform-kubernetes
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
---
title: "Pipeline optimization: Docker Compose"
description: Deploy pipeline resource optimization with Docker Compose
description: Deploy pipeline optimization with Docker Compose
date: "12 Apr 2023"
tags: [docker, compose, groundswell, deployment]
tags: [docker, compose, groundswell, pipeline optimization, deployment]
---

This guide describes how to deploy the pipeline resource optimization service (Groundswell) for Seqera Platform Enterprise with Docker Compose.
This guide describes how to deploy the pipeline optimization service (referred to as `groundswell` in the configuration file) for Seqera Platform Enterprise using Docker Compose.

## Prerequisites

Expand Down Expand Up @@ -76,4 +76,4 @@ When pipeline optimization is active, pipelines with at least one successful run

## Configuration

See [Pipeline resource optimization](./configuration/pipeline_optimization) for additional configuration options.
See [Pipeline optimization](./configuration/pipeline_optimization) for additional configuration options.
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
---
title: "Pipeline optimization: Kubernetes"
description: Deploy pipeline resource optimization on Kubernetes
description: Deploy pipeline optimization on Kubernetes
date: "12 Apr 2023"
tags: [kubernetes, groundswell, deployment]
tags: [kubernetes, groundswell, pipeline optimization, deployment]
---

This guide describes how to deploy the pipeline resource optimization service (Groundswell) for Seqera Platform Enterprise on Kubernetes.
This guide describes how to deploy the pipeline optimization service (referred to as `groundswell` in the configuration file) for Seqera Platform Enterprise on Kubernetes.

## Prerequisites

Expand Down Expand Up @@ -62,4 +62,4 @@ When pipeline optimization is active, pipelines with at least one successful run

## Configuration

See [Pipeline resource optimization](./configuration/pipeline_optimization) for additional configuration options.
See [Pipeline optimization](./configuration/pipeline_optimization) for additional configuration options.
4 changes: 2 additions & 2 deletions platform-enterprise_docs/enterprise/install-groundswell.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
---
title: "Pipeline optimization"
description: Install pipeline resource optimization for Seqera Platform Enterprise
description: Install pipeline optimization for Seqera Platform Enterprise
date: "12 Apr 2023"
tags: [groundswell, pipeline, optimization, installation, deployment]
---

Pipeline resource optimization (Groundswell) uses resource usage data from previous workflow runs to optimize subsequent runs. Deploy after your Platform installation is complete.
Pipeline optimization can be deployed after your Platform installation is complete. It uses resource usage data from previous workflow runs to optimize subsequent runs.

## Deployment options

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ Use a managed Redis service for production:

- The `db` container should be used only for local testing. If you have configured this service elsewhere, you can remove this container.

- To configure the Seqera pipeline resource optimization service (`groundswell`), see [Pipeline resource optimization](./configuration/pipeline_optimization).
- To configure the Seqera pipeline optimization service (`groundswell`), see [Pipeline optimization](./configuration/pipeline_optimization).

- To deploy with Studios, see [Studios deployment](./studios).

Expand Down
4 changes: 2 additions & 2 deletions platform-enterprise_docs/pipeline-optimization/overview.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
---
title: "Pipeline resource optimization"
title: "Pipeline optimization"
description: "Optimize the resource usage of your pipelines to save time and money."
date: "12 Feb 2024"
tags: [compute, resource, optimization]
---

**Available from version 23.3.0**

Pipeline resource optimization takes the resource usage information from previous workflow runs to optimize subsequent runs.
Pipeline optimization takes the resource usage information from previous workflow runs to optimize subsequent runs.

When a run completes successfully, an _optimized profile_ is created. This profile consists of Nextflow configuration settings for each process and each resource directive (where applicable): `cpus`, `memory`, and `time`. The optimized setting for a given process and resource directive is based on the maximum use of that resource across all tasks in that process.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ services:
retries: 10
restart: always
volumes:
# To use the pipeline resource optimization service, uncomment the lines below to mount the initialization script
# To use the pipeline optimization service, uncomment the lines below to mount the initialization script
# - type: bind
# source: ./init.sql
# target: /docker-entrypoint-initdb.d/init.sql
Expand Down Expand Up @@ -139,7 +139,7 @@ services:
# - 7070:7070
# restart: always

# Uncomment the following section to enable the pipeline resource optimization service. Add TOWER_ENABLE_GROUNDSWELL
# Uncomment the following section to enable the pipeline optimization service. Add TOWER_ENABLE_GROUNDSWELL
# or GROUNDSWELL_SERVER_URL to tower.env environment variables
# groundswell:
# image: cr.seqera.io/private/nf-tower-enterprise/groundswell:0.3.3
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ TOWER_SMTP_PORT=587
TOWER_SMTP_USER=foo
TOWER_SMTP_PASSWORD=foo

# Pipeline resource optimization service. Set TOWER_ENABLE_GROUNDSWELL true to use default service URL (http://groundswell:8090)
# Pipeline optimization service. Set TOWER_ENABLE_GROUNDSWELL true to use default service URL (http://groundswell:8090)
# Use GROUNDSWELL_SERVER_URL to specify a custom service URL (enables the service by default)
TOWER_ENABLE_GROUNDSWELL=false
GROUNDSWELL_SERVER_URL="http://groundswell:8090"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ data:
TOWER_ENABLE_PLATFORMS: "altair-platform,awsbatch-platform,azbatch-platform,eks-platform,gke-platform,googlebatch-platform,k8s-platform,lsf-platform,moab-platform,slurm-platform,uge-platform"
TOWER_REDIS_URL: "redis://<YOUR REDIS IP>:6379" # For a managed Redis service, specify the IP address or DNS name of the service endpoint. If you use the containerized Redis service, specify "redis" instead of an IP address.

# Pipeline resource optimization service. Set TOWER_ENABLE_GROUNDSWELL true to use default service URL (http://groundswell:8090)
# Pipeline optimization service. Set TOWER_ENABLE_GROUNDSWELL true to use default service URL (http://groundswell:8090)
# Use GROUNDSWELL_SERVER_URL to specify a custom service URL (enables the service by default)
TOWER_ENABLE_GROUNDSWELL: "false"
GROUNDSWELL_SERVER_URL: http://groundswell:8090
Expand Down
Loading