From fad2e6996212bd0af4230fd0b883f01d6aec9bca Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Fri, 12 Dec 2025 16:32:40 +0100 Subject: [PATCH 1/4] Add APT build template feature specification MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Introduce spec for apt/debian:v1 build template as alternative to conda: - Support for APT packages via package list or requirements file - Both Docker and Singularity builds supported - Configurable via aptOpts (baseImage, basePackages, commands) - Default base image: ubuntu:24.04 - Uses --no-install-recommends for minimal images 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 --- .../checklists/requirements.md | 36 +++++ specs/251212-apt-build-template/spec.md | 147 ++++++++++++++++++ 2 files changed, 183 insertions(+) create mode 100644 specs/251212-apt-build-template/checklists/requirements.md create mode 100644 specs/251212-apt-build-template/spec.md diff --git a/specs/251212-apt-build-template/checklists/requirements.md b/specs/251212-apt-build-template/checklists/requirements.md new file mode 100644 index 000000000..d1c822603 --- /dev/null +++ b/specs/251212-apt-build-template/checklists/requirements.md @@ -0,0 +1,36 @@ +# Specification Quality Checklist: APT Build Template + +**Purpose**: Validate specification completeness and quality before proceeding to planning +**Created**: 2025-12-12 +**Feature**: [spec.md](../spec.md) + +## Content Quality + +- [x] No implementation details (languages, frameworks, APIs) +- [x] Focused on user value and business needs +- [x] Written for non-technical stakeholders +- [x] All mandatory sections completed + +## Requirement Completeness + +- [x] No [NEEDS CLARIFICATION] markers remain +- [x] Requirements are testable and unambiguous +- [x] Success criteria are measurable +- [x] Success criteria are technology-agnostic (no implementation details) +- [x] All acceptance scenarios are defined +- [x] Edge cases are identified +- [x] Scope is clearly bounded +- [x] Dependencies and assumptions identified + +## Feature Readiness + +- [x] All functional requirements have clear acceptance criteria +- [x] User scenarios cover primary flows +- [x] Feature meets measurable outcomes defined in Success Criteria +- [x] No implementation details leak into specification + +## Notes + +- All checklist items pass validation +- Specification is ready for `/speckit.clarify` or `/speckit.plan` +- Assumptions documented for implementation flexibility (default base image, repository scope) diff --git a/specs/251212-apt-build-template/spec.md b/specs/251212-apt-build-template/spec.md new file mode 100644 index 000000000..689fc756f --- /dev/null +++ b/specs/251212-apt-build-template/spec.md @@ -0,0 +1,147 @@ +# Feature Specification: APT Build Template + +**Feature Branch**: `251212-apt-build-template` +**Created**: 2025-12-12 +**Status**: Draft +**Input**: User description: "Add a build template based on apt package manager as alternative to conda. It should be supported by both Docker and Singularity builds. Selection should be made by using the buildTemplate option." + +## User Scenarios & Testing *(mandatory)* + +### User Story 1 - Build Container with APT Packages via Package List (Priority: P1) + +A bioinformatics user needs to create a container with system-level tools available through Debian/Ubuntu APT repositories (e.g., samtools, bedtools, curl, git). They want to specify a list of APT packages directly in their Wave API request and have Wave build a container with those packages installed. + +**Why this priority**: This is the core functionality - allowing users to specify APT packages directly. Without this, the feature has no value. + +**Independent Test**: Can be fully tested by submitting a container token request with `buildTemplate: "apt/debian:v1"` and a list of package names, then verifying the resulting container has those packages installed and functional. + +**Acceptance Scenarios**: + +1. **Given** a user submits a container token request with `buildTemplate: "apt/debian:v1"` and packages `["curl", "wget", "git"]`, **When** Wave processes the request, **Then** a container is built with curl, wget, and git installed and accessible from the command line. + +2. **Given** a user submits a container token request with APT packages and a Singularity format flag, **When** Wave processes the request, **Then** a Singularity container is built with the specified packages installed. + +3. **Given** a user specifies an APT package that doesn't exist in the repositories, **When** Wave attempts to build the container, **Then** the build fails with a clear error message indicating which package was not found. + +--- + +### User Story 2 - Build Container with APT Packages via Requirements File (Priority: P2) + +A user wants to define their APT package dependencies in a file (similar to conda's environment.yml) and have Wave build a container from that file specification. This allows version control of dependencies and easier sharing of environment definitions. + +**Why this priority**: File-based specification is important for reproducibility and CI/CD workflows, but users can achieve basic functionality with package lists first. + +**Independent Test**: Can be tested by submitting a request with a base64-encoded requirements file containing APT package names (one per line), then verifying the container has all listed packages installed. + +**Acceptance Scenarios**: + +1. **Given** a user submits a container token request with `buildTemplate: "apt/debian:v1"` and an `environment` field containing a base64-encoded file with package names, **When** Wave processes the request, **Then** a container is built with all packages from the file installed. + +2. **Given** a requirements file contains comments (lines starting with #) and empty lines, **When** Wave parses the file, **Then** comments and empty lines are ignored and only valid package names are processed. + +--- + +### User Story 3 - Customize APT Build with Base Image and Additional Commands (Priority: P3) + +An advanced user needs to customize their APT-based container by specifying a different base image (e.g., Ubuntu 22.04 instead of default Debian) and running additional commands after package installation (e.g., downloading additional data, setting environment variables). + +**Why this priority**: Customization extends the feature's flexibility but is not required for basic functionality. + +**Independent Test**: Can be tested by submitting a request with custom `aptOpts` including a different base image and post-install commands, then verifying the container uses the specified image and executed the custom commands. + +**Acceptance Scenarios**: + +1. **Given** a user specifies `aptOpts.baseImage: "ubuntu:22.04"` in their request, **When** Wave builds the container, **Then** the resulting container is based on Ubuntu 22.04 instead of the default Debian image. + +2. **Given** a user specifies `aptOpts.commands: ["echo 'setup complete' > /setup.log"]`, **When** Wave builds the container, **Then** the custom command is executed and `/setup.log` exists in the final container. + +3. **Given** a user specifies `aptOpts.basePackages: ["ca-certificates", "locales"]`, **When** Wave builds the container, **Then** these packages are installed in addition to the user-specified packages. + +--- + +### Edge Cases + +- What happens when the user specifies conflicting packages (packages that cannot be installed together)? + - The build should fail with the APT resolver error message. + +- What happens when the user specifies a package with version constraints (e.g., `nginx=1.18.0-0ubuntu1`)? + - Version-pinned packages should be supported using standard APT version syntax. + +- What happens when the base image doesn't have APT available (e.g., Alpine Linux)? + - The build should fail with a clear error indicating APT is not available in the base image. + +- What happens when network issues prevent package download during build? + - Standard Docker/Singularity build timeout and retry behavior applies; build fails with network error. + +## Requirements *(mandatory)* + +### Functional Requirements + +- **FR-001**: System MUST support a new build template identifier `apt/debian:v1` selectable via the `buildTemplate` request parameter. + +- **FR-002**: System MUST generate valid Dockerfile content when APT template is selected for Docker builds. + +- **FR-003**: System MUST generate valid Singularity definition file content when APT template is selected for Singularity builds. + +- **FR-004**: System MUST support specifying APT packages as a list of package names in the `packages.entries` field. + +- **FR-005**: System MUST support specifying APT packages via a file (base64-encoded in `packages.environment` field) with one package name per line. + +- **FR-006**: System MUST support APT package version pinning using standard APT syntax (e.g., `package=version`). + +- **FR-007**: System MUST support an `aptOpts` configuration object in `PackagesSpec` with the following options: + - `baseImage`: Base Docker image to use (default: standard Debian-based image) + - `basePackages`: List of packages to always install (e.g., ca-certificates) + - `commands`: List of additional shell commands to run after package installation + +- **FR-008**: System MUST run `apt-get update` before installing packages to ensure package lists are current. + +- **FR-009**: System MUST clean up APT cache after installation to minimize final image size (`apt-get clean`, remove `/var/lib/apt/lists/*`). + +- **FR-010**: System MUST set `DEBIAN_FRONTEND=noninteractive` during package installation to prevent interactive prompts. + +- **FR-011**: System MUST use `--no-install-recommends` flag by default to produce minimal container images with explicit dependencies only. + +- **FR-012**: System MUST add a new package type `APT` to the `PackagesSpec.Type` enumeration. + +- **FR-013**: System MUST log the build template used for APT builds in the build record for traceability. + +### Key Entities + +- **BuildTemplate**: Extended to include the `apt/debian:v1` constant for APT-based builds. + +- **PackagesSpec.Type**: Extended to include `APT` as a valid package type alongside `CONDA` and `CRAN`. + +- **AptOpts**: New configuration entity for APT-specific build options (baseImage, basePackages, commands). + +- **Requirements File**: Plain text file with one APT package name per line, supporting comments with `#` prefix. + +## Success Criteria *(mandatory)* + +### Measurable Outcomes + +- **SC-001**: Users can successfully build containers with APT packages using the same request flow as existing Conda/CRAN templates. + +- **SC-002**: Both Docker and Singularity container formats are supported with identical package specifications. + +- **SC-003**: Container builds complete successfully for standard Debian/Ubuntu APT packages available in default repositories. + +- **SC-004**: Built containers have all specified packages installed and functional (verifiable via `dpkg -l` or direct command execution). + +- **SC-005**: Final container images follow size optimization best practices (no APT cache, no package lists retained). + +- **SC-006**: Build failures due to invalid packages produce clear, actionable error messages identifying the problematic package. + +## Clarifications + +### Session 2025-12-12 + +- Q: What should be the default base image for APT builds? → A: `ubuntu:24.04` (Latest Ubuntu LTS) +- Q: How should APT recommended packages be handled? → A: Default to `--no-install-recommends` for minimal images + +## Assumptions + +- The default base image is `ubuntu:24.04` (Ubuntu 24.04 LTS), providing long-term support, broad package availability, and security updates through 2029. +- Only official Debian/Ubuntu repositories are supported in v1; custom repository support may be added in future versions. +- Package names follow standard APT naming conventions without architecture suffixes (architecture determined by build platform). +- The requirements file format is intentionally simple (one package per line) to minimize parsing complexity and user errors. From 00210845167e9e0f75ce51f6cf5ed7cabe0498b0 Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Fri, 12 Dec 2025 16:54:28 +0100 Subject: [PATCH 2/4] Add implementation plan for APT build template MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 0-1 artifacts: - plan.md: Technical context, constitution check, project structure - research.md: Architecture decisions, patterns, risk assessment - data-model.md: Entity definitions, validation rules - contracts/api-changes.md: API schema, examples, validation - quickstart.md: Implementation guide Key decisions: - Support both entries (list) and environment (newline-separated file) - 2 template files (dockerfile + singularityfile) - Default base image: ubuntu:24.04 - Uses --no-install-recommends for minimal images 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 --- .../contracts/api-changes.md | 234 ++++++++++++++++++ specs/251212-apt-build-template/data-model.md | 124 ++++++++++ specs/251212-apt-build-template/plan.md | 97 ++++++++ specs/251212-apt-build-template/quickstart.md | 174 +++++++++++++ specs/251212-apt-build-template/research.md | 157 ++++++++++++ specs/251212-apt-build-template/spec.md | 26 +- 6 files changed, 790 insertions(+), 22 deletions(-) create mode 100644 specs/251212-apt-build-template/contracts/api-changes.md create mode 100644 specs/251212-apt-build-template/data-model.md create mode 100644 specs/251212-apt-build-template/plan.md create mode 100644 specs/251212-apt-build-template/quickstart.md create mode 100644 specs/251212-apt-build-template/research.md diff --git a/specs/251212-apt-build-template/contracts/api-changes.md b/specs/251212-apt-build-template/contracts/api-changes.md new file mode 100644 index 000000000..890c6fafd --- /dev/null +++ b/specs/251212-apt-build-template/contracts/api-changes.md @@ -0,0 +1,234 @@ +# API Contract Changes: APT Build Template + +**Feature**: 251212-apt-build-template +**Date**: 2025-12-12 + +## Overview + +This document describes the API changes required to support APT build templates. All changes are additive and backward-compatible. + +## Endpoint: POST /container-token + +**No changes to endpoint URL or method**. Changes are in request body schema only. + +### Request Body Changes + +#### New Enum Value: packages.type + +```json +{ + "packages": { + "type": "APT" // NEW: alongside existing "CONDA", "CRAN" + } +} +``` + +#### New Field: packages.aptOpts + +```json +{ + "packages": { + "type": "APT", + "aptOpts": { + "baseImage": "ubuntu:24.04", + "basePackages": "ca-certificates", + "commands": ["echo 'setup complete'"] + } + } +} +``` + +#### New Build Template Value + +```json +{ + "buildTemplate": "apt/debian:v1" // NEW template identifier +} +``` + +### Full Request Examples + +#### Example 1: APT packages via entries list + +```json +{ + "buildTemplate": "apt/debian:v1", + "packages": { + "type": "APT", + "entries": [ + "curl", + "wget", + "git", + "build-essential" + ] + } +} +``` + +#### Example 2: APT packages via environment file + +```json +{ + "buildTemplate": "apt/debian:v1", + "packages": { + "type": "APT", + "environment": "IyBTeXN0ZW0gdXRpbGl0aWVzCmN1cmwKd2dldApnaXQK" + } +} +``` + +Note: `environment` is base64-encoded. Decoded content (newline-separated packages): +```text +# System utilities +curl +wget +git +``` + +#### Example 3: APT with custom options + +```json +{ + "buildTemplate": "apt/debian:v1", + "packages": { + "type": "APT", + "entries": ["nginx"], + "aptOpts": { + "baseImage": "ubuntu:22.04", + "basePackages": "ca-certificates locales", + "commands": [ + "locale-gen en_US.UTF-8", + "update-locale LANG=en_US.UTF-8" + ] + } + } +} +``` + +#### Example 4: APT with version pinning + +```json +{ + "buildTemplate": "apt/debian:v1", + "packages": { + "type": "APT", + "entries": [ + "nginx=1.18.0-0ubuntu1", + "curl" + ] + } +} +``` + +#### Example 5: Singularity format + +```json +{ + "buildTemplate": "apt/debian:v1", + "format": "sif", + "packages": { + "type": "APT", + "entries": ["samtools", "bcftools"] + } +} +``` + +### Response + +**No changes to response schema**. Existing `SubmitContainerTokenResponse` structure unchanged. + +## Schema Definitions + +### AptOpts Schema + +```yaml +AptOpts: + type: object + properties: + baseImage: + type: string + description: Base Docker image for APT builds + default: "ubuntu:24.04" + example: "ubuntu:22.04" + basePackages: + type: string + description: Space-separated list of packages to always install + default: "ca-certificates" + example: "ca-certificates locales" + commands: + type: array + items: + type: string + description: Additional shell commands to run after package installation + example: ["locale-gen en_US.UTF-8"] +``` + +### PackagesSpec.Type Enum + +```yaml +PackagesSpec.Type: + type: string + enum: + - CONDA + - CRAN + - APT # NEW +``` + +### BuildTemplate Values + +```yaml +BuildTemplate: + type: string + enum: + - "conda/pixi:v1" + - "conda/micromamba:v1" + - "conda/micromamba:v2" + - "cran/installr:v1" + - "apt/debian:v1" # NEW +``` + +## Validation Rules + +### Request Validation + +| Condition | Error | +|-----------|-------| +| `type: APT` without `buildTemplate: apt/debian:v1` | `400 Bad Request: APT packages require buildTemplate 'apt/debian:v1'` | +| `buildTemplate: apt/debian:v1` without `type: APT` | `400 Bad Request: Build template 'apt/debian:v1' requires package type 'APT'` | +| Both `entries` and `environment` provided | `400 Bad Request: Cannot specify both entries and environment` | +| Neither `entries` nor `environment` provided | `400 Bad Request: Must specify either entries or environment` | +| Empty `entries` list | `400 Bad Request: Package entries cannot be empty` | + +## Error Responses + +### Build Failures + +APT-related build failures return standard Wave build error format: + +```json +{ + "status": "FAILED", + "error": "E: Unable to locate package nonexistent-package", + "buildId": "abc123" +} +``` + +## Backward Compatibility + +| Aspect | Status | +|--------|--------| +| Existing CONDA requests | Unchanged | +| Existing CRAN requests | Unchanged | +| Existing build templates | Unchanged | +| Response schema | Unchanged | +| Authentication | Unchanged | + +## Testing Considerations + +### Contract Tests + +1. Verify `type: APT` accepted in request +2. Verify `aptOpts` fields parsed correctly +3. Verify `buildTemplate: apt/debian:v1` accepted +4. Verify validation errors for invalid combinations +5. Verify Singularity format works with APT diff --git a/specs/251212-apt-build-template/data-model.md b/specs/251212-apt-build-template/data-model.md new file mode 100644 index 000000000..89fe0af15 --- /dev/null +++ b/specs/251212-apt-build-template/data-model.md @@ -0,0 +1,124 @@ +# Data Model: APT Build Template + +**Feature**: 251212-apt-build-template +**Date**: 2025-12-12 + +## Entity Overview + +This feature extends existing entities rather than creating new persistence structures. No database schema changes required. + +## Modified Entities + +### 1. PackagesSpec.Type (Enumeration) + +**Location**: `wave-api/src/main/java/io/seqera/wave/api/PackagesSpec.java` + +**Current Values**: +- `CONDA` +- `CRAN` + +**New Value**: +- `APT` - Debian/Ubuntu APT package manager + +### 2. PackagesSpec (API Model) + +**Location**: `wave-api/src/main/java/io/seqera/wave/api/PackagesSpec.java` + +**Existing Fields** (unchanged): +| Field | Type | Description | +|-------|------|-------------| +| `type` | `Type` | Package manager type | +| `environment` | `String` | Base64-encoded environment file | +| `entries` | `List` | List of package names | +| `channels` | `List` | Package channels/repositories | +| `condaOpts` | `CondaOpts` | Conda-specific options | +| `pixiOpts` | `PixiOpts` | Pixi-specific options | +| `cranOpts` | `CranOpts` | CRAN-specific options | + +**New Field**: +| Field | Type | Description | +|-------|------|-------------| +| `aptOpts` | `AptOpts` | APT-specific build options | + +**Validation Rules**: +- When `type == APT`: + - Either `entries` OR `environment` must be provided (not both) + - `environment` is parsed as newline-separated package names (comments with `#` and empty lines ignored) + - `channels` field is ignored (APT uses default repos) + - `aptOpts` is optional (defaults applied if null) + +### 3. BuildTemplate (Constants) + +**Location**: `wave-api/src/main/java/io/seqera/wave/api/BuildTemplate.java` + +**Existing Constants**: +- `CONDA_PIXI_V1 = "conda/pixi:v1"` +- `CONDA_MICROMAMBA_V1 = "conda/micromamba:v1"` +- `CONDA_MICROMAMBA_V2 = "conda/micromamba:v2"` +- `CRAN_INSTALLR_V1 = "cran/installr:v1"` + +**New Constant**: +- `APT_DEBIAN_V1 = "apt/debian:v1"` + +## New Entities + +### 4. AptOpts (Configuration Model) + +**Location**: `wave-api/src/main/java/io/seqera/wave/config/AptOpts.java` + +**Purpose**: Configuration options for APT-based container builds + +| Field | Type | Default | Description | +|-------|------|---------|-------------| +| `baseImage` | `String` | `ubuntu:24.04` | Base Docker image for builds | +| `basePackages` | `String` | `ca-certificates` | Packages always installed before user packages | +| `commands` | `List` | `null` | Additional shell commands to run after installation | + +**Validation Rules**: +- `baseImage` must be non-empty if provided +- `basePackages` can be empty string to skip base packages +- `commands` entries must be valid shell commands + +**Methods**: +- `AptOpts()` - Default constructor with default values +- `AptOpts(Map opts)` - Constructor from map (for JSON deserialization) +- `withBaseImage(String)` - Fluent setter +- `withBasePackages(String)` - Fluent setter +- `withCommands(List)` - Fluent setter +- `equals()`, `hashCode()`, `toString()` - Standard object methods + +## Entity Relationships + +```text +SubmitContainerTokenRequest +├── buildTemplate: String ("apt/debian:v1") +└── packages: PackagesSpec + ├── type: Type.APT + ├── entries: List # Option 1: package list + ├── environment: String # Option 2: base64-encoded newline-separated packages + └── aptOpts: AptOpts + ├── baseImage: String + ├── basePackages: String + └── commands: List +``` + +## State Transitions + +No state machines introduced. Build operations follow existing Wave build lifecycle: + +```text +Request → Validate → Generate Container File → Queue Build → Build → Complete/Fail +``` + +## Data Volume Considerations + +- No new persistence requirements +- Existing `WaveBuildRecord` stores `buildTemplate` field (already exists) +- No additional storage for APT-specific data + +## Backward Compatibility + +- All changes are additive +- Existing API requests without `type: APT` unaffected +- Existing build templates continue to work unchanged +- No migration required diff --git a/specs/251212-apt-build-template/plan.md b/specs/251212-apt-build-template/plan.md new file mode 100644 index 000000000..51f4b6654 --- /dev/null +++ b/specs/251212-apt-build-template/plan.md @@ -0,0 +1,97 @@ +# Implementation Plan: APT Build Template + +**Branch**: `251212-apt-build-template` | **Date**: 2025-12-12 | **Spec**: [spec.md](./spec.md) +**Input**: Feature specification from `/specs/251212-apt-build-template/spec.md` + +## Summary + +Add an `apt/debian:v1` build template to Wave that allows users to build containers with Debian/Ubuntu APT packages. This extends the existing build template system (which supports conda/pixi/cran) to support system-level package installation via APT. The implementation follows the established patterns used by CondaHelper, PixiHelper, and CranHelper. + +## Technical Context + +**Language/Version**: Java 21+ / Groovy 4.x +**Primary Dependencies**: Micronaut 4.x, Netty, Reactor +**Storage**: PostgreSQL (build records), Redis (caching) +**Testing**: Spock 2 framework with Testcontainers +**Target Platform**: Linux server (Kubernetes/Docker) +**Project Type**: Existing microservice extension +**Performance Goals**: Build queue within 100ms, proxy operations <200ms p95 +**Constraints**: Non-blocking I/O required, follows existing template patterns +**Scale/Scope**: Extension to existing system - 6-8 new/modified files + +## Constitution Check + +*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.* + +| Principle | Status | Notes | +|-----------|--------|-------| +| I. Service-Oriented Architecture | PASS | Extends existing ContainerHelper dispatch; AptHelper isolated | +| II. Container Platform Agnosticism | PASS | Supports Docker and Singularity via templates | +| III. Ephemeral-First Design | PASS | No new persistence requirements | +| IV. Proxy Transparency | N/A | Not a proxy feature | +| V. Async-by-Default Operations | PASS | Uses existing async build pipeline | +| VI. Security Scanning Integration | N/A | No changes to scanning | +| VII. Multi-Platform Build Support | PASS | Templates work for amd64/arm64 | +| Testing Requirements | PASS | Will use Spock + Testcontainers pattern | +| Performance Standards | PASS | No new latency paths introduced | + +## Project Structure + +### Documentation (this feature) + +```text +specs/251212-apt-build-template/ +├── spec.md # Feature specification +├── plan.md # This file +├── research.md # Phase 0 output +├── data-model.md # Phase 1 output +├── quickstart.md # Phase 1 output +├── contracts/ # Phase 1 output +│ └── api-changes.md # API contract changes +├── checklists/ +│ └── requirements.md # Specification quality checklist +└── tasks.md # Phase 2 output (created by /speckit.tasks) +``` + +### Source Code (repository root) + +```text +wave-api/src/main/java/io/seqera/wave/ +├── api/ +│ ├── BuildTemplate.java # [MODIFY] Add APT_DEBIAN_V1 constant +│ └── PackagesSpec.java # [MODIFY] Add APT type, aptOpts field +└── config/ + └── AptOpts.java # [NEW] APT build options + +src/main/groovy/io/seqera/wave/util/ +├── ContainerHelper.groovy # [MODIFY] Add APT dispatch logic +└── AptHelper.groovy # [NEW] APT template rendering + +src/main/resources/templates/ +└── apt-debian-v1/ # [NEW] Template directory + ├── dockerfile-apt-packages.txt + └── singularityfile-apt-packages.txt + +src/test/groovy/io/seqera/wave/util/ +└── AptHelperTest.groovy # [NEW] Unit tests + +docs/ +└── api.md # [MODIFY] Add APT build template documentation + # - Add aptOpts to request schema + # - Add APT to packages.type enum + # - Add apt/debian:v1 example + # - Add apt/debian:v1 to buildTemplate values +``` + +**Structure Decision**: Follows existing Wave conventions - API types in `wave-api/`, implementation in `src/main/groovy/`, templates in `src/main/resources/templates/`. Documentation follows existing guide structure. + +## Complexity Tracking + +> No constitution violations requiring justification. + +| Check | Result | +|-------|--------| +| New services required | None - extends existing | +| New persistence entities | None - uses existing BuildRecord | +| Breaking API changes | None - additive only | +| External dependencies | None - APT is in base images | diff --git a/specs/251212-apt-build-template/quickstart.md b/specs/251212-apt-build-template/quickstart.md new file mode 100644 index 000000000..fc6785e76 --- /dev/null +++ b/specs/251212-apt-build-template/quickstart.md @@ -0,0 +1,174 @@ +# Quickstart: APT Build Template Implementation + +**Feature**: 251212-apt-build-template +**Date**: 2025-12-12 + +## Prerequisites + +- Java 21+ installed +- Gradle wrapper available (`./gradlew`) +- Wave development environment configured (see CLAUDE.md) + +## Implementation Order + +Follow this sequence to implement the APT build template: + +### Step 1: Add AptOpts Configuration Class + +**File**: `wave-api/src/main/java/io/seqera/wave/config/AptOpts.java` + +```java +package io.seqera.wave.config; + +import java.util.List; +import java.util.Map; +import java.util.Objects; + +public class AptOpts { + public static final String DEFAULT_BASE_IMAGE = "ubuntu:24.04"; + public static final String DEFAULT_BASE_PACKAGES = "ca-certificates"; + + public String baseImage; + public String basePackages; + public List commands; + + public AptOpts() { + this(Map.of()); + } + + public AptOpts(Map opts) { + this.baseImage = opts.containsKey("baseImage") + ? opts.get("baseImage").toString() + : DEFAULT_BASE_IMAGE; + this.basePackages = opts.containsKey("basePackages") + ? (String)opts.get("basePackages") + : DEFAULT_BASE_PACKAGES; + this.commands = opts.containsKey("commands") + ? (List)opts.get("commands") + : null; + } + + // Add fluent setters, equals, hashCode, toString +} +``` + +### Step 2: Update PackagesSpec + +**File**: `wave-api/src/main/java/io/seqera/wave/api/PackagesSpec.java` + +Add to enum: +```java +public enum Type { CONDA, CRAN, APT } +``` + +Add field: +```java +public AptOpts aptOpts; +``` + +Add fluent setter: +```java +public PackagesSpec withAptOpts(AptOpts opts) { + this.aptOpts = opts; + return this; +} +``` + +### Step 3: Update BuildTemplate + +**File**: `wave-api/src/main/java/io/seqera/wave/api/BuildTemplate.java` + +Add constant: +```java +public static final String APT_DEBIAN_V1 = "apt/debian:v1"; +``` + +### Step 4: Create Template Files + +**Directory**: `src/main/resources/templates/apt-debian-v1/` + +Create 2 template files (APT has no environment file concept, only package list): +- `dockerfile-apt-packages.txt` +- `singularityfile-apt-packages.txt` + +### Step 5: Create AptHelper + +**File**: `src/main/groovy/io/seqera/wave/util/AptHelper.groovy` + +Follow CranHelper pattern for structure. + +### Step 6: Update ContainerHelper + +**File**: `src/main/groovy/io/seqera/wave/util/ContainerHelper.groovy` + +Add dispatch logic in `containerFileFromRequest()`. + +### Step 7: Write Tests + +**File**: `src/test/groovy/io/seqera/wave/util/AptHelperTest.groovy` + +### Step 8: Update Documentation + +**File**: `docs/api.md` + +Add APT build template documentation: +- Add `APT` to the `type` field description +- Add `aptOpts` schema to request parameters +- Add `apt/debian:v1` to `buildTemplate` supported values +- Add example curl command for APT builds + +## Quick Verification + +After implementation, verify with: + +```bash +# Build the project +./gradlew assemble + +# Run tests +./gradlew test --tests 'AptHelperTest' + +# Run all tests +./gradlew test +``` + +## Sample Test Request + +```bash +curl -X POST http://localhost:9090/container-token \ + -H "Content-Type: application/json" \ + -d '{ + "buildTemplate": "apt/debian:v1", + "packages": { + "type": "APT", + "entries": ["curl", "wget", "git"] + } + }' +``` + +## Key Files Reference + +| File | Purpose | +|------|---------| +| `wave-api/.../config/AptOpts.java` | APT build configuration | +| `wave-api/.../api/PackagesSpec.java` | Add APT type and aptOpts | +| `wave-api/.../api/BuildTemplate.java` | Add APT_DEBIAN_V1 constant | +| `src/.../util/AptHelper.groovy` | Template rendering logic | +| `src/.../util/ContainerHelper.groovy` | Add dispatch logic | +| `src/.../resources/templates/apt-debian-v1/` | Template files (2 files) | +| `src/test/.../util/AptHelperTest.groovy` | Unit tests | +| `docs/api.md` | API documentation | + +## Common Issues + +### Import Errors +Ensure `AptOpts` import is added to `PackagesSpec.java`: +```java +import io.seqera.wave.config.AptOpts; +``` + +### Template Not Found +Verify template files are in correct location under `src/main/resources/templates/apt-debian-v1/` + +### Build Failures +Check that `DEBIAN_FRONTEND=noninteractive` is set to prevent APT prompts. diff --git a/specs/251212-apt-build-template/research.md b/specs/251212-apt-build-template/research.md new file mode 100644 index 000000000..3e49260a3 --- /dev/null +++ b/specs/251212-apt-build-template/research.md @@ -0,0 +1,157 @@ +# Research: APT Build Template + +**Feature**: 251212-apt-build-template +**Date**: 2025-12-12 + +## Executive Summary + +Research confirms that adding APT build template support is straightforward by following established patterns. The existing template system provides clear extension points with minimal risk. + +## Research Findings + +### 1. Build Template Architecture + +**Decision**: Follow existing CranHelper pattern for APT implementation + +**Rationale**: +- CranHelper is the closest analogue (single-stage build, system package manager) +- Clear separation: API types in `wave-api/`, helpers in `src/main/groovy/io/seqera/wave/util/` +- Template files in `src/main/resources/templates/{template-name}/` +- TemplateRenderer provides `{{variable}}` substitution + +**Alternatives Considered**: +- Multi-stage build like CondaHelper v2: Rejected - APT doesn't benefit from multi-stage (no separate package manager image) +- Pixi pattern: Rejected - designed for lock files and conda environments + +### 2. PackagesSpec.Type Enumeration + +**Decision**: Add `APT` as new enum value alongside `CONDA` and `CRAN` + +**Rationale**: +- Follows established pattern for package manager types +- Enables type-specific validation and dispatch +- Located in `wave-api/src/main/java/io/seqera/wave/api/PackagesSpec.java` + +**Alternatives Considered**: +- String-based type: Rejected - loses type safety, inconsistent with existing code + +### 3. AptOpts Configuration Class + +**Decision**: Create new `AptOpts.java` following `CranOpts`/`CondaOpts` pattern + +**Rationale**: +- Consistent with existing opts classes +- Fields: `baseImage`, `basePackages`, `commands` +- Default baseImage: `ubuntu:24.04` (per clarification) +- Located in `wave-api/src/main/java/io/seqera/wave/config/AptOpts.java` + +**Alternatives Considered**: +- Reuse CondaOpts: Rejected - different semantics (no mambaImage, different defaults) + +### 4. Template Structure + +**Decision**: Create 2 template files in `templates/apt-debian-v1/` + +**Rationale**: +- Pattern: `{dockerfile|singularityfile}-apt-packages.txt` +- Both `entries` (list) and `environment` (newline-separated file) use the same template +- `environment` file is parsed into a package list before template rendering +- Both Docker and Singularity formats required per spec + +**Template Variables**: +| Variable | Description | Example | +|----------|-------------|---------| +| `{{base_image}}` | Base Docker image | `ubuntu:24.04` | +| `{{base_packages}}` | System packages to pre-install | `ca-certificates` | +| `{{target}}` | Space-separated package list | `curl wget git` | + +### 5. APT Best Practices + +**Decision**: Apply container best practices in templates + +**Rationale**: +- `DEBIAN_FRONTEND=noninteractive` prevents interactive prompts +- `--no-install-recommends` minimizes image size (per clarification) +- `apt-get clean && rm -rf /var/lib/apt/lists/*` removes cache +- Single `RUN` layer for install + cleanup reduces image layers + +**Template Pattern**: +```dockerfile +FROM {{base_image}} +ENV DEBIAN_FRONTEND=noninteractive +RUN apt-get update \ + && apt-get install -y --no-install-recommends {{base_packages}} {{target}} \ + && apt-get clean \ + && rm -rf /var/lib/apt/lists/* +``` + +### 6. ContainerHelper Dispatch Logic + +**Decision**: Add APT dispatch in `containerFileFromRequest()` method + +**Rationale**: +- Existing dispatch at `ContainerHelper.groovy:59-78` +- Add condition: `if(req.buildTemplate == APT_DEBIAN_V1) return AptHelper.containerFile(spec, singularity)` +- No default template for APT (must be explicitly requested) + +**Alternatives Considered**: +- Auto-detect APT type: Rejected - explicit template selection is clearer, avoids ambiguity + +### 7. Requirements File Format + +**Decision**: Simple plain text format (one package per line) + +**Rationale**: +- Consistent with APT conventions (`apt install $(cat packages.txt)`) +- Comments with `#` prefix supported +- Empty lines ignored +- No version constraint syntax in file (use entries for pinning) + +**Example**: +```text +# System utilities +curl +wget +git +# Build tools +build-essential +``` + +### 8. BuildTemplate Constant + +**Decision**: Add `APT_DEBIAN_V1 = "apt/debian:v1"` constant + +**Rationale**: +- Follows naming convention: `{manager}/{variant}:{version}` +- "debian" variant indicates Debian/Ubuntu APT (vs potential future alpine/apk) +- Version v1 allows future iteration + +**Alternatives Considered**: +- `apt/ubuntu:v1`: Rejected - APT works on Debian family, not Ubuntu-specific +- `deb/apt:v1`: Rejected - inconsistent with existing naming + +## Risk Assessment + +| Risk | Likelihood | Impact | Mitigation | +|------|------------|--------|------------| +| Breaking existing templates | Low | High | Additive changes only, existing tests remain | +| APT package not found | Medium | Low | Clear error messages from APT propagated | +| Image size bloat | Medium | Medium | `--no-install-recommends` default | +| Version conflicts | Low | Low | Standard APT resolver handles this | + +## Implementation Dependencies + +1. No external library dependencies required +2. No database schema changes required +3. No configuration file changes required (optional defaults could be added later) + +## Next Steps + +1. Create `AptOpts.java` configuration class +2. Add `APT` to `PackagesSpec.Type` enum +3. Add `APT_DEBIAN_V1` constant to `BuildTemplate.java` +4. Create `AptHelper.groovy` with template rendering logic +5. Create 2 template files in `templates/apt-debian-v1/` (dockerfile + singularityfile) +6. Add dispatch logic to `ContainerHelper.groovy` +7. Write unit tests in `AptHelperTest.groovy` +8. Update `docs/api.md` with APT build template documentation diff --git a/specs/251212-apt-build-template/spec.md b/specs/251212-apt-build-template/spec.md index 689fc756f..101dee63f 100644 --- a/specs/251212-apt-build-template/spec.md +++ b/specs/251212-apt-build-template/spec.md @@ -25,23 +25,7 @@ A bioinformatics user needs to create a container with system-level tools availa --- -### User Story 2 - Build Container with APT Packages via Requirements File (Priority: P2) - -A user wants to define their APT package dependencies in a file (similar to conda's environment.yml) and have Wave build a container from that file specification. This allows version control of dependencies and easier sharing of environment definitions. - -**Why this priority**: File-based specification is important for reproducibility and CI/CD workflows, but users can achieve basic functionality with package lists first. - -**Independent Test**: Can be tested by submitting a request with a base64-encoded requirements file containing APT package names (one per line), then verifying the container has all listed packages installed. - -**Acceptance Scenarios**: - -1. **Given** a user submits a container token request with `buildTemplate: "apt/debian:v1"` and an `environment` field containing a base64-encoded file with package names, **When** Wave processes the request, **Then** a container is built with all packages from the file installed. - -2. **Given** a requirements file contains comments (lines starting with #) and empty lines, **When** Wave parses the file, **Then** comments and empty lines are ignored and only valid package names are processed. - ---- - -### User Story 3 - Customize APT Build with Base Image and Additional Commands (Priority: P3) +### User Story 2 - Customize APT Build with Base Image and Additional Commands (Priority: P2) An advanced user needs to customize their APT-based container by specifying a different base image (e.g., Ubuntu 22.04 instead of default Debian) and running additional commands after package installation (e.g., downloading additional data, setting environment variables). @@ -85,12 +69,12 @@ An advanced user needs to customize their APT-based container by specifying a di - **FR-004**: System MUST support specifying APT packages as a list of package names in the `packages.entries` field. -- **FR-005**: System MUST support specifying APT packages via a file (base64-encoded in `packages.environment` field) with one package name per line. +- **FR-005**: System MUST support specifying APT packages via a file (base64-encoded in `packages.environment` field) with one package name per line, ignoring empty lines and lines starting with `#`. - **FR-006**: System MUST support APT package version pinning using standard APT syntax (e.g., `package=version`). - **FR-007**: System MUST support an `aptOpts` configuration object in `PackagesSpec` with the following options: - - `baseImage`: Base Docker image to use (default: standard Debian-based image) + - `baseImage`: Base Docker image to use (default: ubuntu:24.04) - `basePackages`: List of packages to always install (e.g., ca-certificates) - `commands`: List of additional shell commands to run after package installation @@ -114,8 +98,6 @@ An advanced user needs to customize their APT-based container by specifying a di - **AptOpts**: New configuration entity for APT-specific build options (baseImage, basePackages, commands). -- **Requirements File**: Plain text file with one APT package name per line, supporting comments with `#` prefix. - ## Success Criteria *(mandatory)* ### Measurable Outcomes @@ -144,4 +126,4 @@ An advanced user needs to customize their APT-based container by specifying a di - The default base image is `ubuntu:24.04` (Ubuntu 24.04 LTS), providing long-term support, broad package availability, and security updates through 2029. - Only official Debian/Ubuntu repositories are supported in v1; custom repository support may be added in future versions. - Package names follow standard APT naming conventions without architecture suffixes (architecture determined by build platform). -- The requirements file format is intentionally simple (one package per line) to minimize parsing complexity and user errors. +- The `environment` field for APT is a simple newline-separated list of packages (not a structured file like Conda's environment.yml). From e98808347c6aa8b1a7e333ddc095d76f3025bfa1 Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Fri, 12 Dec 2025 16:59:32 +0100 Subject: [PATCH 3/4] Add tasks.md for APT build template implementation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Defines 31 tasks organized in 5 phases: - Phase 1: Setup (API layer types and constants) - Phase 2: Foundational (templates and helper) - Phase 3: User Story 1 (basic APT builds) - Phase 4: User Story 2 (customization) - Phase 5: Polish (docs and validation) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 --- specs/251212-apt-build-template/tasks.md | 211 +++++++++++++++++++++++ 1 file changed, 211 insertions(+) create mode 100644 specs/251212-apt-build-template/tasks.md diff --git a/specs/251212-apt-build-template/tasks.md b/specs/251212-apt-build-template/tasks.md new file mode 100644 index 000000000..0edc30cb1 --- /dev/null +++ b/specs/251212-apt-build-template/tasks.md @@ -0,0 +1,211 @@ +# Tasks: APT Build Template + +**Input**: Design documents from `/specs/251212-apt-build-template/` +**Prerequisites**: plan.md, spec.md, research.md, data-model.md, contracts/ + +**Tests**: Included per Wave constitution testing requirements (Spock 2 framework). + +**Organization**: Tasks grouped by user story to enable independent implementation and testing. + +## Format: `[ID] [P?] [Story] Description` + +- **[P]**: Can run in parallel (different files, no dependencies) +- **[Story]**: Which user story this task belongs to (e.g., US1, US2) +- Include exact file paths in descriptions + +## Path Conventions + +- **API types**: `wave-api/src/main/java/io/seqera/wave/` +- **Implementation**: `src/main/groovy/io/seqera/wave/util/` +- **Templates**: `src/main/resources/templates/` +- **Tests**: `src/test/groovy/io/seqera/wave/util/` +- **Docs**: `docs/` + +--- + +## Phase 1: Setup (API Layer) + +**Purpose**: Add new types and constants to the wave-api module + +- [ ] T001 [P] Add `APT_DEBIAN_V1 = "apt/debian:v1"` constant in wave-api/src/main/java/io/seqera/wave/api/BuildTemplate.java +- [ ] T002 [P] Add `APT` enum value to `PackagesSpec.Type` in wave-api/src/main/java/io/seqera/wave/api/PackagesSpec.java +- [ ] T003 [P] Create `AptOpts.java` configuration class in wave-api/src/main/java/io/seqera/wave/config/AptOpts.java +- [ ] T004 Add `aptOpts` field and `withAptOpts()` method to `PackagesSpec` in wave-api/src/main/java/io/seqera/wave/api/PackagesSpec.java (depends on T003) + +**Checkpoint**: API types ready - implementation can begin + +--- + +## Phase 2: Foundational (Templates & Helper) + +**Purpose**: Core infrastructure that MUST be complete before user stories can be tested + +**CRITICAL**: No user story validation can occur until templates exist + +- [ ] T005 [P] Create Dockerfile template in src/main/resources/templates/apt-debian-v1/dockerfile-apt-packages.txt +- [ ] T006 [P] Create Singularity template in src/main/resources/templates/apt-debian-v1/singularityfile-apt-packages.txt +- [ ] T007 Create `AptHelper.groovy` with `containerFile()` method in src/main/groovy/io/seqera/wave/util/AptHelper.groovy (depends on T005, T006) +- [ ] T008 Add APT dispatch logic to `containerFileFromRequest()` in src/main/groovy/io/seqera/wave/util/ContainerHelper.groovy (depends on T007) + +**Checkpoint**: Foundation ready - user story implementation can now be validated + +--- + +## Phase 3: User Story 1 - Build Container with APT Packages (Priority: P1) + +**Goal**: Users can build containers with APT packages via package list or environment file + +**Independent Test**: Submit container token request with `buildTemplate: "apt/debian:v1"` and packages `["curl", "wget", "git"]`, verify container has packages installed + +### Tests for User Story 1 + +- [ ] T009 [P] [US1] Create `AptHelperTest.groovy` with test for package list to Dockerfile in src/test/groovy/io/seqera/wave/util/AptHelperTest.groovy +- [ ] T010 [P] [US1] Add test for package list to Singularity file in src/test/groovy/io/seqera/wave/util/AptHelperTest.groovy +- [ ] T011 [P] [US1] Add test for environment file parsing (newline-separated packages) in src/test/groovy/io/seqera/wave/util/AptHelperTest.groovy +- [ ] T012 [P] [US1] Add test for version-pinned packages (e.g., `nginx=1.18.0`) in src/test/groovy/io/seqera/wave/util/AptHelperTest.groovy + +### Implementation for User Story 1 + +- [ ] T013 [US1] Implement `aptPackagesToDockerFile()` method in src/main/groovy/io/seqera/wave/util/AptHelper.groovy +- [ ] T014 [US1] Implement `aptPackagesToSingularityFile()` method in src/main/groovy/io/seqera/wave/util/AptHelper.groovy +- [ ] T015 [US1] Implement `parseEnvironmentFile()` method for newline-separated package parsing in src/main/groovy/io/seqera/wave/util/AptHelper.groovy +- [ ] T016 [US1] Verify Dockerfile template includes `DEBIAN_FRONTEND=noninteractive`, `--no-install-recommends`, and cache cleanup in src/main/resources/templates/apt-debian-v1/dockerfile-apt-packages.txt +- [ ] T017 [US1] Verify Singularity template includes equivalent best practices in src/main/resources/templates/apt-debian-v1/singularityfile-apt-packages.txt + +**Checkpoint**: User Story 1 complete - basic APT builds work for both Docker and Singularity + +--- + +## Phase 4: User Story 2 - Customize APT Build (Priority: P2) + +**Goal**: Users can customize base image, add base packages, and run additional commands + +**Independent Test**: Submit request with `aptOpts.baseImage: "ubuntu:22.04"` and `aptOpts.commands: ["echo test"]`, verify container uses custom image and ran commands + +### Tests for User Story 2 + +- [ ] T018 [P] [US2] Add test for custom baseImage in AptOpts in src/test/groovy/io/seqera/wave/util/AptHelperTest.groovy +- [ ] T019 [P] [US2] Add test for basePackages injection in src/test/groovy/io/seqera/wave/util/AptHelperTest.groovy +- [ ] T020 [P] [US2] Add test for custom commands appended to Dockerfile in src/test/groovy/io/seqera/wave/util/AptHelperTest.groovy +- [ ] T021 [P] [US2] Add test for custom commands appended to Singularity file in src/test/groovy/io/seqera/wave/util/AptHelperTest.groovy + +### Implementation for User Story 2 + +- [ ] T022 [US2] Add baseImage substitution support to template rendering in src/main/groovy/io/seqera/wave/util/AptHelper.groovy +- [ ] T023 [US2] Add basePackages injection to package list in src/main/groovy/io/seqera/wave/util/AptHelper.groovy +- [ ] T024 [US2] Add commands appending logic (RUN for Docker, %post for Singularity) in src/main/groovy/io/seqera/wave/util/AptHelper.groovy +- [ ] T025 [US2] Ensure null/empty AptOpts uses defaults (baseImage: ubuntu:24.04, basePackages: ca-certificates) in src/main/groovy/io/seqera/wave/util/AptHelper.groovy + +**Checkpoint**: User Story 2 complete - APT builds fully customizable + +--- + +## Phase 5: Polish & Cross-Cutting Concerns + +**Purpose**: Documentation, validation, and cleanup + +- [ ] T026 [P] Update docs/api.md with APT type in packages.type enum +- [ ] T027 [P] Update docs/api.md with aptOpts schema documentation +- [ ] T028 [P] Update docs/api.md with buildTemplate `apt/debian:v1` value +- [ ] T029 [P] Add APT build example (curl request) to docs/api.md +- [ ] T030 Run full test suite to verify no regressions: `./gradlew test` +- [ ] T031 Build project and verify compilation: `./gradlew assemble` + +**Checkpoint**: Feature complete and documented + +--- + +## Dependencies & Execution Order + +### Phase Dependencies + +- **Setup (Phase 1)**: No dependencies - can start immediately +- **Foundational (Phase 2)**: Depends on Phase 1 completion (T001-T004) +- **User Story 1 (Phase 3)**: Depends on Phase 2 completion (T005-T008) +- **User Story 2 (Phase 4)**: Depends on Phase 3 completion (T009-T017) +- **Polish (Phase 5)**: Can start after Phase 3 for US1 docs, complete after Phase 4 + +### User Story Dependencies + +- **User Story 1 (P1)**: Core functionality - no dependencies on other stories +- **User Story 2 (P2)**: Extends US1 with customization - depends on US1 being complete + +### Within Each Phase + +- Tests should be written and verified to fail before implementation +- API types (Phase 1) before implementation (Phase 2+) +- Templates before helper methods +- Helper before dispatch logic + +### Parallel Opportunities + +- All Phase 1 tasks (T001-T003) can run in parallel +- Templates (T005-T006) can run in parallel +- All tests within a user story can run in parallel +- All documentation tasks (T026-T029) can run in parallel + +--- + +## Parallel Example: Phase 1 Setup + +```bash +# Launch all API type tasks together: +Task: "Add APT_DEBIAN_V1 constant in wave-api/.../BuildTemplate.java" +Task: "Add APT enum value in wave-api/.../PackagesSpec.java" +Task: "Create AptOpts.java in wave-api/.../config/AptOpts.java" +``` + +## Parallel Example: User Story 1 Tests + +```bash +# Launch all US1 tests together: +Task: "Create AptHelperTest with Dockerfile test" +Task: "Add Singularity file test" +Task: "Add environment file parsing test" +Task: "Add version-pinned packages test" +``` + +--- + +## Implementation Strategy + +### MVP First (User Story 1 Only) + +1. Complete Phase 1: Setup (T001-T004) +2. Complete Phase 2: Foundational (T005-T008) +3. Complete Phase 3: User Story 1 (T009-T017) +4. **STOP and VALIDATE**: Test with `./gradlew test --tests 'AptHelperTest'` +5. Basic APT builds work - can deploy for initial feedback + +### Incremental Delivery + +1. Phase 1 + Phase 2 → Foundation ready +2. Add User Story 1 → Test → Basic APT builds work (MVP!) +3. Add User Story 2 → Test → Full customization support +4. Add Documentation → Feature complete + +--- + +## Notes + +- [P] tasks = different files, no dependencies +- [Story] label maps task to specific user story for traceability +- Follow CranHelper pattern for implementation structure +- Use TemplateRenderer for variable substitution in templates +- Commit after each logical task group +- Run `./gradlew test` frequently to catch regressions + +--- + +## Summary + +| Metric | Value | +|--------|-------| +| Total Tasks | 31 | +| Phase 1 (Setup) | 4 tasks | +| Phase 2 (Foundational) | 4 tasks | +| Phase 3 (User Story 1) | 9 tasks | +| Phase 4 (User Story 2) | 8 tasks | +| Phase 5 (Polish) | 6 tasks | +| Parallel Opportunities | 18 tasks marked [P] | +| MVP Scope | Phases 1-3 (17 tasks) | From 467cc4a25389325e215dba1b8ee2ad1440efd139 Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Fri, 12 Dec 2025 17:18:13 +0100 Subject: [PATCH 4/4] Implement APT/Debian build template (apt/debian:v1) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This adds support for building containers with APT packages as an alternative to Conda. Features include: API Layer: - Add APT_DEBIAN_V1 constant to BuildTemplate.java - Add APT type to PackagesSpec.Type enum - Create AptOpts configuration class for customization Implementation: - Create AptHelper.groovy with containerFile() method - Add Dockerfile and Singularity templates in apt-debian-v1/ - Add APT dispatch logic to ContainerHelper.groovy Features: - Package list via entries or environment file (newline-separated) - Version pinning support (e.g., nginx=1.18.0) - Customizable base image (default: ubuntu:24.04) - Customizable base packages (default: ca-certificates) - Custom commands support - Best practices: DEBIAN_FRONTEND=noninteractive, --no-install-recommends, cache cleanup Tests: - Comprehensive unit tests in AptHelperTest.groovy - Tests for Docker and Singularity output - Tests for environment file parsing - Tests for custom options Documentation: - Update docs/api.md with APT type, aptOpts, and examples 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 --- docs/api.md | 111 +++++- specs/251212-apt-build-template/tasks.md | 62 ++-- .../io/seqera/wave/util/AptHelper.groovy | 152 ++++++++ .../seqera/wave/util/ContainerHelper.groovy | 4 + .../apt-debian-v1/dockerfile-apt-packages.txt | 6 + .../singularityfile-apt-packages.txt | 8 + .../io/seqera/wave/util/AptHelperTest.groovy | 342 ++++++++++++++++++ .../io/seqera/wave/api/BuildTemplate.java | 5 + .../java/io/seqera/wave/api/PackagesSpec.java | 17 +- .../java/io/seqera/wave/config/AptOpts.java | 85 +++++ 10 files changed, 757 insertions(+), 35 deletions(-) create mode 100644 src/main/groovy/io/seqera/wave/util/AptHelper.groovy create mode 100644 src/main/resources/templates/apt-debian-v1/dockerfile-apt-packages.txt create mode 100644 src/main/resources/templates/apt-debian-v1/singularityfile-apt-packages.txt create mode 100644 src/test/groovy/io/seqera/wave/util/AptHelperTest.groovy create mode 100644 wave-api/src/main/java/io/seqera/wave/config/AptOpts.java diff --git a/docs/api.md b/docs/api.md index 41d6dc70d..164851135 100644 --- a/docs/api.md +++ b/docs/api.md @@ -211,7 +211,7 @@ Returns the name of the container request made available by Wave. | `towerWorkspaceId` | ID of the Seqera Platform workspace from where the container registry credentials are retrieved (optional). When omitted the personal workspace is used. | | `packages` | This object specifies Conda or CRAN packages environment information. | | `environment` | The package environment file encoded as a base64 string. | -| `type` | This represents the type of package builder. Use `CONDA` or `CRAN`. | +| `type` | This represents the type of package builder. Use `CONDA`, `CRAN`, or `APT`. | | `entries` | List of the packages names. | | `channels` | List of Conda channels or CRAN repositories, which will be used to download packages. | | `condaOpts` | Conda build options (when type is CONDA). | @@ -223,7 +223,11 @@ Returns the name of the container request made available by Wave. | `pixiImage` | Name of the Docker image used for Pixi package manager (e.g., `ghcr.io/prefix-dev/pixi:latest`). | | `cranOpts` | CRAN build options (when type is CRAN). | | `rImage` | Name of the R Docker image used to build CRAN containers (e.g., `rocker/r-ver:4.4.1`). | -| `buildTemplate` | The build template to use for container builds. Supported values: `conda/pixi:v1` (Pixi with multi-stage builds), `conda/micromamba:v2` (Micromamba 2.x with multi-stage builds). Default: standard conda/micromamba v1 template. | +| `aptOpts` | APT/Debian build options (when type is APT). | +| `aptOpts.baseImage` | Base Docker image for APT builds (default: `ubuntu:24.04`). | +| `aptOpts.basePackages` | Space-separated list of packages to always install before user packages (default: `ca-certificates`). | +| `aptOpts.commands` | List of additional shell commands to run after package installation. | +| `buildTemplate` | The build template to use for container builds. Supported values: `conda/pixi:v1` (Pixi with multi-stage builds), `conda/micromamba:v2` (Micromamba 2.x with multi-stage builds), `apt/debian:v1` (APT/Debian package builds). Default: standard conda/micromamba v1 template. | | `nameStrategy` | The name strategy to be used to create the name of the container built by Wave. Its values can be `none`, `tagPrefix`, or `imageSuffix`. | | #### Response @@ -391,6 +395,109 @@ Returns the name of the container request made available by Wave. } ``` +- Create Docker image with APT packages: + + **Request** + + ```shell + curl --location 'https://wave.seqera.io/v1alpha2/container' \ + --header 'Content-Type: application/json' \ + --data '{ + "buildTemplate": "apt/debian:v1", + "packages":{ + "type": "APT", + "entries": ["curl", "wget", "git", "build-essential"] + } + }' + ``` + + **Response** + + ```json + { + "requestId": "abc123def456", + "containerToken": "abc123def456", + "targetImage": "wave.seqera.io/wt/abc123def456/wave/build:1234567890abcdef", + "expiration": "2025-12-12T12:00:00.000000Z", + "containerImage": "private.cr.seqera.io/wave/build:1234567890abcdef", + "buildId": "bd-1234567890abcdef_1", + "cached": false, + "freeze": false, + "mirror": false + } + ``` + +- Create Docker image with APT packages and custom options: + + **Request** + + ```shell + curl --location 'https://wave.seqera.io/v1alpha2/container' \ + --header 'Content-Type: application/json' \ + --data '{ + "buildTemplate": "apt/debian:v1", + "packages":{ + "type": "APT", + "entries": ["nginx"], + "aptOpts": { + "baseImage": "ubuntu:22.04", + "basePackages": "ca-certificates locales", + "commands": ["locale-gen en_US.UTF-8"] + } + } + }' + ``` + + **Response** + + ```json + { + "requestId": "xyz789abc012", + "containerToken": "xyz789abc012", + "targetImage": "wave.seqera.io/wt/xyz789abc012/wave/build:fedcba0987654321", + "expiration": "2025-12-12T12:00:00.000000Z", + "containerImage": "private.cr.seqera.io/wave/build:fedcba0987654321", + "buildId": "bd-fedcba0987654321_1", + "cached": false, + "freeze": false, + "mirror": false + } + ``` + +- Create Singularity image with APT packages: + + **Request** + + ```shell + curl --location 'https://wave.seqera.io/v1alpha2/container' \ + --header 'Content-Type: application/json' \ + --data '{ + "format": "sif", + "buildTemplate": "apt/debian:v1", + "packages":{ + "type": "APT", + "entries": ["samtools", "bcftools"] + }, + "freeze": true, + "buildRepository": "", + "towerAccessToken": "" + }' + ``` + + **Response** + + ```json + { + "requestId": "def456ghi789", + "targetImage": "oras://:abcd1234efgh5678", + "containerImage": "oras://:abcd1234efgh5678", + "buildId": "bd-abcd1234efgh5678_1", + "freeze": true, + "mirror": false, + "succeeded": true + } + ``` + ### GET `/v1alpha1/builds/{buildId}/status` Get status of build against `buildId` passed as path variable diff --git a/specs/251212-apt-build-template/tasks.md b/specs/251212-apt-build-template/tasks.md index 0edc30cb1..b531f108a 100644 --- a/specs/251212-apt-build-template/tasks.md +++ b/specs/251212-apt-build-template/tasks.md @@ -27,10 +27,10 @@ **Purpose**: Add new types and constants to the wave-api module -- [ ] T001 [P] Add `APT_DEBIAN_V1 = "apt/debian:v1"` constant in wave-api/src/main/java/io/seqera/wave/api/BuildTemplate.java -- [ ] T002 [P] Add `APT` enum value to `PackagesSpec.Type` in wave-api/src/main/java/io/seqera/wave/api/PackagesSpec.java -- [ ] T003 [P] Create `AptOpts.java` configuration class in wave-api/src/main/java/io/seqera/wave/config/AptOpts.java -- [ ] T004 Add `aptOpts` field and `withAptOpts()` method to `PackagesSpec` in wave-api/src/main/java/io/seqera/wave/api/PackagesSpec.java (depends on T003) +- [X] T001 [P] Add `APT_DEBIAN_V1 = "apt/debian:v1"` constant in wave-api/src/main/java/io/seqera/wave/api/BuildTemplate.java +- [X] T002 [P] Add `APT` enum value to `PackagesSpec.Type` in wave-api/src/main/java/io/seqera/wave/api/PackagesSpec.java +- [X] T003 [P] Create `AptOpts.java` configuration class in wave-api/src/main/java/io/seqera/wave/config/AptOpts.java +- [X] T004 Add `aptOpts` field and `withAptOpts()` method to `PackagesSpec` in wave-api/src/main/java/io/seqera/wave/api/PackagesSpec.java (depends on T003) **Checkpoint**: API types ready - implementation can begin @@ -42,10 +42,10 @@ **CRITICAL**: No user story validation can occur until templates exist -- [ ] T005 [P] Create Dockerfile template in src/main/resources/templates/apt-debian-v1/dockerfile-apt-packages.txt -- [ ] T006 [P] Create Singularity template in src/main/resources/templates/apt-debian-v1/singularityfile-apt-packages.txt -- [ ] T007 Create `AptHelper.groovy` with `containerFile()` method in src/main/groovy/io/seqera/wave/util/AptHelper.groovy (depends on T005, T006) -- [ ] T008 Add APT dispatch logic to `containerFileFromRequest()` in src/main/groovy/io/seqera/wave/util/ContainerHelper.groovy (depends on T007) +- [X] T005 [P] Create Dockerfile template in src/main/resources/templates/apt-debian-v1/dockerfile-apt-packages.txt +- [X] T006 [P] Create Singularity template in src/main/resources/templates/apt-debian-v1/singularityfile-apt-packages.txt +- [X] T007 Create `AptHelper.groovy` with `containerFile()` method in src/main/groovy/io/seqera/wave/util/AptHelper.groovy (depends on T005, T006) +- [X] T008 Add APT dispatch logic to `containerFileFromRequest()` in src/main/groovy/io/seqera/wave/util/ContainerHelper.groovy (depends on T007) **Checkpoint**: Foundation ready - user story implementation can now be validated @@ -59,18 +59,18 @@ ### Tests for User Story 1 -- [ ] T009 [P] [US1] Create `AptHelperTest.groovy` with test for package list to Dockerfile in src/test/groovy/io/seqera/wave/util/AptHelperTest.groovy -- [ ] T010 [P] [US1] Add test for package list to Singularity file in src/test/groovy/io/seqera/wave/util/AptHelperTest.groovy -- [ ] T011 [P] [US1] Add test for environment file parsing (newline-separated packages) in src/test/groovy/io/seqera/wave/util/AptHelperTest.groovy -- [ ] T012 [P] [US1] Add test for version-pinned packages (e.g., `nginx=1.18.0`) in src/test/groovy/io/seqera/wave/util/AptHelperTest.groovy +- [X] T009 [P] [US1] Create `AptHelperTest.groovy` with test for package list to Dockerfile in src/test/groovy/io/seqera/wave/util/AptHelperTest.groovy +- [X] T010 [P] [US1] Add test for package list to Singularity file in src/test/groovy/io/seqera/wave/util/AptHelperTest.groovy +- [X] T011 [P] [US1] Add test for environment file parsing (newline-separated packages) in src/test/groovy/io/seqera/wave/util/AptHelperTest.groovy +- [X] T012 [P] [US1] Add test for version-pinned packages (e.g., `nginx=1.18.0`) in src/test/groovy/io/seqera/wave/util/AptHelperTest.groovy ### Implementation for User Story 1 -- [ ] T013 [US1] Implement `aptPackagesToDockerFile()` method in src/main/groovy/io/seqera/wave/util/AptHelper.groovy -- [ ] T014 [US1] Implement `aptPackagesToSingularityFile()` method in src/main/groovy/io/seqera/wave/util/AptHelper.groovy -- [ ] T015 [US1] Implement `parseEnvironmentFile()` method for newline-separated package parsing in src/main/groovy/io/seqera/wave/util/AptHelper.groovy -- [ ] T016 [US1] Verify Dockerfile template includes `DEBIAN_FRONTEND=noninteractive`, `--no-install-recommends`, and cache cleanup in src/main/resources/templates/apt-debian-v1/dockerfile-apt-packages.txt -- [ ] T017 [US1] Verify Singularity template includes equivalent best practices in src/main/resources/templates/apt-debian-v1/singularityfile-apt-packages.txt +- [X] T013 [US1] Implement `aptPackagesToDockerFile()` method in src/main/groovy/io/seqera/wave/util/AptHelper.groovy +- [X] T014 [US1] Implement `aptPackagesToSingularityFile()` method in src/main/groovy/io/seqera/wave/util/AptHelper.groovy +- [X] T015 [US1] Implement `parseEnvironmentFile()` method for newline-separated package parsing in src/main/groovy/io/seqera/wave/util/AptHelper.groovy +- [X] T016 [US1] Verify Dockerfile template includes `DEBIAN_FRONTEND=noninteractive`, `--no-install-recommends`, and cache cleanup in src/main/resources/templates/apt-debian-v1/dockerfile-apt-packages.txt +- [X] T017 [US1] Verify Singularity template includes equivalent best practices in src/main/resources/templates/apt-debian-v1/singularityfile-apt-packages.txt **Checkpoint**: User Story 1 complete - basic APT builds work for both Docker and Singularity @@ -84,17 +84,17 @@ ### Tests for User Story 2 -- [ ] T018 [P] [US2] Add test for custom baseImage in AptOpts in src/test/groovy/io/seqera/wave/util/AptHelperTest.groovy -- [ ] T019 [P] [US2] Add test for basePackages injection in src/test/groovy/io/seqera/wave/util/AptHelperTest.groovy -- [ ] T020 [P] [US2] Add test for custom commands appended to Dockerfile in src/test/groovy/io/seqera/wave/util/AptHelperTest.groovy -- [ ] T021 [P] [US2] Add test for custom commands appended to Singularity file in src/test/groovy/io/seqera/wave/util/AptHelperTest.groovy +- [X] T018 [P] [US2] Add test for custom baseImage in AptOpts in src/test/groovy/io/seqera/wave/util/AptHelperTest.groovy +- [X] T019 [P] [US2] Add test for basePackages injection in src/test/groovy/io/seqera/wave/util/AptHelperTest.groovy +- [X] T020 [P] [US2] Add test for custom commands appended to Dockerfile in src/test/groovy/io/seqera/wave/util/AptHelperTest.groovy +- [X] T021 [P] [US2] Add test for custom commands appended to Singularity file in src/test/groovy/io/seqera/wave/util/AptHelperTest.groovy ### Implementation for User Story 2 -- [ ] T022 [US2] Add baseImage substitution support to template rendering in src/main/groovy/io/seqera/wave/util/AptHelper.groovy -- [ ] T023 [US2] Add basePackages injection to package list in src/main/groovy/io/seqera/wave/util/AptHelper.groovy -- [ ] T024 [US2] Add commands appending logic (RUN for Docker, %post for Singularity) in src/main/groovy/io/seqera/wave/util/AptHelper.groovy -- [ ] T025 [US2] Ensure null/empty AptOpts uses defaults (baseImage: ubuntu:24.04, basePackages: ca-certificates) in src/main/groovy/io/seqera/wave/util/AptHelper.groovy +- [X] T022 [US2] Add baseImage substitution support to template rendering in src/main/groovy/io/seqera/wave/util/AptHelper.groovy +- [X] T023 [US2] Add basePackages injection to package list in src/main/groovy/io/seqera/wave/util/AptHelper.groovy +- [X] T024 [US2] Add commands appending logic (RUN for Docker, %post for Singularity) in src/main/groovy/io/seqera/wave/util/AptHelper.groovy +- [X] T025 [US2] Ensure null/empty AptOpts uses defaults (baseImage: ubuntu:24.04, basePackages: ca-certificates) in src/main/groovy/io/seqera/wave/util/AptHelper.groovy **Checkpoint**: User Story 2 complete - APT builds fully customizable @@ -104,12 +104,12 @@ **Purpose**: Documentation, validation, and cleanup -- [ ] T026 [P] Update docs/api.md with APT type in packages.type enum -- [ ] T027 [P] Update docs/api.md with aptOpts schema documentation -- [ ] T028 [P] Update docs/api.md with buildTemplate `apt/debian:v1` value -- [ ] T029 [P] Add APT build example (curl request) to docs/api.md -- [ ] T030 Run full test suite to verify no regressions: `./gradlew test` -- [ ] T031 Build project and verify compilation: `./gradlew assemble` +- [X] T026 [P] Update docs/api.md with APT type in packages.type enum +- [X] T027 [P] Update docs/api.md with aptOpts schema documentation +- [X] T028 [P] Update docs/api.md with buildTemplate `apt/debian:v1` value +- [X] T029 [P] Add APT build example (curl request) to docs/api.md +- [X] T030 Run full test suite to verify no regressions: `./gradlew test` +- [X] T031 Build project and verify compilation: `./gradlew assemble` **Checkpoint**: Feature complete and documented diff --git a/src/main/groovy/io/seqera/wave/util/AptHelper.groovy b/src/main/groovy/io/seqera/wave/util/AptHelper.groovy new file mode 100644 index 000000000..afdec53b0 --- /dev/null +++ b/src/main/groovy/io/seqera/wave/util/AptHelper.groovy @@ -0,0 +1,152 @@ +/* + * Wave, containers provisioning service + * Copyright (c) 2023-2024, Seqera Labs + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + * + * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ + +package io.seqera.wave.util + +import groovy.transform.CompileStatic +import io.seqera.wave.api.PackagesSpec +import io.seqera.wave.config.AptOpts +import io.seqera.wave.exception.BadRequestException + +/** + * Helper class to create Dockerfile for APT/Debian package manager + * + * @author Paolo Di Tommaso + */ +@CompileStatic +class AptHelper { + + /** + * Generate a container file (Dockerfile or Singularity) for APT packages. + * This is the main entry point for APT_DEBIAN_V1 template builds. + * + * @param spec The packages specification (must be APT type) + * @param singularity When true, generates Singularity format; otherwise Dockerfile + * @return The generated container file content + * @throws BadRequestException if package type is not APT + */ + static String containerFile(PackagesSpec spec, boolean singularity) { + if( spec.type != PackagesSpec.Type.APT ) { + throw new BadRequestException("Package type '${spec.type}' not supported by 'apt/debian:v1' build template") + } + + if( !spec.aptOpts ) + spec.aptOpts = new AptOpts() + + // Get packages from entries or environment + List packageList + if( spec.entries ) { + packageList = spec.entries + } + else if( spec.environment ) { + final decoded = ContainerHelper.decodeBase64OrFail(spec.environment, 'packages.environment') + packageList = parseEnvironmentFile(decoded) + } + else { + throw new BadRequestException("APT packages require either 'entries' or 'environment' field") + } + + if( packageList.isEmpty() ) { + throw new BadRequestException("APT package list cannot be empty") + } + + final String packages = packageList.join(' ') + return singularity + ? aptPackagesToSingularityFile(packages, spec.aptOpts) + : aptPackagesToDockerFile(packages, spec.aptOpts) + } + + /** + * Parse environment file content into list of packages. + * Format: one package per line, comments start with #, empty lines ignored. + * + * @param content The environment file content (newline-separated packages) + * @return List of package names + */ + static List parseEnvironmentFile(String content) { + if( !content ) + return [] + + return content.split('\n') + .collect { it.trim() } + .findAll { it && !it.startsWith('#') } + } + + /** + * Generate Dockerfile for APT packages + */ + static String aptPackagesToDockerFile(String packages, AptOpts opts) { + return aptPackagesTemplate0( + '/templates/apt-debian-v1/dockerfile-apt-packages.txt', + packages, + opts) + } + + /** + * Generate Singularity file for APT packages + */ + static String aptPackagesToSingularityFile(String packages, AptOpts opts) { + return aptPackagesTemplate0( + '/templates/apt-debian-v1/singularityfile-apt-packages.txt', + packages, + opts) + } + + protected static String aptPackagesTemplate0(String template, String packages, AptOpts opts) { + final boolean singularity = template.contains('/singularityfile') + final String baseImage = opts.baseImage ?: AptOpts.DEFAULT_BASE_IMAGE + final String basePackages = opts.basePackages ?: AptOpts.DEFAULT_BASE_PACKAGES + + final Map binding = [:] + binding.put('base_image', baseImage) + binding.put('base_packages', basePackages) + binding.put('target', packages) + + final String result = renderTemplate0(template, binding) + return addCommands(result, opts.commands, singularity) + } + + private static String renderTemplate0(String templatePath, Map binding) { + final URL template = AptHelper.class.getResource(templatePath) + if( template == null ) + throw new IllegalStateException("Unable to load template '${templatePath}' from classpath") + try { + final InputStream reader = template.openStream() + return new TemplateRenderer().render(reader, binding) + } + catch( IOException e ) { + throw new IllegalStateException("Unable to read classpath template '${templatePath}'", e) + } + } + + private static String addCommands(String result, List commands, boolean singularity) { + if( commands == null || commands.isEmpty() ) + return result + if( singularity ) + result += '%post\n' + for( String cmd : commands ) { + if( singularity ) { + result += ' ' + cmd + '\n' + } + else { + result += 'RUN ' + cmd + '\n' + } + } + return result + } +} diff --git a/src/main/groovy/io/seqera/wave/util/ContainerHelper.groovy b/src/main/groovy/io/seqera/wave/util/ContainerHelper.groovy index ed0b55609..c02f00711 100644 --- a/src/main/groovy/io/seqera/wave/util/ContainerHelper.groovy +++ b/src/main/groovy/io/seqera/wave/util/ContainerHelper.groovy @@ -33,6 +33,7 @@ import io.seqera.wave.service.builder.BuildFormat import io.seqera.wave.service.request.ContainerRequest import io.seqera.wave.service.request.TokenData import org.yaml.snakeyaml.Yaml +import static io.seqera.wave.api.BuildTemplate.APT_DEBIAN_V1 import static io.seqera.wave.api.BuildTemplate.CONDA_MICROMAMBA_V1 import static io.seqera.wave.api.BuildTemplate.CONDA_MICROMAMBA_V2 import static io.seqera.wave.api.BuildTemplate.CONDA_PIXI_V1 @@ -73,6 +74,9 @@ class ContainerHelper { if( spec.type == PackagesSpec.Type.CRAN && (!req.buildTemplate || req.buildTemplate == CRAN_INSTALLR_V1) ) { return CranHelper.containerFile(spec, singularity) } + if( spec.type == PackagesSpec.Type.APT && req.buildTemplate == APT_DEBIAN_V1 ) { + return AptHelper.containerFile(spec, singularity) + } throw new BadRequestException("Unexpected or missing package type '${spec?.type?:'-'}' or build template '${req.buildTemplate?:'-'}'") } diff --git a/src/main/resources/templates/apt-debian-v1/dockerfile-apt-packages.txt b/src/main/resources/templates/apt-debian-v1/dockerfile-apt-packages.txt new file mode 100644 index 000000000..122a75a3f --- /dev/null +++ b/src/main/resources/templates/apt-debian-v1/dockerfile-apt-packages.txt @@ -0,0 +1,6 @@ +FROM {{base_image}} +ENV DEBIAN_FRONTEND=noninteractive +RUN apt-get update \ + && apt-get install -y --no-install-recommends {{base_packages}} {{target}} \ + && apt-get clean \ + && rm -rf /var/lib/apt/lists/* diff --git a/src/main/resources/templates/apt-debian-v1/singularityfile-apt-packages.txt b/src/main/resources/templates/apt-debian-v1/singularityfile-apt-packages.txt new file mode 100644 index 000000000..e73f3514a --- /dev/null +++ b/src/main/resources/templates/apt-debian-v1/singularityfile-apt-packages.txt @@ -0,0 +1,8 @@ +BootStrap: docker +From: {{base_image}} +%post + export DEBIAN_FRONTEND=noninteractive + apt-get update + apt-get install -y --no-install-recommends {{base_packages}} {{target}} + apt-get clean + rm -rf /var/lib/apt/lists/* diff --git a/src/test/groovy/io/seqera/wave/util/AptHelperTest.groovy b/src/test/groovy/io/seqera/wave/util/AptHelperTest.groovy new file mode 100644 index 000000000..12b9fc25e --- /dev/null +++ b/src/test/groovy/io/seqera/wave/util/AptHelperTest.groovy @@ -0,0 +1,342 @@ +/* + * Wave, containers provisioning service + * Copyright (c) 2023-2024, Seqera Labs + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + * + * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ + +package io.seqera.wave.util + +import spock.lang.Specification + +import io.seqera.wave.api.PackagesSpec +import io.seqera.wave.config.AptOpts +import io.seqera.wave.exception.BadRequestException + +/** + * Tests for AptHelper + * + * @author Paolo Di Tommaso + */ +class AptHelperTest extends Specification { + + // === T009 - Test for package list to Dockerfile === + + def 'should create docker file with package list via containerFile'() { + given: + def PACKAGES = ['curl', 'wget', 'git'] + def packages = new PackagesSpec(type: PackagesSpec.Type.APT, entries: PACKAGES) + + when: + def result = AptHelper.containerFile(packages, false) + + then: + result.contains('FROM ubuntu:24.04') + result.contains('DEBIAN_FRONTEND=noninteractive') + result.contains('apt-get install -y --no-install-recommends') + result.contains('ca-certificates curl wget git') + result.contains('apt-get clean') + result.contains('rm -rf /var/lib/apt/lists/*') + } + + // === T010 - Test for package list to Singularity file === + + def 'should create singularity file with package list via containerFile'() { + given: + def PACKAGES = ['samtools', 'bcftools'] + def packages = new PackagesSpec(type: PackagesSpec.Type.APT, entries: PACKAGES) + + when: + def result = AptHelper.containerFile(packages, true) + + then: + result.contains('BootStrap: docker') + result.contains('From: ubuntu:24.04') + result.contains('export DEBIAN_FRONTEND=noninteractive') + result.contains('apt-get install -y --no-install-recommends') + result.contains('ca-certificates samtools bcftools') + result.contains('apt-get clean') + result.contains('rm -rf /var/lib/apt/lists/*') + } + + // === T011 - Test for environment file parsing === + + def 'should parse environment file content'() { + expect: + AptHelper.parseEnvironmentFile(CONTENT) == EXPECTED + + where: + CONTENT | EXPECTED + null | [] + '' | [] + 'curl' | ['curl'] + 'curl\nwget' | ['curl', 'wget'] + 'curl\nwget\ngit' | ['curl', 'wget', 'git'] + '# comment\ncurl' | ['curl'] + 'curl\n# comment\nwget' | ['curl', 'wget'] + ' curl \n wget ' | ['curl', 'wget'] + 'curl\n\nwget' | ['curl', 'wget'] + '# System utilities\ncurl\nwget\n\n# Build tools\nbuild-essential' | ['curl', 'wget', 'build-essential'] + } + + def 'should create docker file from environment file via containerFile'() { + given: + def envContent = '''\ + # System utilities + curl + wget + git + '''.stripIndent() + def encoded = Base64.encoder.encodeToString(envContent.bytes) + def packages = new PackagesSpec(type: PackagesSpec.Type.APT, environment: encoded) + + when: + def result = AptHelper.containerFile(packages, false) + + then: + result.contains('FROM ubuntu:24.04') + result.contains('apt-get install -y --no-install-recommends') + result.contains('ca-certificates curl wget git') + } + + // === T012 - Test for version-pinned packages === + + def 'should handle version-pinned packages'() { + given: + def PACKAGES = ['nginx=1.18.0-0ubuntu1', 'curl'] + def packages = new PackagesSpec(type: PackagesSpec.Type.APT, entries: PACKAGES) + + when: + def result = AptHelper.containerFile(packages, false) + + then: + result.contains('nginx=1.18.0-0ubuntu1 curl') + } + + // === Additional tests for containerFile === + + def 'should use default AptOpts when not provided'() { + given: + def PACKAGES = ['curl', 'wget'] + def packages = new PackagesSpec(type: PackagesSpec.Type.APT, entries: PACKAGES) + + when: + def result = AptHelper.containerFile(packages, false) + + then: + // Should use default base image from AptOpts + result.contains('FROM ubuntu:24.04') + // Should use default base packages + result.contains('ca-certificates') + } + + def 'should throw exception for non-APT package type'() { + given: + def packages = new PackagesSpec(type: PackagesSpec.Type.CONDA, entries: ['bwa=0.7.15']) + + when: + AptHelper.containerFile(packages, false) + + then: + def ex = thrown(BadRequestException) + ex.message.contains("not supported by 'apt/debian:v1' build template") + } + + def 'should throw exception when no packages provided'() { + given: + def packages = new PackagesSpec(type: PackagesSpec.Type.APT) + + when: + AptHelper.containerFile(packages, false) + + then: + def ex = thrown(BadRequestException) + ex.message.contains("require either 'entries' or 'environment' field") + } + + def 'should throw exception for empty package list'() { + given: + def packages = new PackagesSpec(type: PackagesSpec.Type.APT, entries: []) + + when: + AptHelper.containerFile(packages, false) + + then: + def ex = thrown(BadRequestException) + // Empty list is treated as no entries in Groovy (falsy), so we get this error + ex.message.contains("require either 'entries' or 'environment' field") + } + + // === Low-level helper tests === + + def 'should create complete dockerfile for apt packages'() { + given: + def APT_OPTS = new AptOpts([baseImage: 'ubuntu:22.04', basePackages: 'ca-certificates']) + def PACKAGES = 'curl wget git' + + when: + def result = AptHelper.aptPackagesToDockerFile(PACKAGES, APT_OPTS) + + then: + result == '''\ + FROM ubuntu:22.04 + ENV DEBIAN_FRONTEND=noninteractive + RUN apt-get update \\ + && apt-get install -y --no-install-recommends ca-certificates curl wget git \\ + && apt-get clean \\ + && rm -rf /var/lib/apt/lists/* + '''.stripIndent(true) + } + + def 'should create complete singularity file for apt packages'() { + given: + def APT_OPTS = new AptOpts([baseImage: 'ubuntu:22.04', basePackages: 'ca-certificates']) + def PACKAGES = 'curl wget' + + when: + def result = AptHelper.aptPackagesToSingularityFile(PACKAGES, APT_OPTS) + + then: + result == '''\ + BootStrap: docker + From: ubuntu:22.04 + %post + export DEBIAN_FRONTEND=noninteractive + apt-get update + apt-get install -y --no-install-recommends ca-certificates curl wget + apt-get clean + rm -rf /var/lib/apt/lists/* + '''.stripIndent(true) + } + + // === US2 Tests - Custom options === + + def 'should use custom baseImage in AptOpts'() { + given: + def APT_OPTS = new AptOpts([baseImage: 'debian:bookworm']) + def PACKAGES = ['curl'] + def packages = new PackagesSpec(type: PackagesSpec.Type.APT, entries: PACKAGES, aptOpts: APT_OPTS) + + when: + def result = AptHelper.containerFile(packages, false) + + then: + result.contains('FROM debian:bookworm') + !result.contains('FROM ubuntu:24.04') + } + + def 'should use custom basePackages in AptOpts'() { + given: + def APT_OPTS = new AptOpts([basePackages: 'ca-certificates locales']) + def PACKAGES = ['curl'] + def packages = new PackagesSpec(type: PackagesSpec.Type.APT, entries: PACKAGES, aptOpts: APT_OPTS) + + when: + def result = AptHelper.containerFile(packages, false) + + then: + result.contains('ca-certificates locales curl') + } + + def 'should add custom commands to Dockerfile'() { + given: + def APT_OPTS = new AptOpts([commands: ['echo "setup complete"', 'locale-gen en_US.UTF-8']]) + def PACKAGES = ['curl'] + def packages = new PackagesSpec(type: PackagesSpec.Type.APT, entries: PACKAGES, aptOpts: APT_OPTS) + + when: + def result = AptHelper.containerFile(packages, false) + + then: + result.contains('RUN echo "setup complete"') + result.contains('RUN locale-gen en_US.UTF-8') + } + + def 'should add custom commands to Singularity file'() { + given: + def APT_OPTS = new AptOpts([commands: ['echo "setup complete"', 'locale-gen en_US.UTF-8']]) + def PACKAGES = ['curl'] + def packages = new PackagesSpec(type: PackagesSpec.Type.APT, entries: PACKAGES, aptOpts: APT_OPTS) + + when: + def result = AptHelper.containerFile(packages, true) + + then: + result.contains('%post') + result.contains(' echo "setup complete"') + result.contains(' locale-gen en_US.UTF-8') + } + + def 'should handle null/empty AptOpts with defaults'() { + given: + def packages = new PackagesSpec(type: PackagesSpec.Type.APT, entries: ['curl']) + + when: + def result = AptHelper.containerFile(packages, false) + + then: + // Default base image + result.contains('FROM ubuntu:24.04') + // Default base packages + result.contains('ca-certificates') + } + + def 'should handle AptOpts with empty basePackages'() { + given: + def APT_OPTS = new AptOpts() + APT_OPTS.basePackages = '' + def PACKAGES = ['curl'] + def packages = new PackagesSpec(type: PackagesSpec.Type.APT, entries: PACKAGES, aptOpts: APT_OPTS) + + when: + def result = AptHelper.containerFile(packages, false) + + then: + // With empty basePackages, we get a leading space before the target package + result.contains('apt-get install -y --no-install-recommends') + result.contains('curl') + } + + def 'should handle multiple packages in complete output'() { + given: + def APT_OPTS = new AptOpts([baseImage: 'ubuntu:24.04', basePackages: 'ca-certificates']) + def PACKAGES = ['curl', 'wget', 'git', 'build-essential'] + def packages = new PackagesSpec(type: PackagesSpec.Type.APT, entries: PACKAGES, aptOpts: APT_OPTS) + + when: + def dockerResult = AptHelper.containerFile(packages, false) + def singularityResult = AptHelper.containerFile(packages, true) + + then: + dockerResult == '''\ + FROM ubuntu:24.04 + ENV DEBIAN_FRONTEND=noninteractive + RUN apt-get update \\ + && apt-get install -y --no-install-recommends ca-certificates curl wget git build-essential \\ + && apt-get clean \\ + && rm -rf /var/lib/apt/lists/* + '''.stripIndent(true) + + singularityResult == '''\ + BootStrap: docker + From: ubuntu:24.04 + %post + export DEBIAN_FRONTEND=noninteractive + apt-get update + apt-get install -y --no-install-recommends ca-certificates curl wget git build-essential + apt-get clean + rm -rf /var/lib/apt/lists/* + '''.stripIndent(true) + } +} diff --git a/wave-api/src/main/java/io/seqera/wave/api/BuildTemplate.java b/wave-api/src/main/java/io/seqera/wave/api/BuildTemplate.java index 9e954fe29..b391175a2 100644 --- a/wave-api/src/main/java/io/seqera/wave/api/BuildTemplate.java +++ b/wave-api/src/main/java/io/seqera/wave/api/BuildTemplate.java @@ -44,6 +44,11 @@ public final class BuildTemplate { */ public static final String CRAN_INSTALLR_V1 = "cran/installr:v1"; + /** + * Build template for APT/Debian package builds + */ + public static final String APT_DEBIAN_V1 = "apt/debian:v1"; + private BuildTemplate() { // Prevent instantiation } diff --git a/wave-api/src/main/java/io/seqera/wave/api/PackagesSpec.java b/wave-api/src/main/java/io/seqera/wave/api/PackagesSpec.java index 4645fd695..973ba4264 100644 --- a/wave-api/src/main/java/io/seqera/wave/api/PackagesSpec.java +++ b/wave-api/src/main/java/io/seqera/wave/api/PackagesSpec.java @@ -20,6 +20,7 @@ import java.util.List; import java.util.Objects; +import io.seqera.wave.config.AptOpts; import io.seqera.wave.config.CondaOpts; import io.seqera.wave.config.CranOpts; import io.seqera.wave.config.PixiOpts; @@ -31,7 +32,7 @@ */ public class PackagesSpec { - public enum Type { CONDA, CRAN } + public enum Type { CONDA, CRAN, APT } public Type type; @@ -60,6 +61,11 @@ public enum Type { CONDA, CRAN } */ public CranOpts cranOpts; + /** + * APT/Debian build options + */ + public AptOpts aptOpts; + /** * channels used for downloading packages */ @@ -76,12 +82,13 @@ public boolean equals(Object object) { && Objects.equals(condaOpts, that.condaOpts) && Objects.equals(pixiOpts, that.pixiOpts) && Objects.equals(cranOpts, that.cranOpts) + && Objects.equals(aptOpts, that.aptOpts) && Objects.equals(channels, that.channels); } @Override public int hashCode() { - return Objects.hash(type, environment, entries, condaOpts, cranOpts, channels); + return Objects.hash(type, environment, entries, condaOpts, cranOpts, aptOpts, channels); } @Override @@ -93,6 +100,7 @@ public String toString() { ", condaOpts=" + condaOpts + ", cranOpts=" + cranOpts + ", pixiOpts=" + pixiOpts + + ", aptOpts=" + aptOpts + ", channels=" + ObjectUtils.toString(channels) + '}'; } @@ -132,4 +140,9 @@ public PackagesSpec withCranOpts(CranOpts opts) { return this; } + public PackagesSpec withAptOpts(AptOpts opts) { + this.aptOpts = opts; + return this; + } + } diff --git a/wave-api/src/main/java/io/seqera/wave/config/AptOpts.java b/wave-api/src/main/java/io/seqera/wave/config/AptOpts.java new file mode 100644 index 000000000..8f5f6c0a4 --- /dev/null +++ b/wave-api/src/main/java/io/seqera/wave/config/AptOpts.java @@ -0,0 +1,85 @@ +/* + * Copyright 2025, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package io.seqera.wave.config; + +import java.util.List; +import java.util.Map; +import java.util.Objects; + +/** + * APT/Debian build options + * + * @author Paolo Di Tommaso + */ +public class AptOpts { + public static final String DEFAULT_BASE_IMAGE = "ubuntu:24.04"; + public static final String DEFAULT_BASE_PACKAGES = "ca-certificates"; + + public String baseImage; + public String basePackages; + public List commands; + + public AptOpts() { + this(Map.of()); + } + + public AptOpts(Map opts) { + this.baseImage = opts.containsKey("baseImage") ? opts.get("baseImage").toString() : DEFAULT_BASE_IMAGE; + this.basePackages = opts.containsKey("basePackages") ? opts.get("basePackages").toString() : DEFAULT_BASE_PACKAGES; + this.commands = opts.containsKey("commands") ? (List) opts.get("commands") : null; + } + + public AptOpts withBaseImage(String value) { + this.baseImage = value; + return this; + } + + public AptOpts withBasePackages(String value) { + this.basePackages = value; + return this; + } + + public AptOpts withCommands(List value) { + this.commands = value; + return this; + } + + @Override + public String toString() { + return String.format("AptOpts(baseImage=%s; basePackages=%s, commands=%s)", + baseImage, + basePackages, + commands != null ? String.join(",", commands) : "null" + ); + } + + @Override + public boolean equals(Object object) { + if (this == object) return true; + if (object == null || getClass() != object.getClass()) return false; + AptOpts aptOpts = (AptOpts) object; + return Objects.equals(baseImage, aptOpts.baseImage) + && Objects.equals(basePackages, aptOpts.basePackages) + && Objects.equals(commands, aptOpts.commands); + } + + @Override + public int hashCode() { + return Objects.hash(baseImage, basePackages, commands); + } +}