Add conda-lock library to generate full conda-lock files for default template

[mashehu]

related to #973 and #971, we at nf-core have environment.ymls with pip dependencies, e.g. environment.yml and currently the micromamba templates don't include non-conda dependencies, see mamba-org/mamba#3623.

Would it be possible to export the lock file with conda-lock lock -f /tmp/environment.yml instead of the current micromamba env export --name base --explicit?

[pditommaso]

Could you please provide more context ?

[ewels]

It turns out that the conda and micromamba env export commands ignore pip packages within conda environment.yml files. For example, this module:

---
# yaml-language-server: $schema=https://raw.githubusercontent.com/nf-core/modules/master/modules/environment-schema.json
channels:
  - conda-forge
  - bioconda
dependencies:
  - bedtools=2.31.1
  - htslib=1.22.1
  - pip=25.2
  - pybedtools=0.12.0
  - pip:
      # NOTE PINTS isn't adding conda builds and is a few versions behind.
      # renovate: datasource=pypi depName=pypints
      - pypints==1.1.13

The conda-lock file contains everything for bedtools, htslib, pybedtools, but has no reference to any of the pypints package.

The Docker / Singularity images build without problem, but the conda lock-file is incomplete, meaning it breaks if we try to use it. This is a problem when we're trying to offer conda lock-files for all modules in nf-core (thankfully not that many use pip, but still).

The easiest fix is to use the separate and stand-alone conda-lock tool to generate the lock files. This does include pip packages and would solve the issue for us. However, we would need to install it into the environment so that the command works:

FROM mambaorg/micromamba:1.5.10-noble
COPY --chown=$MAMBA_USER:$MAMBA_USER conda.yml /tmp/conda.yml
RUN micromamba install -y -n base -f /tmp/conda.yml \
-   && micromamba install -y -n base conda-forge::procps-ng \
+   && micromamba install -y -n base conda-forge::procps-ng conda-forge::conda-lock \
-   && micromamba env export --name base --explicit > environment.lock \
+   && conda-lock --file /tmp/conda.yml --platform linux-64 --lockfile environment.lock \
    && echo ">> CONDA_LOCK_START" \
    && cat environment.lock \
    && echo "<< CONDA_LOCK_END" \
    && micromamba clean -a -y
USER root
ENV PATH="$MAMBA_ROOT_PREFIX/bin:$PATH"

I don't love this, as conda-lock repeats the conda environment resolution itself, so it's not 100% guaranteed that it would give the same packages as the docker image. An alternative solution that gets the best of both is to solve with conda-lock first, then use the generated lockfile in the mamba install:

FROM mambaorg/micromamba:1.5.10-noble
 COPY --chown=$MAMBA_USER:$MAMBA_USER conda.yml /tmp/conda.yml
-RUN micromamba install -y -n base -f /tmp/conda.yml \
-    && micromamba install -y -n base conda-forge::procps-ng \
-    && micromamba env export --name base --explicit > environment.lock \
+RUN micromamba install -y -n base conda-forge::conda-lock \
+    && echo "  - conda-forge::procps-ng" >> /tmp/conda.yml \
+    && conda-lock --file /tmp/conda.yml --platform linux-64 --lockfile environment.lock \
+    && conda-lock install --name base environment.lock \
     && echo ">> CONDA_LOCK_START" \
     && cat environment.lock \
     && echo "<< CONDA_LOCK_END" \
     && micromamba clean -a -y
 USER root
 ENV PATH="$MAMBA_ROOT_PREFIX/bin:$PATH"

Note that we have to append conda-forge::procps-ng to the environment.yml file which is a bit hacky, but it has to be there so that it happens all in one solve.

Warning

This is all hypothetical, I haven't tested any of the above to make sure that it actually works.

Hope that helps fill in the blanks a bit.