`nextstrain run`: Define available workflows in the `nextstrain-pathogen.yaml`

[joverlee521]

Context

Currently it's unclear what workflows are available for a pathogen unless you go to the pathogen repo (or in the future, pathogen docs). We can improve the discoverability of the workflows in a pathogen repo by defining the workflows in nextstrain-pathogen.yaml. This fits in with the idea of having per-workflow compatibility registration suggested in nextstrain/seasonal-flu#226 (comment).

Examples

For most pathogens (e.g. measles), this would just list the 3 standard workflows

---
compatibility:
  nextstrain run:
    - ingest
    - phylogenetic
    - nextclade

For pathogens with custom workflows (e.g. avian-flu), this would list the custom workflows

---
compatibility:
  nextstrain run:
    - genome-focused
    - segment-focused

Then we can add a new command or extend nextstrain version --pathogens that would list these available workflows.

$ nextstrain version --pathogens
Nextstrain CLI x.x.x (standalone)

Pathogens
  avian-flu (default)
    avian-flu@master
      genome-focused
      segment-focused
  measles
    measles@main (default)
      ingest
      phylogenetic
      nextclade 

Additional feature to define workflow Snakefile/config file

If we explicitly list out the workflows in the nextstrain-pathogen.yaml, I think this also opens the door to defining the location of the Snakfile and/or config file for the workflow. This is mainly inspire by my current work in mpox to update the phylo workflow support nextstrain run.

Instead of creating 4 extra directories/Snakefile per build (as shown in nextstrain/mpox#327), the phylogenetic directory structure could remain the same. The nextstrain-pathogen.yaml would define the Snakefile and config file used per workflow.

---
compatibility:
  nextstrain run:
    ingest:
        snakefile: ~ # Set to null to use the default ingest/Snakefile
        configfile: ~ # Set to null to use the default config ingest/defaults/config.yaml
    phylogenetic/all-clades:
        snakefile: phylogenetic/Snakefile
        configfile: phylogenetic/defaults/mpxv/config.yaml
    phylogenetic/clade-I: 
        snakefile: phylogenetic/Snakefile
        configfile: phylogenetic/defaults/clade-i/config.yaml
    phylogenetic/clade-IIb: 
        snakefile: phylogenetic/Snakefile
        configfile: phylogenetic/defaults/hmpxv1/config.yaml
    phylogenetic/lineage-B.1:
        snakefile: phylogenetic/Snakefile
        configfile: phylogenetic/deafults/hmpxv1_big/config.yaml

[jameshadfield]

This seems worth prototyping! I forget if there's any difference between supplying --configfile <default> [<overlay>] vs multiple configfile directives in the snakefile itself - hopefully they're identical. I'd also suggest not using null to represent "use the default", but that's a minor implementation detail.

[joverlee521]

Made some progress on this today register-pathogen-workflows branch.

Locally edited zika's nextstrain-pathogen.yaml to

---
compatibility:
  nextstrain run:
    - ingest
    - phylogenetic

Then symlinked local zika as a pathogen

ln -sv ~/Repos/nextstrain/zika ~/.nextstrain/pathogens/zika/local="$(echo -n local | base32)"

Here's the output of the listed workflows

$ ./bin/nextstrain version --pathogens
Nextstrain CLI 10.2.1.post1+git 

Pathogens
  measles
    measles@main (default)
      No workflows listed, please refer to pathogen docs.

  mpox
    mpox@master (default)
      No workflows listed, please refer to pathogen docs.

  seasonal-flu
    seasonal-flu@master (default)
      No workflows listed, please refer to pathogen docs.

  zika
    zika@local 
      Available workflows:
        ingest
        phylogenetic
    zika@main 
      No workflows listed, please refer to pathogen docs.