Update "build directory" terminology in help and docs

[tsibley]

This terminology was left as-is in #355 (nextstrain-pathogen.yaml) even as it got a little more nuanced: it's no longer "the directory which was passed to nextstrain build." The term "build" has also drifted out of phase from terminology in the wider ecosystem.

Review and revise existing usages to clarify them and extend them to include the behaviour around nextstrain-pathogen.yaml.

There's a pretty good description of the behaviour in the changelog for 8.2.0:

cli/CHANGES.md

Lines 52 to 107 in 14a392e

	* `nextstrain build` and `nextstrain shell` now better support pathogen
	repositories which place workflows in subdirectories. The top-level of the
	repo must contain a `nextstrain-pathogen.yaml` file for this support to
	activate. The file may be empty for now, though we anticipate using it for
	pathogen-level metadata in the future to aid indexing, listing, and
	attribution of pathogen repos.
	As an example of the new support, consider the following repo layout
	mpox/
	├── nextstrain-pathogen.yaml
	├── ingest/
	│ ├── Snakefile
	│ └── …
	├── phylogenetic/
	│ ├── Snakefile
	│ └── …
	├── shared/
	│ ├── reference.fasta
	│ └── …
	└── …
	where `ingest/` and `phylogenetic/` contain workflows that use
	`shared/reference.fasta` via a relative path (i.e.
	`../shared/reference.fasta`).
	It's now possible to invoke those workflows with any of the following:
	nextstrain build mpox/ingest/
	nextstrain build mpox/phylogenetic/
	cd mpox
	nextstrain build ingest/
	nextstrain build phylogenetic/
	cd phylogenetic
	nextstrain build .
	nextstrain build ../ingest/
	regardless of runtime.
	Previously, such workflows required careful invocation, e.g.
	nextstrain build mpox/ -d phylogenetic/ -s phylogenetic/Snakefile
	when using runtimes with filesystem isolation (i.e. the [containerized][]
	ones; Docker, Singularity, and AWS Batch) but not when using runtimes without
	it.
	When active, this feature makes the top-level of the pathogen repo (e.g.
	`mpox/`) available in the container at `/nextstrain/build` while the
	initial working directory is set to the workflow subdirectory in the
	container (e.g. `/nextstrain/build/phylogenetic`). That is, the filesystem
	isolation boundary is drawn at the top-level of the pathogen repo instead of
	at the workflow directory (i.e. what's given to `nextstrain build`).
	([#355](https://github.com/nextstrain/cli/pull/355))

Motivated by @jameshadfield's commentary in review of a separate change.

[jameshadfield]

Here's some notes I made myself which may help others while the docs are forthcoming.

Example: we're in avian-flu running nextstrain --docker ingest.

	Without nextstrain-pathogen.yaml	With nextstrain-pathogen.yaml
/nextstrain/build is	ingest	avian-flu
Starting directory	/nextstrain/build	/nextstrain/build/ingest

If we're in avian-flu/ingest and we run nextstrain --docker . the results are identical to above.

If we're in avian-flu and we run nextstrain --docker . then the presence of nextstrain-pathogen.yaml makes no difference.

When does this matter?

If you are using relative paths which track back to a higher directory than the one your workflow is in. For instance, avian-flu/ingest/Snakemake refers to the fauna location as ../fauna (src) and the repo doesn't define nextstrain-pathogen.yaml . Running this outside of docker doesn't work, as fauna isn't typically located at avian-flu/fauna . Using nextstrain --docker ingest works as we have nextstrain/fauna and nextstrain/build is the ingest dir.

[jameshadfield]

Another push for this - today I ran into it using nextstrain/dengue, which has a nextstrain-pathogen.yaml

cd phylogenetic
nextstrain build --aws-batch ... --download 'auspice/*' .

This doesn't work because the download paths need to be relative to the top-level of the pathogen repo, i.e. --download 'phylogenetic/auspice/*'. The current docs are out-of-date, saying:

<directory> Path to pathogen build directory.

--download <pattern> ... Patterns should be relative to the build directory.

I'm happy to contribute to the docs but I'm almost certain @tsibley will want to author these parts

[tsibley]

I wonder if the best fix for this particular case is making --download (and related options) aware of the working directory vs. pathogen directory. For example, --download 'auspice/*' would internally transform auspice/* into phylogenetic/auspice/*. I think this keeps the option simpler to explain and more intuitive since paths remain relative to the directory you provide to nextstrain build (regardless of if we stop calling it the "build directory" or not).

[jameshadfield]

Yeah, relative to the provided <directory> was my expectation, so I'd support this change. (And I think this both was the situation before the recent changes related to separate analysis directories, and still is the case if we don't have a nextstrain-pathogen.yaml.)

However we still need to convey what's going on here via help/docs messages, and the different directories that will be uploaded / bound. For instance, that command will log things like

[batch] [...]   adding: phylogenetic/results/

which is confusing when you're working from within the phylogenetic directory unless you are really aware of what nextstrain-pathogen.yaml does.

[tsibley]

Yeah, relative to the provided was my expectation, so I'd support this change.

Great!

And I think this both was the situation before the recent changes related to separate analysis directories, and still is the case if we don't have a nextstrain-pathogen.yaml.

Yep, as the help still says.

Besides --download, this also affects --exclude-from-download and --exclude-from-upload. It should be doable to adjust all of them internally so that their interface is consistent regardless of nextstrain-pathogen.yaml.

However we still need to convey what's going on here via help/docs messages

For sure. That's still the remit of this issue.