[aws batch] Structured output of AWS Batch job id for programmatic use

[tsibley]

based on nextstrain/.github#51 (comment)

We've found ourselves extracting the AWS Batch job id from nextstrain build logs using pattern matching/text manipulation. Use cases include programmatic job attachment at a later point and templating of developer messaging.

The job id is the critical piece of information, but the S3 paths for the job and maybe other bits and bobs could be useful too.

Options for emitting this information:

1. nextstrain build writes a JSON file when requested, e.g. by --aws-batch-info job.json (exact spelling TBD).

2. nextstrain build emits delimited YAML in its normal output, always, which can be easily extracted and parsed, e.g.

    Submitting job
    AWS Batch Job ID: 2ad078eb-630b-4ad2-80d0-bb898f5b6c48
    ---
    nextstrain-build-id: …
    aws-batch-job-id: …
    workdir-url: s3://…
    ...
    Watching job status
   

3. As part of [aws batch] Attach to builds no longer in the job queue #113 and/or List previously submitted AWS Batch jobs in the job queue #125, nextstrain build internally tracks job details and a new command provides subsequent access to them, e.g. nextstrain build-info most-recent or something.

(1) probably makes the most sense, especially as a starting point. (3) might be where we go eventually, but there's a lot more design considerations that we probably don't want/need to tackle now. (2) is kinda nice in some ways but annoying in others. And it's accomplishable with (1) anyway (well, assuming there's a YAML output option).

History

• pathogen-repo-build: update AWS Batch job id matching .github#51 (review)
• pathogen-repo-build: Fix extraction of AWS Batch job ids .github#126

[jameshadfield]

👍

While not regarding this specific aim, I've long pushed for jobs within GitHub Actions to use structured output, either via a file (1) or normal output (2). The most recent discussion on this (that I remember, at least) was in the context of ingest-to-phylo workflows where we use a cache hit/miss rather than structured output from a check-new-data program. I remember earlier conversations too, although not the details.

[victorlin]

Another option that was suggested years ago: Add nextstrain build --quiet option which spits out Job ID when combined with --detatch (#61)