Skip to content

Conversation

@dustymugs
Copy link
Contributor

  • Share one instance of the AWS Batch client across all consumers.
  • Separate handling of parameters job submission into submit_jobs(). This gives greater visibility into the job submission process and has the Job Definition only registered once versus every job submission. AWS complained to us that we were maxing out their internal Batch API limits and specifically called out the Job Definition deregister/register cycle per job of a 10k+ rows parameter file
  • More verbosity when assembling (beagle assemble) and tearing down (beagle teardown) Batch resources

@caltechads
Copy link
Collaborator

i'll review this. thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants