Local Runway

[johncmerfeld]

@edandylytics did most of the brainwork for this feature; I think it's finally ready to go! I had to transfer this over from the old Runway repo, so please be on the lookout for any rollbacks of code changes that have come in since the switch.

How to try this

Follow the instructions in app/README.md. It should be pretty quick to get up and running jobs in either local mode.

What this does

When running locally:

• Instead of kicking of an ECS task, the app launches the executor either in a local container or as a Python child process
• Instead of the user uploading files to S3, "select file" sends to a local folder in the Runway repo
• Instead of the executor uploading its artifacts to S3, it writes them to the same storage location, from which the app reads them
• The user should not notice any difference in behavior. Output data are still uploaded to whatever ODS is configured

What's changed

• Minimal changes to the executor. The app indicates it is running locally by way of a DEPLOYMENT_MODE env var and by sending file:// sources instead of s3:// ones. We could couple those more tightly but they are nominally independent
• Heaviest app change is new mechanisms for local file upload/download. These are pretty straightforward but it's kind of a lot of code
• Otherwise the app has a reworked model of the executor, where the AWS version is just one of three operating modes. There are some other small changes due to the app sometimes needing to be available over host.docker.internal
• Documentation and setup
• Delete some files from the old days when we had a different theory of local execution

For reviewers

I think we achieved pretty good demarcation between the local stuff and production code, but this is a ratchet upward in complexity so we'll want to make sure none of the deployed functionality is impacted. There are a couple small TODOs that are really just opinion questions. New Typescript code should be scrutinized :)