Possible future backend using googleCloudRunner?

[MarkEdmondson1234]

Dear Henrik, we have the existing googleComputeEngineR integration with future which has been highly appreciated.

Will asked me over at ropensci/targets#720 if its possible that future can support Google Cloud Build via googleCloudRunner and I'd like to hear your opinion on it. I'm not 100% its possible as Cloud build is API based rather than SSH - you send jobs to it via specifying build steps in a .yaml file including the Docker container to run that step within then arguments such as R scripts or otherwise to run in a shared /workspace.

Is it feasible to have a local future plan, and then each task is thrown up to a separate Cloud build? Advantages would be scale and cost, but with a large (1min?) start-up overheard. Each build is stateless but some kind of communication could be done using Cloud Storage perhaps? Anyhow if you can let me know if its possible then I could look at trying to put something together in 2022.

[wlandau]

What goes on in that 1 minute overhead? Is it just a matter of creating a running instance of a pre-built Docker image?

[MarkEdmondson1234]

Most of the time is downloading docker images for the code to run in. Optimisations could include having the Docker image within the source its running within, but can tweak that once the workflows are established.

[wlandau]

From my perspective, I think the overhead is fine as long as it does not block the main process. Individual cloud-worthy jobs are going to take a long time anyway (in targets, especially with batching). From the perspective of efficient orchestration in targets, I am most concerned about the speed and asynchronicity of:

• future::future(). Can a job be launched while spending as little time as possible blocking the main/parent process? Would efficient initialization require a temporary local background process?
• future::resolved(). The event loop of targets::tar_make_future() unavoidably spends time looping over future::resolved(), and even a couple seconds of wait time are noticeable when there are hundreds of workers to check. (Tough for targets to do better without multithreaded checking within the orchestrating R process.) How asynchronous can this get? What if the first call to resolved() returns FALSE and sends an async Google API request, and subsequent calls either get the result or return FALSE if it is not available yet?