Proto Connector Piping

[alexrichey]

    with TemporaryDirectory() as temp_dir:
        temp_path = Path(temp_dir)

        # Pull entire draft folder to temp
        drafts_conn.pull_versioned(
            key=product, version=draft_version_revision, destination_path=temp_path
        )

        # Push entire folder to published
        published_conn.push_versioned(
            key=product, version=publish_version, source_path=str(temp_path), acl=acl
        )

We end up doing this type of tying all over the code base, namely:

1. Pull the connector to a local location
2. Push to the destination

This is a bummer, because it's verbose, and it's wasteful - we often don't need to pull the files. E.g. in the case above, if we're on S3, we should just use a S3 copy method. This would short-circuit the download. There's got to be a better way...

This begins implementing the following:

drafts_conn.resource("db-pluto", "my-build") \
  >> published_conn.resource("db-pluto", "25v4") \
  >> published_conn.resource("db-pluto", "latest")

Where >> is a chain - basically an optimized pull and push combo.

And in this case, published_conn basically will say "hey, drafts_conn is also using CloudPathLib on the same cloud, so let's just use the copy method under the hood."

It'd be a really easy shorthand for something like

published_conn.resource("db-pluto", "25v4") >> edm_postgres.resource("my-build-db")

Thinking further

One could envision something like the following...

multi-to-one

Resources[recipes_conn.resource("recipe_a", "v1"), recipes_conn.resource("recipe_b", "v2")]  >> edm_postgres.resource("my-build-db")

Or inserting "operation" connectors, e.g.

drafts_conn.resource("db-pluto", "my-build") \
  >> generate_metadata.resource() \
  >> published_conn.resource("db-pluto", "25v4")

I'm going to let this percolate a little. The last example is very compelling, I think.