[Core feature] map_task to support ContainerTask

[litaifang]

Motivation: Why do you think this is important?

Being able to use any arbitrary docker to execute a task (i.e., ContainerTask) is an important feature in many bioinformatic workflow schemes. However, ContainerTask is currently not supported by map tasks.
This and this were never merged so this capability never made it into flyte.
It'd be great to be able to use map_task to spin out ContainerTasks in parallel.

Goal: What should the final outcome look like, ideally?

Something like this should work as intended:

@dynamic
def map_task_of_container_task(...) -> None:
    inner_task = ContainerTask(
        name="inner-task-bring-your-own-container",
        input_data_dir="/var/inputs",
        output_data_dir="/var/outputs",
        inputs=kwtypes(...),
        outputs=kwtypes(...),
        image="docker.io/ubuntu:latest",
        command=["sh", "-c", ...],
    )
    map_task(inner_task)(map_input=[...])

Describe alternatives you've considered

Alternatively I can just loop over the ContainerTask in a dynamic, but it doesn't have the benefits of a map_task.

Propose: Link/Inline OR Additional context

There were two pull requests attempting to add this feature but were never merged: this and this.

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[welcome]

Thank you for opening your first issue here! 🛠

[eapolinario]

@litaifang , contributions are welcome. The flytekit PR will need to be updated to use https://github.com/flyteorg/flytekit/blob/master/flytekit/core/array_node_map_task.py instead, but the logic should be very similar. And the backend PR should be ported to https://github.com/flyteorg/flyte.

[machichima]

#take

[ai-rnatour]

We would also be interested in this!

[rupshac]

Same here.. we need this as well

[github-actions]

"Hello 👋, this feature request has been inactive for over 9 months. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will close the issue if we detect no activity in the next 14 days. Thank you for your contribution and understanding! 🙏"