feat(trainer): run dataset and model initializers in parallel

[Sayan4496]

Summary

Run dataset and model initializer containers in parallel in the container backend instead of sequential execution.

Motivation

Previously, when both dataset and model initializers were configured, they executed sequentially, increasing total startup time.
Since Docker/Podman allow multiple containers to mount the same volume simultaneously, running them in parallel reduces initialization latency to approximately the maximum of the two durations rather than their sum.

Changes

• Execute dataset and model initializers concurrently using ThreadPoolExecutor.
• Wait for all initializers to complete and propagate any failure.
• Preserve existing timeout, logging, and cleanup behavior.
• Maintain backward compatibility when only one initializer is configured.
• Add debug log after successful completion of all initializers.

Testing

• Trainer test suite passes locally:

• 184 passed

• No regressions observed in container backend behavior.

Fixes #290

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign astefanutti for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[github-actions]

🎉 Welcome to the Kubeflow SDK! 🎉

Thanks for opening your first PR! We're happy to have you as part of our community 🚀

Here's what happens next:

• If you haven't already, please check out our Contributing Guide for repo-specific guidelines and the Kubeflow Contributor Guide for general community standards
• Our team will review your PR soon! cc @kubeflow/kubeflow-sdk-team

Join the community:

• Slack: Join our #kubeflow-ml-experience and #kubeflow-trainer Slack channels
• Meetings: Attend the Kubeflow SDK and ML Experience bi-weekly meetings

Feel free to ask questions in the comments if you need any help or clarification!
Thanks again for contributing to Kubeflow! 🙏

[Copilot]

Pull request overview

Updates the container backend to reduce startup latency by running dataset and model initializer containers concurrently instead of sequentially.

Changes:

• Execute dataset and model initializers in parallel via ThreadPoolExecutor.
• Wait for initializer completion and propagate failures to the caller.
• Add a debug log after all initializers finish successfully.

[Copilot]

When iterating as_completed(futures), the first failing initializer raises immediately, which means exceptions from any other initializer future are never consumed (and the surfaced failure becomes completion-order dependent), making debugging nondeterministic when multiple initializers fail; consider capturing results for all futures (e.g., map future→name, collect exceptions from every future.result() in a list, then raise a combined/deterministic error after all have finished).

Suggested change

	# Wait for all initializers to complete and propagate errors
	for future in as_completed(futures):
	future.result()
	# Wait for all initializers to complete and collect errors deterministically
	exceptions: list[Exception] = []
	for future in as_completed(futures):
	try:
	future.result()
	except Exception as exc:
	exceptions.append(exc)
	if exceptions:
	if len(exceptions) == 1:
	raise RuntimeError("Initializer failed") from exceptions[0]
	messages = "\n".join(
	f"{idx + 1}) {type(exc).__name__}: {exc}"
	for idx, exc in enumerate(exceptions)
	)
	raise RuntimeError(
	f"Multiple initializers failed:\n{messages}"
	)