Native Gang Scheduling (Proof of Concept)

[kruegercharles]

This is a proof of concept for gang scheduling in Cortex using the native Kubernetes v1.35 Workload API.

I know it is still undecided whether Cortex is supposed to implement its own gang scheduler or rely on Kubernetes' native support.

If we decide Cortex is supposed to have its own gang scheduling mechanism instead of relying on the solution provided by Kubernetes, we might be able to reuse parts of this PR in any case. Also, even if we don't use the native gang scheduling, it might at some point be useful to have a comparison of both implementations.
If this cannot be used in any way, it was still quite interesting for me to test out the native gang scheduling.

Usage:

• Deploy resources via tilt up in existing cluster, e.g. minikube
• Set context: kubectl config set-context --current --namespace=cortex-system
• Check sample pod gang with kubectl describe pod gang-pod

Interesting links:

• https://kubernetes.io/docs/concepts/workloads/workload-api/
• https://kubernetes.io/docs/concepts/workloads/pods/workload-reference/

Ref Issue: #393