Configurable Stage Thread Pool

[ngngwr]

Issues

• My PR addresses the following Helix issues and references them in the PR description:

Make StageThreadPoolHelper thread pool size configurable

Description

What: Makes the parallel stage execution thread pool size configurable via ClusterConfig.
Why: Currently the thread pool size in StageThreadPoolHelper is hardcoded. Different clusters may have different parallelism requirements based on their workload and resources.
How:

• Added STAGE_PARALLEL_THREAD_POOL_SIZE property to ClusterConfig with getter/setter methods
• Added setPoolSize() method to StageThreadPoolHelper for dynamic pool size updates
• GenericHelixController reads the config on each pipeline run and updates the pool if changed
• Default remains min(4, availableProcessors) when not configured

Usage:

ClusterConfig clusterConfig = new ClusterConfig(clusterName); clusterConfig.setStageParallelThreadPoolSize(8); configAccessor.setClusterConfig(clusterName, clusterConfig);

Tests

TestStageThreadPoolHelper - tests for setPoolSize(), getPoolSize(), and dynamic pool resizing