RayDP-1.6.3

What's Changed

• Standardize executor limit enforcement for dynamic and static allocation by @MehulBatra in #426
• Remove upper bound of the protobuf version by @yhori991 in #428
• CI switch to jdk17, pin click, support Spark 3.5.5-7 by @pang-wu in #438
• Support --py-files to distribute files to executors by @pang-wu in #441
• update network grpc code by @pang-wu in #442
• fix pypi publish by @pang-wu in #443
• fix python version by @pang-wu in #444
• remove usage of ray.worker.global_worker.connected by @pang-wu in #445
• Allow assign memory to RayDPSparkMaster actor by @pang-wu in #448
• tf/estimator.py: only write checkpoint in rank0 by @pang-wu in #447

Thanks @MehulBatra, @yhori991, @myandpr for their contributions to the release!

RayDP-1.6.2

What's Changed

• fix CVE by @minmingzhu in #417
• fix executor custom resource request+support pyarrow >=15 & spark 3.5.2/3/4 by @pang-wu in #419
• fix some markdown typos. by @slfan1989 in #421
• Resolve NPE Caused by Missing EXECUTOR_INSTANCES in Spark Settings. by @slfan1989 in #422
• Fix Some Typos in ReadME.md by @slfan1989 in #423
• Resolve Driver Host Recognition Issue in RayDP Executor. by @slfan1989 in #424

Thanks @pang-wu, @slfan1989, @minmingzhu for their contributions to the release!

RayDP-1.6.1

Highlights

• Support Spark 3.3.x, 3.4.x, 3.5.x #382 #395 #397 #411
• Dynamic allocation improvement #396
• Fault tolerance improvement #391
• Support setting custom actor owner when convert spark dataframe to ray dataset #376

Thanks @pang-wu, @minmingzhu, @kira-lin, @harborn, @raviranak, @KiranP-d11, @gptbert, @max-509, @Deegue for their contributions to the release!

RayDP-1.6.0

Highlights

• Support Ray 2.1 – 2.6
• Support Spark 3.1-3.4
• Fix logging: only logs from the driver is printed to the console
• Enable OneCCL as backend for TorchTrainer and TorchEstimator
• Add instructions and Dockerfiles for Ray on K8S
• Add resource affinity scheduling for RayDP executors #366. Thanks to @pang-wu

RayDP-1.5.0

Highlights

• Support Ray 2.1.0 - 2.2.0
• Support Spark 3.1 - 3.3
• XGBoostEstimator API #289
• Support converting Spark Dataframe to Ray dataset in a way that data can be recovered in case of failure. #249

RayDP-0.6.0

Highlights

• Support Ray 1.9.0 - 2.1.0
• Support Spark 3.1 - 3.3
• Spark master node affinity
• Updated PyTorch and Tensorflow Estimator using new Ray Train API

Thanks @KepingYan, @kira-lin, @pang-wu, @carsonwang for their contributions to the release!

RayDP-0.5.0

Highlights

• Support Ray 1.9.0 - 2.0.0
• Support Spark 3.1/3.2
• Hive support
• Ray placement group support
• Support multiple users running RayDP on the same node
• Support fractional resource scheduling
• Updated Estimator API using Ray Dataset and Ray Train
• Support custom Spark location by picking up $SPARK_HOME
• RayDP executor extra class path support
• Support data ownership transfer for conversion from Spark Dataframe to Ray Dataset
• New Colab tutorials

Thanks @Bowen0729, @carsonwang, @hezhaozhao-git, @jjyao, @KepingYan, @kira-lin, @marin-ma, @n1CkS4x0, @pang-wu, @wybryan, @Yard1 for their contributions to the release!

RayDP-0.4.2

RayDP-0.4.2 supports Ray 1.12.0

RayDP-0.4.1

RayDP-0.4.1 supports Ray 1.8.0

RayDP-0.4.0

Highlights

• Support conversion between Spark Dataframe and Ray Dataset
• Support Ray 1.7.0
• Support Spark 3.2.0
• Various bug fixes and improvements

Thanks @ConeyLiu @zuston @kira-lin @mjschock @edoakes @carsonwang for their contributions to the release!