Add optional Apache Ozone backend to Spark Quickstart Docker demo

[slfan1989]

Background

In #14638, the discussion revolves around replacing MinIO in the Spark Quickstart due to the MinIO open-source repository entering maintenance mode and the presence of unpatched security issues. Currently, PR #14928 proposes using RustFS as a replacement. Based on this, I would like to propose a new solution: adding Apache Ozone as an optional storage backend in the Docker Compose Spark Quickstart example.

Motivation

Apache Ozone is an actively maintained Apache top-level project (with the latest 2.0.0 release in 2025), fully open-source (Apache 2.0 license), and free from commercialization risks. It provides excellent S3 compatibility (s3a://) and native Hadoop file system semantics (ofs://), making it highly compatible with the Iceberg + Spark/Hadoop ecosystem. Ozone has been widely used in production environments for big data and AI workloads, offering outstanding scalability and integration advantages.

Adding Ozone as an option can better demonstrate the storage-agnostic nature of Iceberg, providing users with more local testing options and improving its overall flexibility and usability.

Implementation Plan

The storage backend will be switched via an environment variable, for example:

STORAGE_BACKEND=minio|rustfs|ozone  # Default is minio

The implementation includes:

• Adding Ozone service: Using the official apache/ozone Docker image in a single-node freestyle mode (this mode starts quickly and has low resource usage).
• Conditional Spark configuration: Dynamically configure spark-defaults.conf based on the selected backend, such as setting the warehouse path, S3 endpoint, etc.
• Updating the README: Provide clear instructions in the Quickstart README on how to enable Ozone as a storage backend, ensuring users can easily configure and test it.

Next Steps

If the community finds this proposal valuable, I plan to submit a PR that includes the following:

• Update docker-compose.yml: Add the Ozone service and dynamically adjust service configurations based on the STORAGE_BACKEND environment variable.
• Modify the spark-defaults.conf template: Dynamically configure the relevant parameters (e.g., warehouse path, S3 endpoint) based on the selected storage backend.
• Update the README: Clearly explain how to switch between storage backends and provide steps for enabling Ozone.

[slfan1989]

@adoroszlai @ChenSammi @jojochuang @szetszwo @ayushtkn I would appreciate your thoughts and any suggestions you might have.

[RussellSpitzer]

I'm fine having any implementation that follows proper licensing and is decently supported. I'm a little more afraid of having "optional" pieces in the quick start because i'd like to keep it as simple as possible. So I'd urge us to just pick either RustFS or Ozone (honestly whichever is simpler is probably best for the quickstart)

[slfan1989]

I'm fine having any implementation that follows proper licensing and is decently supported. I'm a little more afraid of having "optional" pieces in the quick start because i'd like to keep it as simple as possible. So I'd urge us to just pick either RustFS or Ozone (honestly whichever is simpler is probably best for the quickstart)

@RussellSpitzer Thanks for the feedback! I fully agree—we need to keep the Quickstart as straightforward as possible.
I'll go ahead and work on integrating Ozone using its official Docker images for the Spark Quickstart. Once both Ozone and RustFS options are implemented, we can compare them directly and collectively decide which one ends up being simpler and more suitable for new users.

[jojochuang]

I mean, with my Ozone hat on, I'd be thrilled to have Iceberg quick start integration example with Ozone, an Apache top level project so more folks can be aware Apache has an S3 compatible object store.

In the worse case if this is not accepted, we can have it it maintained at Ozone which is a WIP: https://ozone-site-v2.staged.apache.org/docs/user-guide/integrations/iceberg

[ChenSammi]

Thanks @slfan1989 for the proposal and put the effort. Ozone 2.1.0 is closed to release now. It can be used for the initial version for docker image. I'm not sure if there is any dependency to adopt this version, for example, requiring a published Ozone 2.1.0 docker image first? cc @adoroszlai.

[umamaheswararao]

Great to see this effort @slfan1989.
As Ozone 2.1.0 is out already, we should go with that.

[SaketaChalamchala]

Hi all, I submitted these patches to be able to use Ozone instead of MinIO in the quickstart envs. Please let me know your thoughts.
databricks/docker-spark-iceberg#246
#15034

[dipankarmazumdar]

@SaketaChalamchala tested Ozone out with the "Iceberg Getting Started" notebook and worked great! Thanks for putting this out.

[slfan1989]

@SaketaChalamchala Thank you very much for your contribution. I would give this a +1.

[myrust-go]

That's a good idea. RustFS and Ozone are both products with friendly licenses.
Let users choose for themselves.

👍