Use Apache Ozone as backend store in Iceberg quickstart docker environment

[SaketaChalamchala]

Following the discussion on the thread apache/iceberg#14945, submitting a patch to use Apache Ozone as the backend store for Iceberg in quickstart docker environment since MinIO is entering maintenance mode.

Manual test using spark-shell:

[SaketaChalamchala]

Couldn't get past the Jupyter notebook security to test the flink example. Looking for a way to bypass the token/password? I tried blank token/password and tried setting a new password using jupyter server password as well.

[Copilot]

Pull request overview

This PR migrates the Iceberg quickstart Docker environment from MinIO to Apache Ozone as the S3-compatible backend storage system. This change addresses MinIO entering maintenance mode and provides a stable alternative for the quickstart environment.

Changes:

• Replaced MinIO services with a complete Apache Ozone cluster consisting of multiple services (SCM, OM, Datanode, Recon, S3 Gateway)
• Updated all S3 endpoint configurations from http://minio:9000 to http://s3.ozone:9878 across Spark, Flink, and PyIceberg configurations
• Updated documentation to reflect Apache Ozone as the storage backend

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
docker-compose.yml	Replaced MinIO and mc services with Apache Ozone cluster services (ozone-datanode, ozone-om, ozone-scm, ozone-recon, ozone-s3g) and updated S3 endpoint configuration
flink-example/docker-compose.yml	Applied identical Ozone migration changes for the Flink example environment
spark/spark-defaults.conf	Updated S3 endpoint from minio:9000 to s3.ozone:9878
spark/.pyiceberg.yaml	Updated S3 endpoint from minio:9000 to s3.ozone:9878
flink-example/src/main/java/io/tabular/flink/lor/example/LORSink.java	Updated default S3 endpoint parameter from minio:9000 to s3.ozone:9878
README.md	Updated documentation to mention Apache Ozone instead of MinIO as storage backend

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The wait command uses double dollar signs ($$s3g_pid) which is incorrect. This should be a single dollar sign ($s3g_pid) to properly reference the shell variable containing the PID of the background s3g process.

Suggested change

	wait "$$s3g_pid"
	wait "$s3g_pid"

[Copilot]

The shell variable assignment uses double dollar signs ($$!) which is incorrect for capturing the background process PID in a YAML multi-line string. This should be a single dollar sign ($!) to properly capture the PID of the background process. The double dollar sign is typically used in Makefiles or when you need to escape the dollar sign, but not needed here in the YAML command string.

Suggested change

	s3g_pid=$$!
	until ozone sh volume list >/dev/null 2>&1; do echo '...waiting...' && sleep 1; done;
	ozone sh bucket delete /s3v/warehouse || true
	ozone sh bucket create /s3v/warehouse
	wait "$$s3g_pid"
	s3g_pid=$!
	until ozone sh volume list >/dev/null 2>&1; do echo '...waiting...' && sleep 1; done;
	ozone sh bucket delete /s3v/warehouse || true
	ozone sh bucket create /s3v/warehouse
	wait "$s3g_pid"

[Copilot]

The wait command uses double dollar signs ($$s3g_pid) which is incorrect. This should be a single dollar sign ($s3g_pid) to properly reference the shell variable containing the PID of the background s3g process.

Suggested change

	s3g_pid=$$!
	until ozone sh volume list >/dev/null 2>&1; do echo '...waiting...' && sleep 1; done;
	ozone sh bucket delete /s3v/warehouse || true
	ozone sh bucket create /s3v/warehouse
	wait "$$s3g_pid"
	s3g_pid=$!
	until ozone sh volume list >/dev/null 2>&1; do echo '...waiting...' && sleep 1; done;
	ozone sh bucket delete /s3v/warehouse || true
	ozone sh bucket create /s3v/warehouse
	wait "$s3g_pid"

[Copilot]

The ozone-s3g service should have explicit depends_on declarations for the other Ozone services it relies on (ozone-om, ozone-scm, ozone-datanode) to ensure proper service startup order. While the WAITFOR environment variable in ozone-om and the polling loop in the command handle some of the initialization sequencing, adding depends_on ensures Docker Compose starts the services in the correct order.

Suggested change

	- 9878:9878
	- 9878:9878
	depends_on:
	- ozone-om
	- ozone-scm
	- ozone-datanode

[Copilot]

The ozone-s3g service should have explicit depends_on declarations for the other Ozone services it relies on (ozone-om, ozone-scm, ozone-datanode) to ensure proper service startup order. While the WAITFOR environment variable in ozone-om and the polling loop in the command handle some of the initialization sequencing, adding depends_on ensures Docker Compose starts the services in the correct order.

Suggested change

	<<: *ozone-common-config
	<<: *ozone-common-config
	depends_on:
	- ozone-om
	- ozone-scm
	- ozone-datanode

[Copilot]

The ozone-om service should have an explicit depends_on declaration for ozone-scm since it uses WAITFOR: ozone-scm:9876 to ensure SCM is ready before starting. While the WAITFOR mechanism handles the runtime dependency, adding depends_on ensures Docker Compose starts ozone-scm before ozone-om.

Suggested change

	iceberg_net:
	iceberg_net:
	depends_on:
	- ozone-scm

[Copilot]

The ozone-om service should have an explicit depends_on declaration for ozone-scm since it uses WAITFOR: ozone-scm:9876 to ensure SCM is ready before starting. While the WAITFOR mechanism handles the runtime dependency, adding depends_on ensures Docker Compose starts ozone-scm before ozone-om.

Suggested change

	iceberg_net:
	iceberg_net:
	depends_on:
	- ozone-scm

[Copilot]

The shell variable assignment uses double dollar signs ($$!) which is incorrect for capturing the background process PID in a YAML multi-line string. This should be a single dollar sign ($!) to properly capture the PID of the background process. The double dollar sign is typically used in Makefiles or when you need to escape the dollar sign, but not needed here in the YAML command string.

Suggested change

	s3g_pid=$$!
	until ozone sh volume list >/dev/null 2>&1; do echo '...waiting...' && sleep 1; done;
	ozone sh bucket delete /s3v/warehouse || true
	ozone sh bucket create /s3v/warehouse
	wait "$$s3g_pid"
	s3g_pid=$!
	until ozone sh volume list >/dev/null 2>&1; do echo '...waiting...' && sleep 1; done;
	ozone sh bucket delete /s3v/warehouse || true
	ozone sh bucket create /s3v/warehouse
	wait "$s3g_pid"

[Copilot]

The spark.sql.catalog.demo.s3.endpoint is configured to use plain HTTP to reach the object store, which sends S3 requests and any associated credentials over the network without encryption. An attacker with access to the host or the Docker network could intercept or tamper with catalog and warehouse data in transit. Consider configuring this endpoint to use TLS (HTTPS) and enabling the corresponding TLS support on the Ozone S3 gateway to protect data and credentials in transit.

Suggested change

	spark.sql.catalog.demo.s3.endpoint http://s3.ozone:9878
	spark.sql.catalog.demo.s3.endpoint https://s3.ozone:9878

[dipankarmazumdar]

@SaketaChalamchala Hi! were you able to get past the Jupyter thing?

[SaketaChalamchala]

@SaketaChalamchala Hi! were you able to get past the Jupyter thing?

@dipankarmazumdar, I still can't find a way to get to the notebook server past the security page even though I set the password. Just wanted to test the flink example, might try the flink REST API for creating the DB. Has anybody tried to use the instructions in the flink-example/README.md recently though? Does it work?

Just clarifying that this is not a blocker for Ozone adoption. The updated docker-compose.yaml can be used to bring up a working quickstart cluster with Ozone as a backend store for Iceberg.

[dipankarmazumdar]

@SaketaChalamchala Hi! were you able to get past the Jupyter thing?

@dipankarmazumdar, I still can't find a way to get to the notebook server past the security page even though I set the password. Just wanted to test the flink example, might try the flink REST API for creating the DB. Has anybody tried to use the instructions in the flink-example/README.md recently though? Does it work?

Just clarifying that this is not a blocker for Ozone adoption. The updated docker-compose.yaml can be used to bring up a working quickstart cluster with Ozone as a backend store for Iceberg.

Thanks! I was able to test the "Iceberg - Getting Started" Ipynb and it worked well.

Question: How do I see the content of the /metadata and /data folder in Ozone? Is it Ozone Recon or Manager? Sorry I am new to Ozone.

[SaketaChalamchala]

Question: How do I see the content of the /metadata and /data folder in Ozone? Is it Ozone Recon or Manager? Sorry I am new to Ozone.

Ozone exposes a both a Hadoop compatible file system and S3 client interfaces.

S3 interface: Use any S3 client like AWS cli. Sample commands are below

aws s3api --endpoint http://s3.ozone:9878/ list-objects --bucket warehouse
aws s3api --endpoint http://s3.ozone:9878/ get-object --bucket warehouse --key nyc/logs/metadata/<meta file> <local file name>

File system interface: Use ofs protocol from any Ozone client (any of the ozone containers in the docker environment like ozone-*). The file system URL will look like ofs://<OM service ID>/<volume name>/<bucket name>/<key name>. All S3 buckets are created under s3v volume and the Ozone service ID in the docker environment is ozone-om. Below are a list of sample commands:

## Recursive list of warehouse bucket
ofs fs -ls -R ofs://ozone-om/s3v/warehouse
## list data and metadata 
ofs fs -ls ofs://ozone-om/s3v/warehouse/nyc/logs/metadata
ofs fs -ls ofs://ozone-om/s3v/warehouse/nyc/logs/data
## Read a key
ofs fs -cat ofs://ozone-om/s3v/warehouse/nyc/logs/metadata/<meta file>
## Get a key
ofs fs -get ofs://ozone-om/s3v/warehouse/nyc/logs/metadata/<meta file> <local file name>

[umamaheswararao]

@dipankarmazumdar since you asked about OzoneManager and Recon above, here is my explanation for that processes.

• OzoneManager is a master service manages the metadata for the ozone.
• Storage Container Manager process manages the container/block pools to datanode locations mappings. These services use Ratis/Raft to provide the high availability.
• Recon is an observability service. It will provide the cluster metrics and insights.

As @SaketaChalamchala mentioned, for a specific path access, you should be able to browse via fs/s3 apis.

Thank you @SaketaChalamchala for providing the commands to look.

[dipankarmazumdar]

@dipankarmazumdar since you asked about OzoneManager and Recon above, here is my explanation for that processes.

• OzoneManager is a master service manages the metadata for the ozone.
• Storage Container Manager process manages the container/block pools to datanode locations mappings. These services use Ratis/Raft to provide the high availability.
• Recon is an observability service. It will provide the cluster metrics and insights.

As @SaketaChalamchala mentioned, for a specific path access, you should be able to browse via fs/s3 apis.

Thank you @SaketaChalamchala for providing the commands to look.

Thanks, this helps!

[SaketaChalamchala]

@dipankarmazumdar Would you be able to help get an approval for this patch?

[dipankarmazumdar]

Hi @nastra! would you be able to review this?

[SaketaChalamchala]

HI @nastra / @dipankarmazumdar, The CI is failing when building the spark container. Looks like https://dlcdn.apache.org/spark/ which no longer hosts v3.5.6.

------
 > [ 7/24] RUN mkdir -p /opt/spark  && curl https://dlcdn.apache.org/spark/spark-3.5.6/spark-3.5.6-bin-hadoop3.tgz -o spark-3.5.6-bin-hadoop3.tgz  && tar xvzf spark-3.5.6-bin-hadoop3.tgz --directory /opt/spark --strip-components 1  && rm -rf spark-3.5.6-bin-hadoop3.tgz:
Dockerfile:50
....

failed to solve: process "/bin/sh -c mkdir -p ${SPARK_HOME}  && curl [https://dlcdn.apache.org/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop3.tgz](https://dlcdn.apache.org/spark/spark-$%7BSPARK_VERSION%7D/spark-$%7BSPARK_VERSION%7D-bin-hadoop3.tgz) -o spark-${SPARK_VERSION}-bin-hadoop3.tgz  && tar xvzf spark-${SPARK_VERSION}-bin-hadoop3.tgz --directory /opt/spark --strip-components 1  && rm -rf spark-${SPARK_VERSION}-bin-hadoop3.tgz" did not complete successfully: exit code: 2

 0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   196  100   196    0     0   2252      0 --:--:-- --:--:-- --:--:--  2252
0.208 
0.208 gzip: stdin: not in gzip format
0.208 tar: Child returned status 1
0.208 tar: Error is not recoverable: exiting now
------
Error: Process completed with exit code 1.

[adoroszlai]

CI is failing when building the spark container. Looks like https://dlcdn.apache.org/spark/ no longer hosts v3.5.6.

#247 would fix this

[SaketaChalamchala]

@nastra or @Fokko, I made an update to the docker-compose.yml to wait for the bucket creation to complete before starting the iceberg-spark container and running the tests. It works on my local.
Could you please trigger the CI again?

[liuml07]

Replacing MinIO makes sense given its current state. Also using an Apache top-level project, Ozone, sounds good. Using an S3-compatible system is also a sensible choice.

My only question is about the simplicity. I feel we may want to make this repo as simple as possible, enabling users to start the local Spark/Flink environment with Iceberg quickly and easily. If things go wrong, it's easier to debug; also maintainers and advanced users of this would feel straightforward when making changes.

For that, I have some quick questions:

• We are starting fully-fledged Ozone system with 5 new services. Is there a way to consolidate them into fewer ones?
• There are config definitions for Ozone, which are less interesting to the Iceberg quickstart. Can we fallback to defaults, or put configs to an env_file and reference it in the docker compose file?
• Alternatively, can we encapsulate all Ozone services into another file such as docker-compose.ozone.yml, and include it into the main docker compose file? That way the main docker compose file would be more concise and relevant as a quickstart local docker.

Last, some auto-review comments made by Copilot seems valid, for example, the depends_on.

[liuml07]

Probably we can link to the project

[adoroszlai]

can we encapsulate all Ozone services into another file such as docker-compose.ozone.yml, and include it into the main docker compose file? That way the main docker compose file would be more concise and relevant as a quickstart local docker.

@liuml07 That's a good idea. I have prepared another patch towards that end:

1. extract minio and flink services into separate compose files, making the S3 backend replaceable
2. add the Ozone S3 backend, as default (reusing the definition from this PR with minor tweaks, hence set author to @SaketaChalamchala), but still keeping the MinIO implementation

This approach also allows adding the RustFS backend as an alternative.

Please let me know what you think.

(I have moved Ozone's S3 service to minio:9000 to make it work with the existing tabulario/spark-iceberg image, which has http://minio:9000 in its config. If that's confusing or may trigger complaints from MinIO, we can change that.)

[ssulav]

can we encapsulate all Ozone services into another file such as docker-compose.ozone.yml, and include it into the main docker compose file? That way the main docker compose file would be more concise and relevant as a quickstart local docker.

@liuml07 That's a good idea. I have prepared another patch towards that end:

1. extract minio and flink services into separate compose files, making the S3 backend replaceable
2. add the Ozone S3 backend, as default (reusing the definition from this PR with minor tweaks, hence set author to @SaketaChalamchala), but still keeping the MinIO implementation

This approach also allows adding the RustFS backend as an alternative.

Please let me know what you think.

(I have moved Ozone's S3 service to minio:9000 to make it work with the existing tabulario/spark-iceberg image, which has http://minio:9000 in its config. If that's confusing or may trigger complaints from MinIO, we can change that.)

Would not this be confusing, as we are using ozone storage internally but hiding it behind Minio as the name? It does make it compatible with fewer changes, but it may get confusing if we want to change the underlying storage in future

[SaketaChalamchala]

can we encapsulate all Ozone services into another file such as docker-compose.ozone.yml, and include it into the main docker compose file? That way the main docker compose file would be more concise and relevant as a quickstart local docker.

@liuml07 That's a good idea. I have prepared another patch towards that end:

1. extract minio and flink services into separate compose files, making the S3 backend replaceable
2. add the Ozone S3 backend, as default (reusing the definition from this PR with minor tweaks, hence set author to @SaketaChalamchala), but still keeping the MinIO implementation

This approach also allows adding the RustFS backend as an alternative.
Please let me know what you think.
(I have moved Ozone's S3 service to minio:9000 to make it work with the existing tabulario/spark-iceberg image, which has http://minio:9000 in its config. If that's confusing or may trigger complaints from MinIO, we can change that.)

Would not this be confusing, as we are using ozone storage internally but hiding it behind Minio as the name? It does make it compatible with fewer changes, but it may get confusing if we want to change the underlying storage in future

I agree, for the quickstart we could just keep the docker-compose.yml and have the backend storage(Ozone) included and have the users just do a docker-compose up to have a working environment for them to get started with instead of providing options.
+1 for not referring to Ozone S3 as MinIO.
I updated this patch as well. We can proceed with this or @adoroszlai's patch, whichever is acceptable.

[SaketaChalamchala]

@Fokko @nastra @liuml07 , Do you have anymore thoughts on this?