Upgrade Iceberg to 1.9.2 and Avro to 1.12.0 #784

kevinjqliu · 2026-01-15T04:09:07Z

Closes Iceberg Version Upgrade #783

What is the purpose of the pull request

Upgrades Iceberg and Avro libraries to newer versions to benefit from bug fixes, performance improvements, and new features.

Version Changes

Library	Previous	New
Iceberg	1.4.2	1.9.2
Iceberg Hive Runtime	1.4.2	1.7.2
Avro	1.11.4	1.12.0

Note:

iceberg-hive-runtime is pinned to 1.7.2 as it was removed in Iceberg 1.8.0. The Hive runtime functionality has been restructured in newer Iceberg versions.
Iceberg 1.9.2 relies on Avro 1.12.0 APIs, https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-core/1.9.2/iceberg-core-1.9.2.pom

Verify this pull request

This pull request is already covered by existing tests, such as (please describe tests).

pom.xml

the-other-tim-brown · 2026-01-15T22:52:43Z

Tracking the CI issues in this issue #787

the-other-tim-brown · 2026-01-16T21:18:30Z

Tracking the CI issues in this issue #787

The CI issue is resolved, please rebase your PR

the-other-tim-brown · 2026-01-19T19:05:40Z

xtable-core/src/test/resources/schemas/basic_schema.avsc

+      "type": ["null", "bytes"],
+      "default": null


Let's not modify the test schema as part of this change. It seems like it is not required.

i believe this is needed for the new parquet library version, but let me revert and see

yep, fails with

Error: org.apache.xtable.hudi.ITHudiConversionSource.insertAndUpsertData(HoodieTableType, PartitionConfig)[3] -- Time elapsed: 10.34 s <<< ERROR! org.apache.hudi.exception.HoodieInsertException: Failed to bulk insert for commit time 20260119224246020 at org.apache.hudi.table.action.commit.JavaBulkInsertCommitActionExecutor.execute(JavaBulkInsertCommitActionExecutor.java:63) at org.apache.hudi.table.HoodieJavaCopyOnWriteTable.bulkInsert(HoodieJavaCopyOnWriteTable.java:118) at org.apache.hudi.table.HoodieJavaCopyOnWriteTable.bulkInsert(HoodieJavaCopyOnWriteTable.java:85) at org.apache.hudi.client.HoodieJavaWriteClient.bulkInsert(HoodieJavaWriteClient.java:169) at org.apache.hudi.client.HoodieJavaWriteClient.bulkInsert(HoodieJavaWriteClient.java:158) at org.apache.xtable.TestJavaHudiTable.insertRecordsWithCommitAlreadyStarted(TestJavaHudiTable.java:195) at org.apache.xtable.hudi.ITHudiConversionSource.insertAndUpsertData(ITHudiConversionSource.java:245) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.tryRemoveAndExec(ForkJoinPool.java:1062) at java.base/java.util.concurrent.ForkJoinPool.awaitJoin(ForkJoinPool.java:1688) at java.base/java.util.concurrent.ForkJoinTask.doJoin(ForkJoinTask.java:397) at java.base/java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1004) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) Caused by: java.lang.RuntimeException: org.apache.hudi.exception.HoodieException: org.apache.hudi.exception.HoodieException: org.apache.avro.AvroTypeException: Invalid default for field bytes_field: "" not a "bytes" at org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:121) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) at org.apache.hudi.table.action.commit.JavaBulkInsertHelper.bulkInsert(JavaBulkInsertHelper.java:131) at org.apache.hudi.table.action.commit.JavaBulkInsertHelper.bulkInsert(JavaBulkInsertHelper.java:84) at org.apache.hudi.table.action.commit.JavaBulkInsertCommitActionExecutor.execute(JavaBulkInsertCommitActionExecutor.java:58) ... 17 more Caused by: org.apache.hudi.exception.HoodieException: org.apache.hudi.exception.HoodieException: org.apache.avro.AvroTypeException: Invalid default for field bytes_field: "" not a "bytes" at org.apache.hudi.execution.JavaLazyInsertIterable.computeNext(JavaLazyInsertIterable.java:71) at org.apache.hudi.execution.JavaLazyInsertIterable.computeNext(JavaLazyInsertIterable.java:37) at org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:119) ... 21 more Caused by: org.apache.hudi.exception.HoodieException: org.apache.avro.AvroTypeException: Invalid default for field bytes_field: "" not a "bytes" at org.apache.hudi.common.util.queue.SimpleExecutor.execute(SimpleExecutor.java:75) at org.apache.hudi.execution.JavaLazyInsertIterable.computeNext(JavaLazyInsertIterable.java:67) ... 23 more Caused by: org.apache.avro.AvroTypeException: Invalid default for field bytes_field: "" not a "bytes" at org.apache.avro.Schema.validateDefault(Schema.java:1719) at org.apache.avro.Schema$Field.<init>(Schema.java:578) at org.apache.avro.Schema$Field.<init>(Schema.java:614) at org.apache.hudi.avro.HoodieAvroUtils.addMetadataFields(HoodieAvroUtils.java:291) at org.apache.hudi.io.HoodieWriteHandle.<init>(HoodieWriteHandle.java:96) at org.apache.hudi.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:89) at org.apache.hudi.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:76) at org.apache.hudi.io.CreateHandleFactory.create(CreateHandleFactory.java:45) at org.apache.hudi.execution.CopyOnWriteInsertHandler.consume(CopyOnWriteInsertHandler.java:85) at org.apache.hudi.execution.CopyOnWriteInsertHandler.consume(CopyOnWriteInsertHandler.java:42) at org.apache.hudi.common.util.queue.SimpleExecutor.execute(SimpleExecutor.java:69) ... 24 more

https://github.com/apache/incubator-xtable/actions/runs/21153333020/job/60833552403?pr=784

This looks like an avro issue, not parquet. Is the avro upgrade required?

oops yea, i meant new avro library version. let me double check if the upgrade is necessary

the avro upgrade is required, Iceberg 1.9.2 relies on Avro 1.12.0 APIs

and the schema changes here in basic_schema.avsc is due to compatibility with Avro 1.12.0's stricter default value handling

kevinjqliu · 2026-01-19T22:38:10Z

xtable-core/src/test/java/org/apache/xtable/iceberg/TestIcebergSync.java

 * Validates that the metadata for the table is properly created/updated. {@link
 * ITConversionController} validates that the table and its data can be properly read.
 */
+@Execution(ExecutionMode.SAME_THREAD)


this is so that github CI wont run into concurrency issues, i see this is done for TestDeltaSync as well

What is the concurrency issue? This wasn't required on the lower version of Iceberg

I remember its about the InMemory catalog the tests are using.

kevinjqliu · 2026-01-19T22:38:40Z

xtable-core/src/test/resources/schemas/basic_schema.avsc

+      "type": ["null", "bytes"],
+      "default": null


i believe this is needed for the new parquet library version, but let me revert and see

pom.xml

kevinjqliu · 2026-01-20T00:28:36Z

CI is green @the-other-tim-brown could you take another look?

i tried removing as many unrelated changes as possible

the-other-tim-brown · 2026-01-20T14:12:50Z

@kevinjqliu why not upgrade to the latest iceberg? Can you update the PR description with why each version was chosen?

kevinjqliu · 2026-01-21T21:24:10Z

Thanks for the review @the-other-tim-brown
I'll double check if the avro version upgrade is needed here.

I was going to update #783 on why 1.9.2 is picked.
Latest Iceberg version (1.10.x) uses parquet 1.15 which conflicts with the parquet version in Hudi. Specifically, for the new variant feature that was added
I choose 1.9.x for now and we can deal with the parquet version as a follow up

the-other-tim-brown · 2026-01-22T04:05:56Z

@kevinjqliu for the hive support, is there a new import we should use to keep the version consistent? Or is it removed completely?

kevinjqliu · 2026-01-22T05:17:33Z

unfortunately iceberg-hive-runtime is removed after 1.7.2, https://mvnrepository.com/artifact/org.apache.iceberg/iceberg-hive-runtime
i want to punt this as a follow up

kevinjqliu · 2026-01-22T05:21:54Z

I added a note in the PR description on why we need the avro version upgrade.

I want to keep this PR scoped to just upgrading the iceberg library version. LMK what you think.

jbonofre · 2026-01-22T06:31:55Z

I think there are also other updates required to bump Iceberg version. Let me take a look.

This was referenced Jan 15, 2026

Iceberg Version Upgrade #783

Open

try changing quarkus to latest version with java11 support #785

Closed

the-other-tim-brown reviewed Jan 15, 2026

View reviewed changes

pom.xml Outdated Show resolved Hide resolved

kevinjqliu added 10 commits January 19, 2026 10:54

upgrade to iceberg 1.10.1

a642519

api changes

0b2f6ba

use best available iceberg-hive-runtime

1d55f64

upgrade avro version

b4650b1

to run with java 17

7b16556

tests

915318b

fix avro

56ecab5

dont change notebook

6add74a

fix tests

f9f2995

use iceberg 1.9.2

afea83c

kevinjqliu force-pushed the kevinjqliu/iceberg-upgrade branch from a4b2a8c to afea83c Compare January 19, 2026 18:54

the-other-tim-brown reviewed Jan 19, 2026

View reviewed changes

kevinjqliu added 4 commits January 19, 2026 11:28

unnecessary changes

a1a4974

address ci issue

9bb4947

fix version in LICENSE

9f0b673

try revert

863e7c4

kevinjqliu commented Jan 19, 2026

View reviewed changes

add back avro schema change

9c6949b

kevinjqliu changed the title ~~upgrade to iceberg 1.10.1~~ Upgrade Iceberg to 1.9.2 and Avro to 1.12.0 Jan 19, 2026

kevinjqliu added 2 commits January 19, 2026 15:09

fix ci concurrency

5804786

mvn spotless:apply

f1b59b5

kevinjqliu marked this pull request as ready for review January 20, 2026 00:28

kevinjqliu requested a review from the-other-tim-brown January 22, 2026 05:19

Upgrade Iceberg to 1.9.2 and Avro to 1.12.0 #784

Are you sure you want to change the base?

Upgrade Iceberg to 1.9.2 and Avro to 1.12.0 #784

Uh oh!

Conversation

kevinjqliu commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is the purpose of the pull request

Version Changes

Verify this pull request

Uh oh!

Uh oh!

the-other-tim-brown commented Jan 15, 2026

Uh oh!

the-other-tim-brown commented Jan 16, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kevinjqliu Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kevinjqliu Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kevinjqliu commented Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

the-other-tim-brown commented Jan 20, 2026

Uh oh!

kevinjqliu commented Jan 21, 2026

Uh oh!

the-other-tim-brown commented Jan 22, 2026

Uh oh!

kevinjqliu commented Jan 22, 2026

Uh oh!

kevinjqliu commented Jan 22, 2026

Uh oh!

jbonofre commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kevinjqliu commented Jan 15, 2026 •

edited

Loading

kevinjqliu Jan 19, 2026 •

edited

Loading

kevinjqliu Jan 22, 2026 •

edited

Loading

kevinjqliu commented Jan 20, 2026 •

edited

Loading

jbonofre commented Jan 22, 2026 •

edited

Loading