[REST | SPARK]: Reference implementation of referenced-by in the loadTable call #13979

singhpk234 · 2025-09-03T00:54:15Z

About the change

This provides a reference implementation for passing the view name that table is referenced in as part of which the loadTable call is being made.

Details on the spec change proposal and motivation here

SPEC: Add referenced-by in loadTable API #13810

gaborkaszab

Thanks for the PR @singhpk234 !
I took a look for my own benefit, and left some observations and questions meanwhile. Let me know what you think!

api/src/main/java/org/apache/iceberg/catalog/ContextAwareTableCatalog.java

gaborkaszab · 2025-09-05T07:51:53Z

api/src/main/java/org/apache/iceberg/catalog/ContextAwareTableCatalog.java

+   * @throws NoSuchTableException if the table does not exist
+   */
+  Table loadTableViaView(TableIdentifier identifier, Map<String, Object> context)
+      throws NoSuchTableException;


What if the referencing view doesn't exist? Shouldn't we check for the referencing view and throw NoSuchViewException, or should we silently omit if the view doesn't exist?

Is the concern that its a public API, that we should validate this ? My intention was to treat this as a runtime property, I renamed this API to loadTableWithContext, please let me know if this helps addressing this concern.

gaborkaszab · 2025-09-05T07:53:25Z

api/src/main/java/org/apache/iceberg/catalog/ContextAwareTableCatalog.java

+   * @return the loaded table
+   * @throws NoSuchTableException if the table does not exist
+   */
+  Table loadTableViaView(TableIdentifier identifier, Map<String, Object> context)


Can we expect more context params to this function? Can't we have a simple String parameter to hold the name of the view? Or Maybe a TableIdentifier?

Can we expect more context params to this function

That was the intention, to leave door open to pass more runtime context to the catalog, i was trying to model this as something like we already do i.e SessionContext this is a more runtime context so that engines can plumb their context back to the catalog and then catalog client can use this to make subsequent calls to the server. Hence didn't just have view identifier in the signature.

gaborkaszab · 2025-09-05T08:13:36Z

core/src/main/java/org/apache/iceberg/rest/RESTCatalog.java


+  @Override
+  public Table loadTableViaView(TableIdentifier identifier, Map<String, Object> viewContext) {
+    if (delegate instanceof ContextAwareTableCatalog) {


delegate is a BaseSessionCatalog.AsCatalog that now implements ContextAwareTableCatalog. Is there a use-case or configuration where delegate doesn't implement the new interface?

I see the concept in this class is rather to have a new member of type ContextAwareTableCatalog that is converted from delegate and this new function could directly call that. Similarly as nsCatalog.

Would this also work here? Do I miss something?

core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java

api/src/main/java/org/apache/iceberg/catalog/ContextAwareTableCatalog.java

gaborkaszab · 2025-09-05T08:58:44Z

spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java

+  }
+
+  @Override
+  public Table loadTableViaView(Identifier ident, String version, Map<String, Object> context)


Same comment as in core/: Now a loadTableViaView function performs the load table even not via view, that seems a bit weird.

Renamed it to loadTableWithContext where context would be a runtime context, let me know does is sounds better.

api/src/main/java/org/apache/iceberg/catalog/ContextAwareTableCatalog.java

core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java

gaborkaszab · 2025-09-05T09:51:22Z

core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java

+        String[] namespaceParts = new String[parts.length - 1];
+        System.arraycopy(parts, 0, namespaceParts, 0, parts.length - 1);
+        String viewName = parts[parts.length - 1];
+        queryParams.put(


Just for my understanding:
Let's take the following input: "ns1.ns2.view1"
We convert this to "ns1%1Fns2%1F.view1"
Is this because we wan't to tell the REST catalog which part is the namespace and which is the view name? If we include a dot in the query params, can't we just send the view path as it is with dots and let the catalog server figure out the namespace/view name part that we do here? Or can we also use the same special char between the namespace and the view name?

one way to look at this is anything from the last . to the end is the view name and the every thing before it is the namespace, the reason %1F was chosen the separator was to use something similar, the reason i choose the %1F is the language of the existing rest spec for the nested namespaces.

I think we are yet to conclude on the separator discussion.

api/src/main/java/org/apache/iceberg/catalog/ContextAwareTableCatalog.java

core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java

...ain/scala/org/apache/spark/sql/catalyst/plans/logical/views/UnResolvedRelationFromView.scala

github-actions · 2025-12-04T00:18:43Z

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

gaborkaszab

I went through the api/ and core/ part for now (haven't checked Spark). Left some questions and nits. I also found some of my previous comments so I'm not sure this was ready for review.
Anyway, some test coverage in TestRESTCatalog would be great!

gaborkaszab · 2026-01-13T11:34:59Z

api/src/main/java/org/apache/iceberg/catalog/ContextAwareTableCatalog.java

+   * <p>Common context keys:
+   *
+   * <ul>
+   *   <li>{@link #VIEW_IDENTIFIER_KEY}: A List&lt;TableIdentifier&gt; of view(s) referencing this


Is it necessary to escape the lt and gt chars?
"List<TableIdentifier>"

gaborkaszab · 2026-01-13T11:37:03Z

api/src/main/java/org/apache/iceberg/catalog/SessionCatalog.java

+
+  default Table loadTableWithContext(
+      SessionContext sessionContext, TableIdentifier ident, Map<String, Object> context) {
+    return loadTable(sessionContext, ident);


Have you considered throwing UnsupportedOperationException instead of silently ignore the context param?

gaborkaszab · 2026-01-13T11:46:38Z

core/src/main/java/org/apache/iceberg/catalog/BaseSessionCatalog.java

+
+    @Override
+    public Table loadTable(TableIdentifier identifier, Map<String, Object> loadingContext) {
+      return BaseSessionCatalog.this.loadTableWithContext(context, identifier, loadingContext);


nit: maybe consider renaming context to sessionContext to avoid confusion with the new type of context?

gaborkaszab · 2026-01-13T11:50:35Z

core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java

        .get(
            paths.table(identifier),
-            snapshotModeToParam(mode),
+            referencedByToQueryParam(snapshotModeToParam(mode), viewContext),


nit: here at the calcite the reader could have the expression that the order of referencedByToQueryParam and snapshotModeToParam is interchangeable, but in practice it isn't. Maybe having a separate function to paramsForLoadTable that produces both?

gaborkaszab · 2026-01-13T11:53:26Z

core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java

+    Map<String, String> queryParams = Maps.newHashMap(params);
+    Object viewIdentifierObj = context.get(ContextAwareTableCatalog.VIEW_IDENTIFIER_KEY);
+
+    if (!(viewIdentifierObj instanceof List)) {


nit: maybe Preconditions.checkArgument(viewIdentifierObj instanceof List, error msg)

gaborkaszab · 2026-01-13T11:59:40Z

core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java

+    }
+
+    @SuppressWarnings("unchecked")
+    List<TableIdentifier> viewIdentifiers = (List<TableIdentifier>) viewIdentifierObj;


Shouldn't we include a check into the above that not just check for List type but for List? Wouldn't this result in a ClassCastException or something similar if we input List that contains anything other than TableIdentifier?

gaborkaszab · 2026-01-13T12:00:36Z

core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java

+    }
+
+    // Format per REST API spec:
+    // - Namespace parts joined by namespace separator (0x1F, URL-encoded as %1F)


separator is configurable now, the comment now is invalid

gaborkaszab · 2026-01-13T12:01:46Z

core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java

+    // - Commas in view names must be percent-encoded as %2C
+    List<String> formattedViews = Lists.newArrayListWithCapacity(viewIdentifiers.size());
+    for (TableIdentifier viewId : viewIdentifiers) {
+      // Encode namespace using RESTUtil which handles the namespace separator


nit: I don't think these comments in the for loop are necessary. The code seems pretty straightforward and the longer comment above the loop also helps

gaborkaszab · 2026-01-13T12:11:03Z

core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java

+
  @Override
-  public Table loadTable(SessionContext context, TableIdentifier identifier) {
+  public Table loadTableWithContext(


Maybe this had been discussed already, but can't this be simply loadTable? The params will tell us anyway that this is with context.

github-actions bot added API spark core labels Sep 3, 2025

singhpk234 mentioned this pull request Sep 3, 2025

SPEC: Add referenced-by in loadTable API #13810

Open

singhpk234 marked this pull request as ready for review September 5, 2025 01:04

gaborkaszab reviewed Sep 5, 2025

View reviewed changes

nandorKollar reviewed Sep 5, 2025

View reviewed changes

api/src/main/java/org/apache/iceberg/catalog/ContextAwareTableCatalog.java Outdated Show resolved Hide resolved

gaborkaszab reviewed Sep 5, 2025

View reviewed changes

singhpk234 force-pushed the feature/ref-imp-referenced-by branch 2 times, most recently from 97296ab to bcec3f2 Compare September 8, 2025 00:22

nastra reviewed Sep 22, 2025

View reviewed changes

api/src/main/java/org/apache/iceberg/catalog/ContextAwareTableCatalog.java Outdated Show resolved Hide resolved

nastra reviewed Sep 22, 2025

View reviewed changes

core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java Outdated Show resolved Hide resolved

nastra reviewed Sep 22, 2025

View reviewed changes

...ain/scala/org/apache/spark/sql/catalyst/plans/logical/views/UnResolvedRelationFromView.scala Outdated Show resolved Hide resolved

sfc-gh-prsingh force-pushed the feature/ref-imp-referenced-by branch from a34c0f2 to fb6e824 Compare November 3, 2025 12:12

singhpk234 changed the title ~~Spark, Rest: Reference implementation of referenced-by in the loadTable call~~ [REST | SPARK]: Reference implementation of referenced-by in the loadTable call Nov 3, 2025

sfc-gh-prsingh force-pushed the feature/ref-imp-referenced-by branch from 20d30b3 to 737fc50 Compare November 3, 2025 12:31

github-actions bot added the stale label Dec 4, 2025

singhpk234 added not-stale and removed stale labels Dec 4, 2025

Reference implementation of referenced-by in the loadTable call

80aed3c

sfc-gh-prsingh force-pushed the feature/ref-imp-referenced-by branch 2 times, most recently from 3271d52 to f368355 Compare January 11, 2026 05:30

Address review feedback

57e3e30

sfc-gh-prsingh force-pushed the feature/ref-imp-referenced-by branch from f368355 to 57e3e30 Compare January 11, 2026 05:39

gaborkaszab reviewed Jan 13, 2026

View reviewed changes

[REST | SPARK]: Reference implementation of referenced-by in the loadTable call #13979

Are you sure you want to change the base?

[REST | SPARK]: Reference implementation of referenced-by in the loadTable call #13979

Uh oh!

Conversation

singhpk234 commented Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

About the change

Uh oh!

gaborkaszab left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

singhpk234 Sep 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

singhpk234 Sep 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Dec 4, 2025

Uh oh!

gaborkaszab left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

singhpk234 commented Sep 3, 2025 •

edited

Loading

singhpk234 Sep 8, 2025 •

edited

Loading

singhpk234 Sep 8, 2025 •

edited

Loading