Skip to content

IGNITE-27700 SQL Calcite: Fix merge filter to scan after trimmed fields #12675

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
alex-plekhanov wants to merge 5 commits into from
Closed

IGNITE-27700 SQL Calcite: Fix merge filter to scan after trimmed fields #12675

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
3c06571
IGNITE-27700 SQL Calcite: Fix merge filter to scan after trimmed fields
alex-plekhanov Jan 29, 2026
63c682f
IGNITE-27700 SQL Calcite: Fix merge filter to scan after trimmed fields
alex-plekhanov Feb 2, 2026
bde8f0f
Merge remote-tracking branch 'apache/master' into ignite-27700
alex-plekhanov Feb 3, 2026
769ce7a
IGNITE-27700 SQL Calcite: Fix merge filter to scan after trimmed fields
alex-plekhanov Feb 3, 2026
c4242fc
IGNITE-27700 SQL Calcite: Fix merge filter to scan after trimmed fields
alex-plekhanov Feb 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 16 additions & 3 deletions ...va/org/apache/ignite/internal/processors/query/calcite/exec/exp/IgniteScalarFunction.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,28 +26,41 @@
* Implementation of {@link ScalarFunction} for Ignite user defined functions.
*/
public class IgniteScalarFunction extends IgniteReflectiveFunctionBase implements ScalarFunction {
/** */
private final boolean deterministic;

/**
* Private constructor.
*/
private IgniteScalarFunction(Method method, CallImplementor implementor) {
private IgniteScalarFunction(Method method, CallImplementor implementor, boolean deterministic) {
super(method, implementor);

this.deterministic = deterministic;
Copy link
Contributor

@zstan zstan Feb 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need mock? tests for such a case ? Can you exted java doc - what unexpected behavior will be raised if this flag missconfigured ?

Copy link
Contributor Author

@alex-plekhanov alex-plekhanov Feb 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test added.
What do you mean by the mock?
JavaDoc for public QuerySqlFunction annotation? I think it's already contains all required information. This annotation used for both query engines, so it's not correct to describe behavour for one of this engine in JavaDoc. Also this behavior can be extended later, for example if Calcite allow to change deterministic flag for SqlUserDefinedFunction class this flag will be processed also by Calcite and we can't control (and desrcibe on any change) all usages of this flag in Calcite code.

}

/**
* Creates {@link ScalarFunction} from given method.
*
* @param method Method that is used to implement the function.
* @param deterministic Is function deterministic.
* @return Created {@link ScalarFunction}.
*/
public static ScalarFunction create(Method method) {
public static ScalarFunction create(Method method, boolean deterministic) {
CallImplementor implementor = RexImpTable.createImplementor(
new ReflectiveCallNotNullImplementor(method), NullPolicy.NONE, false);

return new IgniteScalarFunction(method, implementor);
return new IgniteScalarFunction(method, implementor, deterministic);
}

/** {@inheritDoc} */
@Override public RelDataType getReturnType(RelDataTypeFactory typeFactory) {
return typeFactory.createJavaType(method.getReturnType());
}

/**
* @return Deterministic flag.
*/
public boolean isDeterministic() {
return deterministic;
}
}
28 changes: 27 additions & 1 deletion ...in/java/org/apache/ignite/internal/processors/query/calcite/exec/exp/RexExecutorImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,6 @@
import java.lang.reflect.Modifier;
import java.lang.reflect.Type;
import java.util.List;

import com.google.common.collect.ImmutableList;
import org.apache.calcite.DataContext;
import org.apache.calcite.adapter.java.JavaTypeFactory;
Expand All @@ -35,14 +34,19 @@
import org.apache.calcite.rel.type.RelDataType;
import org.apache.calcite.rel.type.RelDataTypeFactory;
import org.apache.calcite.rex.RexBuilder;
import org.apache.calcite.rex.RexCall;
import org.apache.calcite.rex.RexExecutable;
import org.apache.calcite.rex.RexExecutor;
import org.apache.calcite.rex.RexNode;
import org.apache.calcite.rex.RexProgram;
import org.apache.calcite.rex.RexProgramBuilder;
import org.apache.calcite.rex.RexVisitor;
import org.apache.calcite.rex.RexVisitorImpl;
import org.apache.calcite.sql.validate.SqlConformance;
import org.apache.calcite.sql.validate.SqlConformanceEnum;
import org.apache.calcite.sql.validate.SqlUserDefinedFunction;
import org.apache.calcite.util.BuiltInMethod;
import org.apache.calcite.util.ControlFlowException;
import org.apache.calcite.util.Util;
import org.apache.ignite.internal.processors.query.calcite.type.IgniteCustomType;

Expand Down Expand Up @@ -133,12 +137,34 @@ public static RexExecutable getExecutable(RexBuilder rexBuilder, List<RexNode> e
* Do constant reduction using generated code.
*/
@Override public void reduce(RexBuilder rexBuilder, List<RexNode> constExps, List<RexNode> reducedValues) {
RexVisitor<Void> nonDeterministicUdfFinder = new RexVisitorImpl<Void>(true) {
@Override public Void visitCall(RexCall call) {
if (call.getOperator() instanceof SqlUserDefinedFunction) {
SqlUserDefinedFunction udfFunc = (SqlUserDefinedFunction)call.getOperator();

if (udfFunc.getFunction() instanceof IgniteScalarFunction
&& !((IgniteScalarFunction)udfFunc.getFunction()).isDeterministic())
throw Util.FoundOne.NULL; // Don't reduce non-deterministic UDF functions.
}

return super.visitCall(call);
}
};

for (RexNode node : constExps) {
// Do not simplify custom types, since we can't convert it to literal of this type.
if (node.getType() instanceof IgniteCustomType) {
reducedValues.addAll(constExps);
return;
}

try {
node.accept(nonDeterministicUdfFinder);
}
catch (ControlFlowException foundNonDeterministic) {
reducedValues.addAll(constExps);
return;
}
}
final String code = compile(rexBuilder, constExps,
(list, index, storageType) -> {
Expand Down
4 changes: 2 additions & 2 deletions .../main/java/org/apache/ignite/internal/processors/query/calcite/prepare/PlannerHelper.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,10 +127,10 @@ public static IgniteRel optimize(SqlNode sqlNode, IgnitePlanner planner, IgniteL

rel = planner.extractConjunctionOverDisjunctionCommonPart(rel);

rel = planner.trimUnusedFields(root.withRel(rel)).rel;

rel = planner.transform(PlannerPhase.HEP_FILTER_PUSH_DOWN, rel.getTraitSet(), rel);

rel = planner.trimUnusedFields(root.withRel(rel)).rel;

// The following pushed down project can erase top-level hints. We store them to reassign hints for join nodes.
// Clear the inherit pathes to consider the hints as not propogated ones.
List<RelHint> topHints = HintUtils.allRelHints(rel).stream().map(h -> h.inheritPath.isEmpty()
Expand Down
1 change: 1 addition & 0 deletions ...c/main/java/org/apache/ignite/internal/processors/query/calcite/prepare/PlannerPhase.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,7 @@ public enum PlannerPhase {
@Override public RuleSet getRules(PlanningContext ctx) {
return ctx.rules(
RuleSets.ofList(
CoreRules.FILTER_REDUCE_EXPRESSIONS,
FilterScanMergeRule.TABLE_SCAN_SKIP_CORRELATED,

CoreRules.FILTER_MERGE,
Expand Down
4 changes: 3 additions & 1 deletion .../main/java/org/apache/ignite/internal/processors/query/calcite/rel/AbstractIndexScan.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
import org.apache.calcite.rel.RelInput;
import org.apache.calcite.rel.RelWriter;
import org.apache.calcite.rel.metadata.RelMetadataQuery;
import org.apache.calcite.rel.type.RelDataType;
import org.apache.calcite.rex.RexBuilder;
import org.apache.calcite.rex.RexNode;
import org.apache.calcite.rex.RexUtil;
Expand Down Expand Up @@ -67,12 +68,13 @@ protected AbstractIndexScan(
RelTraitSet traitSet,
RelOptTable table,
String idxName,
@Nullable RelDataType rowType,
@Nullable List<RexNode> proj,
@Nullable RexNode cond,
@Nullable List<SearchBounds> searchBounds,
@Nullable ImmutableBitSet reqColumns
) {
super(cluster, traitSet, Collections.emptyList(), table, proj, cond, reqColumns);
super(cluster, traitSet, Collections.emptyList(), table, rowType, proj, cond, reqColumns);

this.idxName = idxName;
this.searchBounds = searchBounds;
Expand Down
13 changes: 9 additions & 4 deletions ...rc/main/java/org/apache/ignite/internal/processors/query/calcite/rel/IgniteIndexScan.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
import org.apache.calcite.rel.RelCollation;
import org.apache.calcite.rel.RelInput;
import org.apache.calcite.rel.RelWriter;
import org.apache.calcite.rel.type.RelDataType;
import org.apache.calcite.rex.RexNode;
import org.apache.calcite.util.ImmutableBitSet;
import org.apache.ignite.internal.processors.query.calcite.prepare.bounds.SearchBounds;
Expand Down Expand Up @@ -64,6 +65,7 @@ public IgniteIndexScan(RelInput input) {
* @param traits Traits of this relational expression
* @param tbl Table definition.
* @param idxName Index name.
* @param rowType Row type.
* @param proj Projects.
* @param cond Filters.
* @param requiredCols Participating columns.
Expand All @@ -74,13 +76,14 @@ public IgniteIndexScan(
RelTraitSet traits,
RelOptTable tbl,
String idxName,
@Nullable RelDataType rowType,
@Nullable List<RexNode> proj,
@Nullable RexNode cond,
@Nullable List<SearchBounds> searchBounds,
@Nullable ImmutableBitSet requiredCols,
RelCollation collation
) {
this(-1L, cluster, traits, tbl, idxName, proj, cond, searchBounds, requiredCols, collation);
this(-1L, cluster, traits, tbl, idxName, rowType, proj, cond, searchBounds, requiredCols, collation);
}

/**
Expand All @@ -89,6 +92,7 @@ public IgniteIndexScan(
* @param traits Traits of this relational expression
* @param tbl Table definition.
* @param idxName Index name.
* @param rowType Row type.
* @param proj Projects.
* @param cond Filters.
* @param requiredCols Participating colunms.
Expand All @@ -100,13 +104,14 @@ private IgniteIndexScan(
RelTraitSet traits,
RelOptTable tbl,
String idxName,
@Nullable RelDataType rowType,
@Nullable List<RexNode> proj,
@Nullable RexNode cond,
@Nullable List<SearchBounds> searchBounds,
@Nullable ImmutableBitSet requiredCols,
RelCollation collation
) {
super(cluster, traits, tbl, idxName, proj, cond, searchBounds, requiredCols);
super(cluster, traits, tbl, idxName, rowType, proj, cond, searchBounds, requiredCols);

this.sourceId = sourceId;
this.collation = collation;
Expand All @@ -132,13 +137,13 @@ private IgniteIndexScan(
/** {@inheritDoc} */
@Override public IgniteRel clone(long sourceId) {
return new IgniteIndexScan(sourceId, getCluster(), getTraitSet(), getTable(),
idxName, projects, condition, searchBounds, requiredColumns, collation);
idxName, rowType, projects, condition, searchBounds, requiredColumns, collation);
}

/** {@inheritDoc} */
@Override public IgniteRel clone(RelOptCluster cluster, List<IgniteRel> inputs) {
return new IgniteIndexScan(sourceId, cluster, getTraitSet(), getTable(),
idxName, projects, condition, searchBounds, requiredColumns, collation);
idxName, rowType, projects, condition, searchBounds, requiredColumns, collation);
}

/** {@inheritDoc} */
Expand Down
17 changes: 12 additions & 5 deletions ...rc/main/java/org/apache/ignite/internal/processors/query/calcite/rel/IgniteTableScan.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
import org.apache.calcite.plan.RelTraitSet;
import org.apache.calcite.rel.RelInput;
import org.apache.calcite.rel.RelWriter;
import org.apache.calcite.rel.type.RelDataType;
import org.apache.calcite.rex.RexNode;
import org.apache.calcite.util.ImmutableBitSet;
import org.jetbrains.annotations.Nullable;
Expand Down Expand Up @@ -63,14 +64,15 @@ public IgniteTableScan(
RelTraitSet traits,
RelOptTable tbl
) {
this(cluster, traits, tbl, null, null, null);
this(cluster, traits, tbl, null, null, null, null);
}

/**
* Creates a TableScan.
* @param cluster Cluster that this relational expression belongs to
* @param traits Traits of this relational expression
* @param tbl Table definition.
* @param rowType Row type.
* @param proj Projects.
* @param cond Filters.
* @param requiredColunms Participating colunms.
Expand All @@ -79,18 +81,20 @@ public IgniteTableScan(
RelOptCluster cluster,
RelTraitSet traits,
RelOptTable tbl,
@Nullable RelDataType rowType,
@Nullable List<RexNode> proj,
@Nullable RexNode cond,
@Nullable ImmutableBitSet requiredColunms
) {
this(-1L, cluster, traits, tbl, proj, cond, requiredColunms);
this(-1L, cluster, traits, tbl, rowType, proj, cond, requiredColunms);
}

/**
* Creates a TableScan.
* @param cluster Cluster that this relational expression belongs to
* @param traits Traits of this relational expression
* @param tbl Table definition.
* @param rowType Row type.
* @param proj Projects.
* @param cond Filters.
* @param requiredColunms Participating colunms.
Expand All @@ -100,11 +104,12 @@ private IgniteTableScan(
RelOptCluster cluster,
RelTraitSet traits,
RelOptTable tbl,
@Nullable RelDataType rowType,
@Nullable List<RexNode> proj,
@Nullable RexNode cond,
@Nullable ImmutableBitSet requiredColunms
) {
super(cluster, traits, ImmutableList.of(), tbl, proj, cond, requiredColunms);
super(cluster, traits, ImmutableList.of(), tbl, rowType, proj, cond, requiredColunms);
this.sourceId = sourceId;
}

Expand All @@ -126,11 +131,13 @@ private IgniteTableScan(

/** {@inheritDoc} */
@Override public IgniteRel clone(long sourceId) {
return new IgniteTableScan(sourceId, getCluster(), getTraitSet(), getTable(), projects, condition, requiredColumns);
return new IgniteTableScan(sourceId, getCluster(), getTraitSet(), getTable(), rowType, projects, condition,
requiredColumns);
}

/** {@inheritDoc} */
@Override public IgniteRel clone(RelOptCluster cluster, List<IgniteRel> inputs) {
return new IgniteTableScan(sourceId, cluster, getTraitSet(), getTable(), projects, condition, requiredColumns);
return new IgniteTableScan(sourceId, cluster, getTraitSet(), getTable(), rowType, projects, condition,
requiredColumns);
}
}
13 changes: 9 additions & 4 deletions ...g/apache/ignite/internal/processors/query/calcite/rel/ProjectableFilterableTableScan.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,12 +75,16 @@ protected ProjectableFilterableTableScan(
RelTraitSet traitSet,
List<RelHint> hints,
RelOptTable table,
@Nullable RelDataType rowType,
@Nullable List<RexNode> proj,
@Nullable RexNode cond,
@Nullable ImmutableBitSet reqColumns
) {
super(cluster, traitSet, hints, table);

assert proj == null || rowType != null : "rowType should be provided if project != null";

this.rowType = rowType;
projects = proj;
condition = cond;
requiredColumns = reqColumns;
Expand All @@ -90,6 +94,7 @@ protected ProjectableFilterableTableScan(
protected ProjectableFilterableTableScan(RelInput input) {
super(input);
condition = input.getExpression("filters");
rowType = input.get("rowType") == null ? null : input.getRowType("rowType");
projects = input.get("projects") == null ? null : input.getExpressionList("projects");
requiredColumns = input.get("requiredColumns") == null ? null : input.getBitSet("requiredColumns");
}
Expand Down Expand Up @@ -129,6 +134,7 @@ protected RelWriter explainTerms0(RelWriter pw) {
}

return pw
.itemIf("rowType", rowType, projects != null) // Intentional project check here.
.itemIf("projects", projects, projects != null)
.itemIf("requiredColumns", requiredColumns, requiredColumns != null);
}
Expand All @@ -151,10 +157,9 @@ protected RelWriter explainTerms0(RelWriter pw) {

/** {@inheritDoc} */
@Override public RelDataType deriveRowType() {
if (projects != null)
return RexUtil.createStructType(Commons.typeFactory(getCluster()), projects);
else
return table.unwrap(IgniteTable.class).getRowType(Commons.typeFactory(getCluster()), requiredColumns);
assert projects == null : "For merged projects rowType should be provided explicetely";

return table.unwrap(IgniteTable.class).getRowType(Commons.typeFactory(getCluster()), requiredColumns);
}

/** */
Expand Down
7 changes: 6 additions & 1 deletion ...g/apache/ignite/internal/processors/query/calcite/rel/logical/IgniteLogicalIndexScan.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
import org.apache.calcite.plan.RelOptCluster;
import org.apache.calcite.plan.RelOptTable;
import org.apache.calcite.plan.RelTraitSet;
import org.apache.calcite.rel.type.RelDataType;
import org.apache.calcite.rex.RexNode;
import org.apache.calcite.util.ImmutableBitSet;
import org.apache.ignite.internal.processors.query.calcite.prepare.bounds.SearchBounds;
Expand All @@ -37,6 +38,7 @@ public static IgniteLogicalIndexScan create(
RelTraitSet traits,
RelOptTable table,
String idxName,
@Nullable RelDataType rowType,
@Nullable List<RexNode> proj,
@Nullable RexNode cond,
@Nullable ImmutableBitSet requiredColumns
Expand All @@ -51,6 +53,7 @@ public static IgniteLogicalIndexScan create(
traits,
table,
idxName,
rowType,
proj,
cond,
searchBounds,
Expand All @@ -63,6 +66,7 @@ public static IgniteLogicalIndexScan create(
* @param traits Traits of this relational expression
* @param tbl Table definition.
* @param idxName Index name.
* @param rowType Row type.
* @param proj Projects.
* @param cond Filters.
* @param searchBounds Index search bounds.
Expand All @@ -73,11 +77,12 @@ private IgniteLogicalIndexScan(
RelTraitSet traits,
RelOptTable tbl,
String idxName,
@Nullable RelDataType rowType,
@Nullable List<RexNode> proj,
@Nullable RexNode cond,
@Nullable List<SearchBounds> searchBounds,
@Nullable ImmutableBitSet requiredCols
) {
super(cluster, traits, tbl, idxName, proj, cond, searchBounds, requiredCols);
super(cluster, traits, tbl, idxName, rowType, proj, cond, searchBounds, requiredCols);
}
}
Loading
Loading