Skip to content

[FEATURE] Add unified query evaluate API as PPL reference runtime #4894

@dai-chen

Description

@dai-chen

Is your feature request related to a problem?

Yes. Today, there is no engine-agnostic way to evaluate PPL queries outside of the tightly coupled OpenSearch PPL runtime. Consumers like:

  1. The SQL/PPL CLI, which wants to execute PPL queries either against OpenSearch DSL or local data sources like CSV files, but today has to talk directly to Calcite internals (execute, parse results etc);
  2. Query engines that want to execute PPL directly on top of their own storage/compute layer by embedding a lightweight evaluator (this is how OpenSearch PPL works today);
  3. Conformance test suites, which need a canonical “oracle” implementation to generate and verify expected results when testing PPL integrations (e.g., Spark PPL) against shared datasets and queries.

What solution would you like?

Following the existing Unified Query API design (UnifiedQueryPlanner, UnifiedQueryTranspiler), add a new Unified Query Evaluate API that provides a small, reusable reference runtime on top of Calcite.

The API should support:

  • Evaluating PPL queries against provided catalog;
  • Returning results in a simple, engine-agnostic row model (schema + row iterator);
  • Acting as the reference implementation for PPL semantics and functions.

What alternatives have you considered?

  • Each consumer implements its own PPL runtime: Without a unified evaluate API, every consumer has to talk to the existing PPL/Calcite internals directly. This duplicates effort makes it hard to keep PPL behavior consistent across consumers.
  • Transpile everything to SQL and delegate execution: This alone doesn’t satisfy the use cases above that need a lightweight, in-process runtime and a canonical reference implementation independent of any specific engine.

Do you have any additional context?

N/A

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

Status

New

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions