generated from amazon-archives/__template_Custom
-
Notifications
You must be signed in to change notification settings - Fork 181
Closed
Labels
enhancementNew feature or requestNew feature or request
Description
Is your feature request related to a problem?
Yes. Today, there is no engine-agnostic way to evaluate PPL queries outside of the tightly coupled OpenSearch PPL runtime. Consumers like:
- The SQL/PPL CLI, which wants to execute PPL queries either against OpenSearch DSL or local data sources like CSV files, but today has to talk directly to Calcite internals (execute, parse results etc);
- Query engines that want to execute PPL directly on top of their own storage/compute layer by embedding a lightweight evaluator (this is how OpenSearch PPL works today);
- Conformance test suites, which need a canonical “oracle” implementation to generate and verify expected results when testing PPL integrations (e.g., Spark PPL) against shared datasets and queries.
What solution would you like?
Following the existing Unified Query API design (UnifiedQueryPlanner, UnifiedQueryTranspiler), add a new Unified Query Evaluate API that provides a small, reusable reference runtime on top of Calcite.
The API should support:
- Evaluating PPL queries against provided catalog;
- Returning results in a simple, engine-agnostic row model (schema + row iterator);
- Acting as the reference implementation for PPL semantics and functions.
What alternatives have you considered?
- Each consumer implements its own PPL runtime: Without a unified evaluate API, every consumer has to talk to the existing PPL/Calcite internals directly. This duplicates effort makes it hard to keep PPL behavior consistent across consumers.
- Transpile everything to SQL and delegate execution: This alone doesn’t satisfy the use cases above that need a lightweight, in-process runtime and a canonical reference implementation independent of any specific engine.
Do you have any additional context?
N/A
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request
Type
Projects
Status
New