Specify timestamp field with `timefield` in timechart command #4784

yuancu · 2025-11-12T06:05:16Z

Description

Allow users to specify a timestamp field instead of assuming there exists a @timestamp field when using timechart command.

For example, the following query is made possible:

source=events | timechart timefield=start_at span=1hour by category

Related Issues

Resolves #4576

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
New PPL command checklist all confirmed.
API changes companion pull request created.
Commits are signed per the DCO using --signoff or -s.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

… visitor Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

dai-chen · 2025-11-17T19:04:41Z

ppl/src/main/java/org/opensearch/sql/ppl/parser/AstBuilder.java

+        Optional.ofNullable(argMap.get("timefield"))
+            .map(l -> (String) l.getValue())
+            .orElse(OpenSearchConstants.IMPLICIT_FIELD_TIMESTAMP);
+    Field spanField = AstDSL.field(timeFieldName);


Could you check if we need to use this specified time field in transformPerFunction for per functions as well?

Yes, I think we need to use the specified timefield in per functions because otherwise they'll refer to the non-existing @timestamp field. I added an integration test for this case.

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

dai-chen

Thanks for the changes!

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

LantaoJin · 2025-11-26T05:25:22Z

non-blocking: @yuancu looks like this PR would impact the reverse enhancement #4775 more or less.

yuancu · 2025-11-26T06:32:28Z

non-blocking: @yuancu looks like this PR would impact the reverse enhancement #4775 more or less.

Yes it will. If this PR gets merged first, I'll help @ahkcs replace time field reference from hard-coded ones to the timefield variable.

opensearch-trigger-bot · 2025-11-26T06:44:57Z

The backport to 2.19-dev failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.19-dev 2.19-dev
# Navigate to the new working tree
pushd ../.worktrees/sql/backport-2.19-dev
# Create a new branch
git switch --create backport/backport-4784-to-2.19-dev
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 afc98dd6757288b8b269b78d8e2e5f323f78cae6
# Push it to GitHub
git push --set-upstream origin backport/backport-4784-to-2.19-dev
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.19-dev

Then, create a pull request where the base branch is 2.19-dev and the compare/head branch is backport/backport-4784-to-2.19-dev.

…arch-project#4784) * Support param timefield to specify span field in timechart Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Update doc to introduce timefield parameter Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Update ASTBuilderTest for chart: default args are handled in rel node visitor Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Fix ast expression builder test Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Fix anomanyzer test Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Support using specified timefield in per functions Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Omit by-timestamp clause in timechart command Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Mask timefield argument in anonymizer Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Anonymize argument span Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> --------- Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> (cherry picked from commit afc98dd) Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

…#4863) * Support param timefield to specify span field in timechart * Update doc to introduce timefield parameter * Update ASTBuilderTest for chart: default args are handled in rel node visitor * Fix ast expression builder test * Fix anomanyzer test * Support using specified timefield in per functions * Omit by-timestamp clause in timechart command * Mask timefield argument in anonymizer * Anonymize argument span --------- (cherry picked from commit afc98dd) Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

…arch-project#4784) * Support param timefield to specify span field in timechart Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Update doc to introduce timefield parameter Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Update ASTBuilderTest for chart: default args are handled in rel node visitor Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Fix ast expression builder test Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Fix anomanyzer test Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Support using specified timefield in per functions Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Omit by-timestamp clause in timechart command Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Mask timefield argument in anonymizer Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Anonymize argument span Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> --------- Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

yuancu added the enhancement New feature or request label Nov 12, 2025

yuancu changed the title ~~Use param timefield to specify span field in timechart command~~ Specify timestamp field for aggregation with param timefield in timechart command Nov 12, 2025

yuancu added 4 commits November 12, 2025 23:44

Support param timefield to specify span field in timechart

5c17695

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Update doc to introduce timefield parameter

328f987

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Update ASTBuilderTest for chart: default args are handled in rel node…

74c2eaf

… visitor Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Fix ast expression builder test

3f128a1

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

yuancu force-pushed the issues/4576 branch from a1d982d to 3f128a1 Compare November 12, 2025 15:45

Fix anomanyzer test

e07cd19

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

yuancu changed the title ~~Specify timestamp field for aggregation with param timefield in timechart command~~ Specify timestamp field with timefield in timechart command Nov 12, 2025

yuancu marked this pull request as ready for review November 12, 2025 16:33

yuancu requested review from GumpacG, LantaoJin, MaxKsyunz, Swiddis, YANG-DB, Yury-Fridlyand, acarbonetto, anirudha, dai-chen, derek-ho, forestmvey, joshuali925, kavithacm, mengweieric, noCharger, penghuo, ps48, seankao-az, vamsimanohar and ykmr1224 as code owners November 12, 2025 16:33

yuancu requested review from RyanL1997 and qianheng-aws as code owners November 12, 2025 16:33

yuancu added the backport 2.19-dev label Nov 13, 2025

dai-chen reviewed Nov 17, 2025

View reviewed changes

Support using specified timefield in per functions

4a8850a

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

dai-chen previously approved these changes Nov 18, 2025

View reviewed changes

Merge remote-tracking branch 'origin/main' into issues/4576

9073fae

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

yuancu dismissed dai-chen’s stale review via 9073fae November 19, 2025 12:54

yuancu added 3 commits November 24, 2025 14:04

Merge remote-tracking branch 'origin/main' into issues/4576

6adb081

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Omit by-timestamp clause in timechart command

9318f62

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Mask timefield argument in anonymizer

73040b8

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

yuancu force-pushed the issues/4576 branch from afdcde6 to 73040b8 Compare November 24, 2025 06:32

yuancu added 2 commits November 24, 2025 14:57

Anonymize argument span

bc7631e

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Merge remote-tracking branch 'origin/main' into issues/4576

f0ca094

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

LantaoJin approved these changes Nov 26, 2025

View reviewed changes

qianheng-aws approved these changes Nov 26, 2025

View reviewed changes

qianheng-aws merged commit afc98dd into opensearch-project:main Nov 26, 2025
35 checks passed

opensearch-trigger-bot bot added the backport-failed label Nov 26, 2025

yuancu deleted the issues/4576 branch November 26, 2025 07:25

This was referenced Nov 26, 2025

[Backport 2.19-dev] Specify timestamp field with timefield in timechart command (#4784) #4863

Merged

Implement reverse performance optimization #4775

Open

LantaoJin added the backport-manually Filed a PR to backport manually. label Nov 26, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Specify timestamp field with `timefield` in timechart command #4784

Specify timestamp field with `timefield` in timechart command #4784

Uh oh!

yuancu commented Nov 12, 2025 •

edited

Loading

Uh oh!

dai-chen Nov 17, 2025

Uh oh!

yuancu Nov 18, 2025

Uh oh!

dai-chen left a comment

Uh oh!

LantaoJin commented Nov 26, 2025

Uh oh!

yuancu commented Nov 26, 2025

Uh oh!

Uh oh!

opensearch-trigger-bot bot commented Nov 26, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Specify timestamp field with timefield in timechart command #4784

Specify timestamp field with timefield in timechart command #4784

Uh oh!

Conversation

yuancu commented Nov 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

dai-chen Nov 17, 2025

Choose a reason for hiding this comment

Uh oh!

yuancu Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

dai-chen left a comment

Choose a reason for hiding this comment

Uh oh!

LantaoJin commented Nov 26, 2025

Uh oh!

yuancu commented Nov 26, 2025

Uh oh!

Uh oh!

opensearch-trigger-bot bot commented Nov 26, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Specify timestamp field with `timefield` in timechart command #4784

Specify timestamp field with `timefield` in timechart command #4784

yuancu commented Nov 12, 2025 •

edited

Loading