-
Notifications
You must be signed in to change notification settings - Fork 3.7k
[improve][broker] PIP-423: Add a new admin API to acknowledge a single message #23907
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Denovo1998
wants to merge
19
commits into
apache:master
Choose a base branch
from
Denovo1998:delay_msg_cancel
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+1,096
−3
Open
Changes from all commits
Commits
Show all changes
19 commits
Select commit
Hold shift + click to select a range
7afe161
Implementing delayed message cancellation in pulsar
Denovo1998 88ffdf9
feat(broker): Add support for canceling delayed messages in DelayedDe…
Denovo1998 212f174
add cancelDelayedMessage admin api.
Denovo1998 8686255
Merge branch 'master' into delay_msg_cancel
Denovo1998 6fe2b11
the cancel command does not need to be added to the sharedBucketPrior…
Denovo1998 dc079a6
fix test.
Denovo1998 f66f1b5
clean up useless canceledMessages.
Denovo1998 391a426
Implement the delayed message cancellation function through acknowled…
Denovo1998 d7f025c
feat(admin): add skipMessages by message IDs and remove cancelDelayed…
Denovo1998 5f170cc
use skipByMessageIds as the new path
Denovo1998 bce6ae3
Merge branch 'master' into delay_msg_cancel
Denovo1998 347e52f
feat: enhance skipMessages functionality to support batch indices
Denovo1998 a4911e5
feat: implement SkipMessageIdsRequest to support multiple formats for…
Denovo1998 2e77580
feat: refactor skipMessages to accept List<SkipEntry> and update rela…
Denovo1998 e7a988a
feat: add SkipEntry model for skipping messages with optional batch i…
Denovo1998 4613789
fix: update subscription description for clarity and add constructor …
Denovo1998 275a3cb
fix: remove unnecessary whitespace in SkipEntry.java
Denovo1998 a88ace7
Merge branch 'master' into delay_msg_cancel
Denovo1998 b5efe78
feat(broker): Support skipping messages by ID on partitioned topics
Denovo1998 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
201 changes: 201 additions & 0 deletions
201
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/SkipMessageIdsRequest.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,201 @@ | ||
| /* | ||
| * Licensed to the Apache Software Foundation (ASF) under one | ||
| * or more contributor license agreements. See the NOTICE file | ||
| * distributed with this work for additional information | ||
| * regarding copyright ownership. The ASF licenses this file | ||
| * to you under the Apache License, Version 2.0 (the | ||
| * "License"); you may not use this file except in compliance | ||
| * with the License. You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, | ||
| * software distributed under the License is distributed on an | ||
| * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| * KIND, either express or implied. See the License for the | ||
| * specific language governing permissions and limitations | ||
| * under the License. | ||
| */ | ||
| package org.apache.pulsar.broker.admin; | ||
|
|
||
| import com.fasterxml.jackson.core.JsonParser; | ||
| import com.fasterxml.jackson.core.ObjectCodec; | ||
| import com.fasterxml.jackson.databind.DeserializationContext; | ||
| import com.fasterxml.jackson.databind.JsonDeserializer; | ||
| import com.fasterxml.jackson.databind.JsonNode; | ||
| import com.fasterxml.jackson.databind.annotation.JsonDeserialize; | ||
| import com.fasterxml.jackson.databind.node.ArrayNode; | ||
| import com.fasterxml.jackson.databind.node.ObjectNode; | ||
| import io.netty.buffer.Unpooled; | ||
| import java.io.IOException; | ||
| import java.util.ArrayList; | ||
| import java.util.Base64; | ||
| import java.util.List; | ||
| import lombok.Getter; | ||
| import org.apache.pulsar.common.api.proto.MessageIdData; | ||
|
|
||
| /** | ||
| * Server-side request body for skipping messages by message IDs with support for multiple formats. | ||
| */ | ||
| @Getter | ||
| @JsonDeserialize(using = SkipMessageIdsRequest.Deserializer.class) | ||
| public class SkipMessageIdsRequest { | ||
| private final List<MessageIdItem> items = new ArrayList<>(); | ||
|
|
||
| public SkipMessageIdsRequest() { } | ||
|
|
||
| private void addItem(long ledgerId, long entryId, Integer batchIndex) { | ||
| items.add(new MessageIdItem(ledgerId, entryId, batchIndex)); | ||
| } | ||
|
|
||
| public record MessageIdItem(long ledgerId, long entryId, Integer batchIndex) { | ||
| public long getLedgerId() { | ||
| return ledgerId; | ||
| } | ||
|
|
||
| public long getEntryId() { | ||
| return entryId; | ||
| } | ||
|
|
||
| public Integer getBatchIndex() { | ||
| return batchIndex; | ||
| } | ||
| } | ||
|
|
||
| public static class Deserializer extends JsonDeserializer<SkipMessageIdsRequest> { | ||
| @Override | ||
| public SkipMessageIdsRequest deserialize(JsonParser p, DeserializationContext ctxt) throws IOException { | ||
| ObjectCodec codec = p.getCodec(); | ||
| JsonNode node = codec.readTree(p); | ||
| SkipMessageIdsRequest r = new SkipMessageIdsRequest(); | ||
|
|
||
| if (node == null || node.isNull()) { | ||
| throw new IOException("Invalid skipByMessageIds payload: empty body"); | ||
| } | ||
|
|
||
| if (node.isArray()) { | ||
| // Treat as default byteArray list | ||
| ArrayNode arr = (ArrayNode) node; | ||
| for (JsonNode idNode : arr) { | ||
| if (idNode != null && !idNode.isNull()) { | ||
| appendFromBase64(idNode.asText(), r); | ||
| } | ||
| } | ||
| return r; | ||
| } | ||
|
|
||
| if (node.isObject()) { | ||
| ObjectNode obj = (ObjectNode) node; | ||
| JsonNode typeNode = obj.get("type"); | ||
| String type = typeNode != null && !typeNode.isNull() ? typeNode.asText() : null; | ||
| JsonNode messageIdsNode = obj.get("messageIds"); | ||
|
|
||
| if (messageIdsNode != null) { | ||
| if (messageIdsNode.isArray()) { | ||
| ArrayNode arr = (ArrayNode) messageIdsNode; | ||
| if (type == null || type.isEmpty() || "byteArray".equalsIgnoreCase(type)) { | ||
| for (JsonNode idNode : arr) { | ||
| if (idNode != null && !idNode.isNull()) { | ||
| appendFromBase64(idNode.asText(), r); | ||
| } | ||
| } | ||
| } else if ("messageId".equalsIgnoreCase(type)) { | ||
| for (JsonNode idObj : arr) { | ||
| if (idObj == null || idObj.isNull()) { | ||
| continue; | ||
| } | ||
| if (!idObj.isObject()) { | ||
| throw new IOException("Invalid skipByMessageIds payload:" | ||
| + " messageIds elements must be objects"); | ||
| } | ||
| long ledgerId = requiredLong(idObj.get("ledgerId"), "ledgerId"); | ||
| long entryId = requiredLong(idObj.get("entryId"), "entryId"); | ||
| Integer batchIndex = optionalNonNegativeInt(idObj.get("batchIndex"), "batchIndex"); | ||
| r.addItem(ledgerId, entryId, batchIndex); | ||
| } | ||
| } else { | ||
| // Unknown type with array payload => reject | ||
| throw new IOException("Invalid skipByMessageIds payload: unsupported type for array"); | ||
| } | ||
| return r; | ||
| } else if (messageIdsNode.isObject()) { | ||
| throw new IOException("Invalid skipByMessageIds payload: messageIds must be an array"); | ||
| } else { | ||
| throw new IOException("Invalid skipByMessageIds payload: unsupported messageIds type"); | ||
| } | ||
| } | ||
|
|
||
| // No messageIds field => reject | ||
| throw new IOException("Invalid skipByMessageIds payload: missing messageIds"); | ||
| } | ||
|
|
||
| throw new IOException("Invalid skipByMessageIds payload: unsupported top-level JSON"); | ||
| } | ||
|
|
||
| private static long requiredLong(JsonNode node, String fieldName) throws IOException { | ||
| if (node == null || node.isNull()) { | ||
| throw new IOException("Invalid skipByMessageIds payload: missing " + fieldName); | ||
| } | ||
| try { | ||
| if (node.isNumber()) { | ||
| return node.longValue(); | ||
| } | ||
| if (node.isTextual()) { | ||
| return Long.parseLong(node.asText()); | ||
| } | ||
| } catch (Exception e) { | ||
| throw new IOException("Invalid skipByMessageIds payload: invalid " + fieldName, e); | ||
| } | ||
| throw new IOException("Invalid skipByMessageIds payload: invalid " + fieldName); | ||
| } | ||
|
|
||
| private static Integer optionalNonNegativeInt(JsonNode node, String fieldName) throws IOException { | ||
| if (node == null || node.isNull()) { | ||
| return null; | ||
| } | ||
| try { | ||
| int v; | ||
| if (node.isNumber()) { | ||
| v = node.intValue(); | ||
| } else if (node.isTextual()) { | ||
| v = Integer.parseInt(node.asText()); | ||
| } else { | ||
| throw new IOException("Invalid skipByMessageIds payload: invalid " + fieldName); | ||
| } | ||
| return v >= 0 ? v : null; | ||
| } catch (NumberFormatException e) { | ||
| throw new IOException("Invalid skipByMessageIds payload: invalid " + fieldName, e); | ||
| } | ||
| } | ||
|
|
||
| private static void appendFromBase64(String base64, SkipMessageIdsRequest r) | ||
| throws IOException { | ||
| if (base64 == null) { | ||
| return; | ||
| } | ||
| byte[] data; | ||
| try { | ||
| data = Base64.getDecoder().decode(base64); | ||
| } catch (IllegalArgumentException e) { | ||
| throw new IOException("Invalid skipByMessageIds payload: invalid base64 messageId", e); | ||
| } | ||
| if (data.length == 0) { | ||
| throw new IOException("Invalid skipByMessageIds payload: invalid base64 messageId (empty)"); | ||
| } | ||
| MessageIdData idData = new MessageIdData(); | ||
| try { | ||
| idData.parseFrom(Unpooled.wrappedBuffer(data, 0, data.length), data.length); | ||
| } catch (Exception e) { | ||
| throw new IOException(e); | ||
| } | ||
| long ledgerId = idData.getLedgerId(); | ||
| long entryId = idData.getEntryId(); | ||
| int batchIndex = idData.hasBatchIndex() ? idData.getBatchIndex() : -1; | ||
| if (batchIndex >= 0) { | ||
| r.addItem(ledgerId, entryId, batchIndex); | ||
| } else { | ||
| r.addItem(ledgerId, entryId, null); | ||
| } | ||
| } | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.