Support authentication for ORCA metrics channels

[twthorn]

Is your feature request related to a problem?

There is no way to include authentication information with a GRPC ORCA metrics channel since it's created with default call options. If we try to subscribe to ORCA metrics we get this error

ORCA reporting stream closed with Status{code=UNAUTHENTICATED, description=username and password must be provided, cause=null}

Describe the solution you'd like

We want clients to be able to pass authentication information. Or that the ORCA channel can inherit that authentication information from the main channel.

Describe alternatives you've considered

We could remove the authentication from the server, but this seems a bad practice from security standpoint. We set up the orca metrics server-side here

Additional context

[ejona86]

ORCA should already be using the same credentials as the channel. Are you sure you configured the channel with CallCredentials? What version of grpc-java are you using?

[twthorn]

We are using 1.75.0

We add the call credentials here to the stub for the channel.

We set up the channel with the custom load balancing policy here (this is feature branch I'm working on to support ORCA)

How can we properly pass the credentials on to ORCA?

[ejona86]

You need to pass the CallCredentials when creating the channel. Something like:

ChannelCredentials creds = InsecureChannelCredentials.create();
if (config.getVtgateUsername() != null && config.getVtgatePassword() != null) {
  CallCredentials callCreds = new StaticAuthCredentials(config.getVtgateUsername(), config.getVtgatePassword());
  creds = CompositeChannelCredentials.create(creds, callCreds);
}
ManagedChannel channel = Grpc.newChannelBuilderForAddress(config.getVtgateHost(), config.getVtgatePort(), creds)
   ...

You then would stop attaching the call creds to the stub, because the channel will already use the call creds.

[twthorn]

Thanks! That got the authentication working 🙌

I have a related question (let me know if I should ask elsewhere or if I missed the docs on this).

With our current code, we create a gRPC channel for streaming, runs until an error is received. By using weighted round robin, the initial connection is round robin, and those ORCA metrics are only used on subsequent requests with the same channel. Since this is a streaming gRPC call, there aren't additional requests over the channel. If an error occurs (or we set max connection age to try to force periodic rebalance), then the channel is deleted and a new channel is created. Since it's a new channel, it will lack ORCA metrics on start up, and will not use the metrics for proper load balancing (once again just uses round robin).

What is the correct way for allowing ORCA metrics to be used by streaming gRPC? Is ORCA compatible with such use cases? Can the LB policy wait until it has the ORCA report before connecting? Is the solution to recycle the connections so that we preserve the ORCA state? For this last part, I am not sure how to do that when server side a max connection age is enforced (it means the server terminates the channel, so it seems not possible to recycle).

Thank you again for the help on this.

[ejona86]

we set max connection age to try to force periodic rebalance

I would suggest you do that within the RPC on the server. Don't abuse the connection logic to regularly rebalance your RPC when doing L7 load balancing (like round robin). MAX_CONNECTION_AGE is for L4 load balancing.

then the channel is deleted and a new channel is created

I hope you mean connection.

it will lack ORCA metrics on start up, and will not use the metrics for proper load balancing

Because it'll be in the blackout period. I don't like that behavior of WRR, but I haven't been able to convince others that it doesn't make sense. (The assumption they have is that reconnecting might get a different machine, because there's an L4 LB. But in my mind it doesn't make sense to use L4 LB with WRR.) But this got brought up recently in grpc/proposal#498 (comment)

What is the correct way for allowing ORCA metrics to be used by streaming gRPC?

I'd say your server should periodically close the streaming RPC to allow for rebalancing. Don't force a reconnection. Even if we change the blackout period handling of WRR, I'd still discourage using MAX_CONNECTION_AGE for this purpose. I don't think my perspective has changed since https://youtu.be/Naonb2XD_2Q?t=334

[shivaspeaks]

Closing the issue for now. We can reopen this if author comes back.

[twthorn]

Thanks for the guidance @ejona86 . We plan to implement RPC-level rebalancing as suggested.

One idea that occurred to me: it seems like every long-running streaming RPC with L7 LB needs to implement the same pattern (interval + jitter + graceful close for rebalancing). At our scale (1000 client streams, 150 servers), this is important to do to avoid the mass connection churn for the idle connections (just receiving ORCA metrics, but not for a stream) by reusing max_connection_age/grace (although this would work, just inefficient, but it is tempting because of no additional implementation server-side).

Would the gRPC team consider a MaxStreamAge server option as a complement to MaxConnectionAge? This would:

• Handle the common pattern of "periodic stream rebalancing for ORCA/WRR"
• Work correctly with L7 LB (closes RPC, not connection)
• Prevent every application from re-implementing interval/jitter/status logic

We're happy to implement it ourselves for now, but this feels like a gap that affects many use cases (CDC streaming, Pub/Sub, Bigtable, etc.).

Let me know what you all think and thank you for the help on this!

[twthorn]

I opened a new issue #12575 with my proposed changes. @ejona86 @shivaspeaks if you can take a look when you get a chance, thank you!