Add an undocumented general "run with RPC service"

[nalind]

What type of PR is this?

/kind feature

What this PR does / why we need it:

Add an undocumented general "run this with an RPC service" command, which right now doesn't do much, but provides some of the scaffolding we'll need should we want to expose any APIs that way.

How to verify it

New integration test!

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Note

[nalind]

/packit rebuild-failed

[Honny1]

LGTM

[nalind]

Rebased.

[giuseppe]

@nalind how will this be used? Do you already have any use case in mind?

I am playing with something similar in container-libs/storage to allow copying a layer while preserving reflinks (containers/container-libs#651) and @cgwalters suggested json-rpc for that.

Would it make sense to unify on the protocol (and dependencies)? @cgwalters would grpc work for composefs?

[cgwalters]

IPC protocols, always a fun topic. Obviously a lot of usage of gRPC in the ecosystem, though https://github.com/containerd/ttrpc is notable - and particularly containerd/ttrpc#75

gRPC (like base json-rpc) doesn't have any defined way to pass fds, though I'm sure we could invent one as the containerd folks were trying.

conmon-rs is using capnproto and then of course there's varlink for which I think systemd is still doing a lot with, but not sure about wider industry adoption. And DBus (which supports fd passing) but that's not cool anymore.

Then of course we have REST for the Docker protocol as implemented by podman too...

Of course json-rpc(+fdpass) doesn't have to be used for everything! One option is to negotiate its use over a side channel. (Though negotating that side channel is ugly in the general case, but I guess the simplest is "two fds", in a typical case probably each a socketpair())

---

Anyways I think whatever we do for a low level API in this space has to handle fd passing.

[cgwalters]

I'm kind of just repeating bootc-dev/bootc#522 here - and it looks like that hyper bug with HTTP over unix domain is still unfixed 😢

Basically gRPC over non-TCP|UDP is just not widely used (because really HTTP-over-non-IP is not widely used) But it could be fixed of course.

The appeal of jsonrpc is its bare-bones simplicity. For local communication a lot of the arguments for a binary protocol are lesser (same argument varlink makes for why JSON).

[cgwalters]

OK this data is Assisted-by: OpenCode (Opus 4.5) so take it with a bit of a grain of salt, but it all smells right.

Key takeaways:

• Rust is way, way, way better than Go on binary sizes
• For buildah and podman we're already pulling in the main grpc package indirectly and it seems podman is also now pulling it in directly for a noop service?

IPC Library Binary Size Comparison

Context: evaluating IPC options for buildah PR #6675
and related work in the containers ecosystem.

Go: jsonrpc-fdpass-go vs ttrpc vs gRPC

Context	Baseline	jsonrpc-fdpass	ttrpc	gRPC
Standalone	2.47 MB	+112 KB (+4.5%)	+5.2 MB (+216%)	+12.9 MB (+535%)
Buildah	45.41 MB	+0.2 KB (0.0005%)	+31 KB (0.07%)	+275 KB (0.62%)
Podman	61.85 MB	+2 KB (0.003%)	+37 KB (0.06%)	+37 KB (0.06%)

Rust: jsonrpc-fdpass vs tonic

All builds: release mode with LTO.

Context	Baseline	jsonrpc-fdpass	tonic (gRPC)
Standalone	382 KB	+307 KB (+80%)	+261 KB (+68%)
composefs-rs (cfsctl)	5.80 MB	+75 KB (1.3%)	+692 KB (11.9%)

Why existing projects absorb gRPC cheaply

In buildah and podman, the marginal cost of gRPC is small because
both already depend on it:

• buildah: ocicrypt/keywrap/keyprovider uses gRPC to talk to
  external key management services for encrypted container images
• podman: directly uses gRPC in its own API server
  (pkg/api/server/server.go multiplexes REST + gRPC on the same
  listener via HTTP/2 content-type routing; cmd/podman-testing is
  a gRPC client). Also pulls it in transitively via ocicrypt.
• ttrpc itself imports grpc/codes and grpc/status, so it
  does not avoid the grpc-go module

In Rust, composefs-rs has no existing gRPC dependency, so tonic's
full HTTP/2 stack (hyper, h2, tower, prost) adds ~692 KB. jsonrpc-fdpass
shares deps already present (tokio, serde), keeping overhead to ~75 KB.

fd passing

• jsonrpc-fdpass: native SCM_RIGHTS support built-in (both Go and Rust)
• ttrpc: exploring fd passing, not merged
• gRPC/tonic: no native fd passing; would need custom extension
• The hyper h2 PR #487 for
  Unix domain socket support remains unresolved

Go dependencies

Library	Key dependencies
jsonrpc-fdpass-go	golang.org/x/sys
ttrpc	google.golang.org/grpc (codes/status), protobuf, logrus
gRPC	google.golang.org/grpc, protobuf, genproto, golang.org/x/net

Rust dependencies

Library	Key dependencies
jsonrpc-fdpass	rustix, serde, jsonrpsee, tokio, tracing
tonic	tonic, hyper, h2, tower, prost, tokio

[cgwalters]

I am playing with something similar in container-libs/storage to allow copying a layer while preserving reflinks (containers/container-libs#651) and @cgwalters suggested json-rpc for that.I am playing with something similar in container-libs/storage to allow copying a layer while preserving reflinks (containers/container-libs#651) and @cgwalters suggested json-rpc for that.

I would elaborate on that slightly, see composefs/composefs-rs#228 (comment)

So I think a really key decision here is "IPC for container-libs" - and right now it's I think just the indirect dependency

image/copy → ocicrypt → ocicrypt/keywrap/keyprovider → google.golang.org/grpc

pulling gRPC in there (which honestly...is anyone using that?)

[cgwalters]

I also did bootc-dev/jsonrpc-fdpass#4
TL;DR systemd's varlink uses fdpassing quite a bit, but it's not officially part of the protocol, and not exposed by the bindings apart from the libsystemd C version. Which basically puts it in a similar boat as jsonrpc-fdpass really.

[nalind]

@nalind how will this be used? Do you already have any use case in mind?

Yes. There are build APIs which we'll need to provide for compose use cases, and they're expected to be exposed via grpc. As you've noted, podman gained a listener with a no-op handler in its server, but we'll need to be able to test anything that we provide for it to hook up to that listener.

[cgwalters]

Based on some research, gRPC + fd passing is just fundamentally in conflict (mostly due to HTTP/2's arbitrary interleaving of messages). There's so many layers to that abstraction. But it's not fatal: again we can use gRPC for most things, and negotiate say jsonrpc-fdpass over a separate socket.

BTW something also to consider here (and I definitely thought about when working on https://github.com/cgwalters/cstor-dist ) is that it would be I think a quite straightforward thing to have containers-storage implement the registry HTTP protocol - with extensions. I almost did that for the skopeo proxy but the thing I disliked the most is that HTTP/1.1 has head-of-line blocking which is bad for container image fetches, and HTTP/2 is basically starting to head towards gRPC anyways.

Anyways, also it looks like containerd is using grpc for snapshotters, so that's an argument for it.

So:

• Move grpc usage to https://github.com/containers/container-libs/tree/main/storage
• Scope out what it'd look like to expose an API similar to that of skopeo proxy there
• Scope out extension to gRPC that allows fd passing

(Here's an idea for the latter: if the very first content received on a grpc channel starts with say a NUL byte, that signals that that message should have an attached fd where we can also speak jsonrpc-fdpass)

?

[nalind]

I don't think this pull request is the place to have a wide-ranging conversation about IPC.

[cgwalters]

Fair, but where do you think we should have it? Or do you think we should just ignore the fact that one person is working on something related to IPC for containers in one format and there's a different one here?

[cgwalters]

@giuseppe After digging into this I think if we go gRPC for containers-libs/storage (which I'm OK with) by far the simplest thing is to have a separate jsonrpc-fdpass interface which is basically:

• OpenGRPC -> [fd]: thereafter speak gRPC over that socketpair
• ...other methods which do fd passing here

WDYT?

[giuseppe]

@giuseppe After digging into this I think if we go gRPC for containers-libs/storage (which I'm OK with) by far the simplest thing is to have a separate jsonrpc-fdpass interface which is basically:

• OpenGRPC -> [fd]: thereafter speak gRPC over that socketpair
• ...other methods which do fd passing here

WDYT?

I am fine either way.

the json-rpc protocol, as you said, is much simpler and maybe works better for low level functionalities.

We can workaround that if we have grpc.

@nalind is there any plan/desire to move the grpc server down to container-libs?

[nalind]

@nalind is there any plan/desire to move the grpc server down to container-libs?

I have no such plan or desire.

[cgwalters]

OK let's move then the discussion about the RPC need for reflink copies over to containers/container-libs#144 for now, and maybe anyone who is interested in sharing ideas/code for RPC in projects in containers/ space can join, and people who aren't can ignore it.