jackady dev command fails

[gotjoshua]

Describe the bug

./jackady -v dev -d

... all the build stuff is green up to here ...
🟢 Preparing daemon runtime ⏱️  4ms
🟢 Running the ipfsd binary /mnt/stripe/gotjosh/dev/nutria/prebuilts/x86_64-unknown-linux-gnu/ipfsd in /mnt/stripe/gotjosh/dev/nutria/builder/output/dev with --cfg /mnt/stripe/gotjosh/dev/nutria/defaults/ipfsd.toml
🔴 Command failed: Desktop command error: Io error: No such file or directory (os error 2)

if i run
/mnt/stripe/gotjosh/dev/nutria/prebuilts/x86_64-unknown-linux-gnu/ipfsd
i get:

fish: Job 1, '/mnt/stripe/gotjosh/dev/nutria/…' terminated by signal SIGSEGV (Address boundary error)

** Environment **

• Desktop Linux (NixOS)

To Reproduce
Steps to reproduce the behavior:
Follow instructions from readme

Expected behavior
the app to run in dev mode

Additional context
I assume some traditional linux libs/binaries are missing ( as is often the case with NixOS), but not sure how to find out which ones

[gotjoshua]

just noticed that the ipfsd is a prebuilt that seems to be built for debian...

i assume it expects things to be in a certain way, that just aren't in NixOS

[fabricedesre]

Hi, thanks for the report!

You're correct that ipfs is build on a debian based (ubuntu LTS) machine. Do you know what should be done to support NixOS?

[gotjoshua]

am i correct to assume that it is a build of your beetle fork?
https://github.com/capyloon/beetle

i see that there is a dockerfile for a distroless iroh-one
https://github.com/capyloon/beetle/blob/main/docker/Dockerfile.iroh-one

could be useful in these efforts

[fabricedesre]

Yes, this is a build of iroh-one from this beetle fork. If you can confirm that the docker build fixes the crash on Nix I'll switch to it.

[gotjoshua]

i wonder if adding it as an additional target could make sense...

i'll let you know if i manage to confirm that

[fabricedesre]

Is NixOS musl-based like Alpine?

[gotjoshua]

I think I can't really do justice to describing what nixos is. Minimalistic, deterministic, declarative...

This repo has quite a few links at the top
https://github.com/NixOS/mobile-nixos

I send that one as I think it could be a very interesting experiment to get capyloon running on a nixos based mobile device !

But first I'd like to see it in dev mode on my desktop... Thanks for your engagement!

[fabricedesre]

Btw I looked a the beetle docker files and they pull a rust image which is debian based. Not sure if this would produce a binary that works on NixOS

[gotjoshua]

From what I understood, it uses the debian distro to build and then creates a distroless and copies in the build results
Have a look a bit lower down:
FROM gcr.io/distroless/cc

Unfortunately I didn't get around to test before traveling, maybe over the weekend

[tennox]

Hi, a friend of @gotjoshua here. I'm not sure what causes the SIGSEGV, do you have an idea?

Missing gcc was my first thought, but that would look like this (in docker alpine):

/app # ./prebuilts/x86_64-unknown-linux-gnu/ipfsd
/bin/sh: ./prebuilts/x86_64-unknown-linux-gnu/ipfsd: not found

(note: in docker debian the same binary works fine)

Short NixOS explanation

Roughly speaking, in NixOS, instead of the OS providing dependencies (like gcc/protoc), each package specifies its' dependencies.
So statically linked is one way to not have external dependencies, or we'd need to install the the package the nix way, specifying a way to build it and its' dependencies - can explain more if interested & I already started a nix flake definition, but:

iroh-one

I managed to set up both: the linked docker image (although it was weirdly using COPY ../ which is not allowed in the docker I know, so I had to change that & some other things 😕) and a nix flake for building iroh-one, but I'm failing with both strategies at compiling the dependencies:

Details

   Compiling trust-dns-proto v0.22.0
   Compiling libp2p-quic v0.7.0-alpha
error[E0308]: mismatched types
  --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/libp2p-quic-0.7.0-alpha/src/connection/connecting.rs:85:44
   |
85 |                     return Poll::Ready(Ok((peer_id, self.connection.take().unwrap())));
   |                                            ^^^^^^^ expected `PeerId`, found `libp2p_identity::peer_id::PeerId`
   |
   = note: `libp2p_identity::peer_id::PeerId` and `PeerId` have similar names, but are actually distinct types
note: `libp2p_identity::peer_id::PeerId` is defined in crate `libp2p_identity`
  --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/libp2p-identity-0.1.3/src/peer_id.rs:50:1
   |
50 | pub struct PeerId {
   | ^^^^^^^^^^^^^^^^^
note: `PeerId` is defined in crate `libp2p_core`
  --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/libp2p-core-0.38.0/src/peer_id.rs:40:1
   |
40 | pub struct PeerId {
   | ^^^^^^^^^^^^^^^^^
error[E0308]: mismatched types

--> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/libp2p-quic-0.7.0-alpha/src/endpoint.rs:81:73

|

81 |         let client_tls_config = Arc::new(libp2p_tls::make_client_config(keypair, None).unwrap());

|                                          ------------------------------ ^^^^^^^ expected libp2p_identity::keypair::Keypair, found libp2p_core::identity::Keypair

|                                          |

|                                          arguments to this function are incorrect

|

= note: libp2p_core::identity::Keypair and libp2p_identity::keypair::Keypair have similar names, but are actually distinct types

note: libp2p_core::identity::Keypair is defined in crate libp2p_core

--> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/libp2p-core-0.38.0/src/identity.rs:68:1

|

68 | pub enum Keypair {

| ^^^^^^^^^^^^^^^^

note: libp2p_identity::keypair::Keypair is defined in crate libp2p_identity

--> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/libp2p-identity-0.1.3/src/keypair.rs:58:1

|

58 | pub enum Keypair {

| ^^^^^^^^^^^^^^^^

note: function defined here

--> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/libp2p-tls-0.1.0/src/lib.rs:43:8

|

43 | pub fn make_client_config(

|        ^^^^^^^^^^^^^^^^^^
error[E0308]: mismatched types

--> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/libp2p-quic-0.7.0-alpha/src/endpoint.rs:82:73

|

82 |         let server_tls_config = Arc::new(libp2p_tls::make_server_config(keypair).unwrap());

|                                          ------------------------------ ^^^^^^^ expected libp2p_identity::keypair::Keypair, found libp2p_core::identity::Keypair

|                                          |

|                                          arguments to this function are incorrect

|

= note: libp2p_core::identity::Keypair and libp2p_identity::keypair::Keypair have similar names, but are actually distinct types

note: libp2p_core::identity::Keypair is defined in crate libp2p_core

--> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/libp2p-core-0.38.0/src/identity.rs:68:1

|

68 | pub enum Keypair {

| ^^^^^^^^^^^^^^^^

note: libp2p_identity::keypair::Keypair is defined in crate libp2p_identity

--> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/libp2p-identity-0.1.3/src/keypair.rs:58:1

|

58 | pub enum Keypair {

| ^^^^^^^^^^^^^^^^

note: function defined here

--> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/libp2p-tls-0.1.0/src/lib.rs:65:8

|

65 | pub fn make_server_config(

|        ^^^^^^^^^^^^^^^^^^
For more information about this error, try rustc --explain E0308.

error: could not compile libp2p-quic (lib) due to 3 previous errors

warning: build failed, waiting for other jobs to finish...

The command '/bin/sh -c cargo build --bin iroh-one --profile=docker' returned a non-zero code: 101

I noticed there is no Cargo.lock committed, so maybe you might be using a different version?
Could you push that maybe? Or do you have different idea? 🤔

I can push a reproduction repo if that helps 💁‍♂️

[fabricedesre]

@tennox You need to be on the capyloon branch of the beetle repo, where Cargo.lock is commited along with all the crates vendored under ./third-party to make builds reproducible.

[fabricedesre]

Also, please build with --features=http-uds-gateway such as:

pushd iroh-one
cargo build --release  --features=http-uds-gateway
popd

[tennox]

That helped 😅 👍

managed to nix build the binary after figuring out & defining quite some dependencies 😓 (not an expert of dynamic linking and libs and so on)
.. will post details tomorrow