NixOS: Flake fails with Chaotic Nyx CachyOS LTO kernel

[keenanweaver]

Using the yeetmouse flake on a system with the Chaotic Nyx build of the CachyOS kernel built with LLVM/LTO fails. I get this build error:

Running phase: unpackPhase
@nix { "action": "setPhase", "phase": "unpackPhase" }
unpacking source archive /nix/store/0qkvdd2jj9nxvy4cq8ky94q5xkwki0ps-source
source root is /build/source
Running phase: patchPhase
@nix { "action": "setPhase", "phase": "patchPhase" }
Running phase: updateAutotoolsGnuConfigScriptsPhase
@nix { "action": "setPhase", "phase": "updateAutotoolsGnuConfigScriptsPhase" }
Running phase: configurePhase
@nix { "action": "setPhase", "phase": "configurePhase" }
no configure script, doing nothing
Running phase: buildPhase
@nix { "action": "setPhase", "phase": "buildPhase" }
build flags: SHELL=/nix/store/rlq03x4cwf8zn73hxaxnx0zn5q9kifls-bash-5.3p3/bin/bash CC=/nix/store/kzq78n13l8w24jn8bx4djj79k5j717f1-gcc-14.3.0/bin/gcc LD=/nix/store/dc9vaz50jg7mibk9xvqw5dqv89cxzla3-binutils-2.44/bin/ld AR=/nix/store/vr15iyyykg9zai6fpgvhcgyw7gckl78w-gcc-wrapper-14.3.0/bin/ar NM=/nix/store/vr15iyyykg9zai6fpgvhcgyw7gckl78w-gcc-wrapper-14.3.0/bin/nm STRIP=/nix/store/dc9vaz50jg7mibk9xvqw5dqv89cxzla3-binutils-2.44/bin/strip OBJCOPY=/nix/store/vr15iyyykg9zai6fpgvhcgyw7gckl78w-gcc-wrapper-14.3.0/bin/objcopy OBJDUMP=/nix/store/vr15iyyykg9zai6fpgvhcgyw7gckl78w-gcc-wrapper-14.3.0/bin/objdump READELF=/nix/store/vr15iyyykg9zai6fpgvhcgyw7gckl78w-gcc-wrapper-14.3.0/bin/readelf HOSTCC=/nix/store/vr15iyyykg9zai6fpgvhcgyw7gckl78w-gcc-wrapper-14.3.0/bin/cc HOSTCXX=/nix/store/vr15iyyykg9zai6fpgvhcgyw7gckl78w-gcc-wrapper-14.3.0/bin/c++ HOSTAR=/nix/store/xwydcyvlsa3cvssk0y5llgdhlhjvmqdm-binutils-wrapper-2.44/bin/ar HOSTLD=/nix/store/xwydcyvlsa3cvssk0y5llgdhlhjvmqdm-binutils-wrapper-2.44/bin/ld ARCH=x86_64 CROSS_COMPILE= KBUILD_OUTPUT=/nix/store/ydaj8kmad8kq4c6jgc1m1a7z3yx58w62-linux-6.12.58-dev/lib/modules/6.12.58/build KBUILD_OUTPUT=/nix/store/dv452w8rywrjj99l5xx9vh64azx8xsyl-linux-x86_64-unknown-linux-gnu-6.17.8-dev/lib/modules/6.17.8-cachyos/build -C /nix/store/dv452w8rywrjj99l5xx9vh64azx8xsyl-linux-x86_64-unknown-linux-gnu-6.17.8-dev/lib/modules/6.17.8-cachyos/build M=\$\(sourceRoot\)/driver modules
make: Entering directory '/nix/store/dv452w8rywrjj99l5xx9vh64azx8xsyl-linux-x86_64-unknown-linux-gnu-6.17.8-dev/lib/modules/6.17.8-cachyos/build'
make[1]: Entering directory '/build/source/driver'
warning: the compiler differs from the one used to build the kernel
 The kernel was built by: clang version 21.1.2
 You are using:           gcc (GCC) 14.3.0
 CC [M]  accel.o
gcc: error: unrecognized command-line option '-mretpoline-external-thunk'
gcc: error: unrecognized command-line option '-fsplit-lto-unit'
make[3]: *** [/nix/store/dv452w8rywrjj99l5xx9vh64azx8xsyl-linux-x86_64-unknown-linux-gnu-6.17.8-dev/lib/modules/6.17.8-cachyos/source/scripts/Makefile.build:287: accel.o] Error 1
make[2]: *** [/nix/store/dv452w8rywrjj99l5xx9vh64azx8xsyl-linux-x86_64-unknown-linux-gnu-6.17.8-dev/lib/modules/6.17.8-cachyos/source/Makefile:2016: .] Error 2
make[1]: *** [/nix/store/dv452w8rywrjj99l5xx9vh64azx8xsyl-linux-x86_64-unknown-linux-gnu-6.17.8-dev/lib/modules/6.17.8-cachyos/source/Makefile:248: __sub-make] Error 2
make[1]: Leaving directory '/build/source/driver'
make: *** [/nix/store/dv452w8rywrjj99l5xx9vh64azx8xsyl-linux-x86_64-unknown-linux-gnu-6.17.8-dev/lib/modules/6.17.8-cachyos/source/Makefile:248: __sub-make] Error 2
make: Leaving directory '/nix/store/dv452w8rywrjj99l5xx9vh64azx8xsyl-linux-x86_64-unknown-linux-gnu-6.17.8-dev/lib/modules/6.17.8-cachyos/build'

How I'm using it: https://github.com/keenanweaver/nix-config/blob/8c2e58ff2ca03ed02c26f2e34095458a3440f720/hosts/desktop/default.nix#L43
Their package is here: https://github.com/chaotic-cx/nyx/tree/main/pkgs/linux-cachyos

Building the system with linuxPackages_cachyos-gcc works, so I think the derivation should be changed here (assuming Yeetmouse works with non-GCC compiled kernels).

[AndyFilter]

Hey, so unfortunately I don't know much about Nix (or the Nix module/flake) so I don't think I'll be able to help very much (at all).

From what I can see the fsplit-lto-unit flag is supported by Clang, but I can't find anything about gcc supporting it.

Building the system with linuxPackages_cachyos-gcc works (...)

I think it's best to stick to whatever works for You.

(...) so I think the derivation should be changed here

No idea what this means, but I assume You're referencing this:

YeetMouse/nix/package.nix

Lines 18 to 23 in 99844bb

	}: (stdenv.mkDerivation rec {
	pname = "yeetmouse";
	version = shortRev;
	src = lib.fileset.toSource {
	root = ./..;
	fileset = ./..;

If so, what changes do You suggest should be made here?

(assuming Yeetmouse works with non-GCC compiled kernels).

I don't know, I assume it should, but I'm an Ubuntu user, and I have no idea how to compile a kernel.

[keenanweaver]

Thanks for replying!

If so, what changes do You suggest should be made here?

I am unsure at this point. I will need to play around and pinpoint what aspect isn't working with the package. It's not really an issue, as I can just use the 'traditional' kernel build package instead. But it'd just be nice to use the optimized kernel built for LTO, as this is the only module I've come across that doesn't work with it.

Thanks for making this tool, by the way!

[AndyFilter]

Hey @keenanweaver, I just realised there is a PR (possibly) on this topic: #61. Could you perhaps take a look at it in your spare time and see if it helps?

[keenanweaver]

Somehow missed that PR. Tried it, but sadly still failing:

Running phase: unpackPhase
@nix { "action": "setPhase", "phase": "unpackPhase" }
unpacking source archive /nix/store/27a5l9ym0qccyi8nxs52159pbzw1q28v-source
source root is /build/source
Running phase: patchPhase
@nix { "action": "setPhase", "phase": "patchPhase" }
Running phase: updateAutotoolsGnuConfigScriptsPhase
@nix { "action": "setPhase", "phase": "updateAutotoolsGnuConfigScriptsPhase" }
Running phase: configurePhase
@nix { "action": "setPhase", "phase": "configurePhase" }
no configure script, doing nothing
Running phase: buildPhase
@nix { "action": "setPhase", "phase": "buildPhase" }
build flags: SHELL=/nix/store/rlq03x4cwf8zn73hxaxnx0zn5q9kifls-bash-5.3p3/bin/bash CC=/nix/store/kzq78n13l8w24jn8bx4djj79k5j717f1-gcc-14.3.0/bin/gcc LD=/nix/store/dc9vaz50jg7mibk9xvqw5dqv89cxzla3-binutils-2.44/bin/ld AR=/nix/store/vr15iyyykg9zai6fpgvhcgyw7gckl78w-gcc-wrapper-14.3.0/bin/ar NM=/nix/store/vr15iyyykg9zai6fpgvhcgyw7gckl78w-gcc-wrapper-14.3.0/bin/nm STRIP=/nix/store/dc9vaz50jg7mibk9xvqw5dqv89cxzla3-binutils-2.44/bin/strip OBJCOPY=/nix/store/vr15iyyykg9zai6fpgvhcgyw7gckl78w-gcc-wrapper-14.3.0/bin/objcopy OBJDUMP=/nix/store/vr15iyyykg9zai6fpgvhcgyw7gckl78w-gcc-wrapper-14.3.0/bin/objdump READELF=/nix/store/vr15iyyykg9zai6fpgvhcgyw7gckl78w-gcc-wrapper-14.3.0/bin/readelf HOSTCC=/nix/store/vr15iyyykg9zai6fpgvhcgyw7gckl78w-gcc-wrapper-14.3.0/bin/cc HOSTCXX=/nix/store/vr15iyyykg9zai6fpgvhcgyw7gckl78w-gcc-wrapper-14.3.0/bin/c++ HOSTAR=/nix/store/xwydcyvlsa3cvssk0y5llgdhlhjvmqdm-binutils-wrapper-2.44/bin/ar HOSTLD=/nix/store/xwydcyvlsa3cvssk0y5llgdhlhjvmqdm-binutils-wrapper-2.44/bin/ld ARCH=x86_64 CROSS_COMPILE= KBUILD_OUTPUT=/nix/store/z81f9m94hbijkw7sdxvknfhzyncpj8p2-linux-6.12.60-dev/lib/modules/6.12.60/build KBUILD_OUTPUT=/nix/store/b0mxbjvws1rs0v69bwif1fi9f6jw7xj4-linux-x86_64-unknown-linux-gnu-6.18.0-dev/lib/modules/6.18.0-cachyos/build -C /nix/store/b0mxbjvws1rs0v69bwif1fi9f6jw7xj4-linux-x86_64-unknown-linux-gnu-6.18.0-dev/lib/modules/6.18.0-cachyos/build M=\$\(sourceRoot\)/driver modules
make: Entering directory '/nix/store/b0mxbjvws1rs0v69bwif1fi9f6jw7xj4-linux-x86_64-unknown-linux-gnu-6.18.0-dev/lib/modules/6.18.0-cachyos/build'
make[1]: Entering directory '/build/source/driver'
warning: the compiler differs from the one used to build the kernel
  The kernel was built by: clang version 21.1.2
  You are using:           gcc (GCC) 14.3.0
  CC [M]  accel.o
gcc: error: unrecognized command-line option '-mstack-alignment=8'
gcc: error: unrecognized command-line option '-mretpoline-external-thunk'
gcc: error: unrecognized command-line option '-fsplit-lto-unit'
make[3]: *** [/nix/store/b0mxbjvws1rs0v69bwif1fi9f6jw7xj4-linux-x86_64-unknown-linux-gnu-6.18.0-dev/lib/modules/6.18.0-cachyos/source/scripts/Makefile.build:287: accel.o] Error 1
make[2]: *** [/nix/store/b0mxbjvws1rs0v69bwif1fi9f6jw7xj4-linux-x86_64-unknown-linux-gnu-6.18.0-dev/lib/modules/6.18.0-cachyos/source/Makefile:2016: .] Error 2
make[1]: *** [/nix/store/b0mxbjvws1rs0v69bwif1fi9f6jw7xj4-linux-x86_64-unknown-linux-gnu-6.18.0-dev/lib/modules/6.18.0-cachyos/source/Makefile:248: __sub-make] Error 2
make[1]: Leaving directory '/build/source/driver'
make: *** [/nix/store/b0mxbjvws1rs0v69bwif1fi9f6jw7xj4-linux-x86_64-unknown-linux-gnu-6.18.0-dev/lib/modules/6.18.0-cachyos/source/Makefile:248: __sub-make] Error 2
make: Leaving directory '/nix/store/b0mxbjvws1rs0v69bwif1fi9f6jw7xj4-linux-x86_64-unknown-linux-gnu-6.18.0-dev/lib/modules/6.18.0-cachyos/build'

Rebuilt my system with this input:

    yeetmouse = {
      url = "github:umbrageodotus/YeetMouse/patch-1?dir=nix";
      inputs.nixpkgs.follows = "nixpkgs";
    };

Using latest unstable NixOS channels.

[kitten]

It's tricky for me to guess what the intended way of deriving this from config.boot.kernelPackages.kernel is (haven't had a look and haven't had much time in a little bit), but you'd likely want to try playing with this:

YeetMouse/nix/package.nix

Line 10 in 0b103b1

kernelModuleMakeFlags ? pkgs.linuxPackages.kernelModuleMakeFlags,

This is where kernel is slotted in but kernelModuleMakeFlags isn't yet:

YeetMouse/nix/module.nix

Lines 465 to 467 in 20c193c

	yeetmouse = pkgs.yeetmouse.override {
	kernel = config.boot.kernelPackages.kernel;
	};

If the kernel is swapped out with the compiler having a different target for gcc, this will also need to be dynamically derived per kernel package.

Stands to reason that #61 is correct but needs kernelModuleMakeFlags from kernel. Presumably, a property will have been made available for this (?). Otherwise, this could be passed in manually, potentially

Edit: Worst case, copying this would be an option: https://github.com/NixOS/nixpkgs/blob/06a039f88598a4b9156cd52e2ef7bf65212fb76a/pkgs/top-level/linux-kernels.nix#L346-L348

Kind of as per: https://github.com/NixOS/nixpkgs/blob/06a039f88598a4b9156cd52e2ef7bf65212fb76a/nixos/tests/kernel-generic/default.nix#L22-L28

Edit 2: yup, pretty confident that self-defining it is correct. Since KBUILD_OUTPUT is already overridden/defined, we'd only need to swap out kernelModuleMakeFlags for kernel.commonMakeFlags and axe the kernelModuleMakeFlags input, i.e.

makeFlags = kernel.commonMakeFlags ++ [
  "KBUILD_OUTPUT=${kernel.dev}/lib/modules/${kernel.modDirVersion}/build"
  "-C"
  "${kernel.dev}/lib/modules/${kernel.modDirVersion}/build"
  "M=$(sourceRoot)/driver"
];