Enable brane build for Windows

[albatros13]

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is.

BRANE for Windows brane-windows-x86_64.exe tries to build packages for Linux.

brane build [path to .yml] produces the following error (tested on https://github.com/nightmare224/brane-programming-project)

It looks like it calls:

docker buildx build --output type=docker,dest=image.tar --tag preprocessing:1.0.0 --platform linux/x86_64 --build-arg BRANELET_ARCH=x86_64 --build-arg JUICEFS_ARCH=amd64

There is no documentation how to override the platform via options, the "brane build --arch" does not recognize windows/amd64 that docker buildx build would accept and no suitable options are listed in the documentation.

Describe the solution you'd like
A clear and concise description of what you want to happen.

Enable brane package building from Windows.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[DanielVoogsgerd]

Hi, thanks for reaching out. This is indeed broken, but I think this has little to do with windows itself. Your host operating system is windows, but the containers themselves run ubuntu.

Instead, this is a regression from the move from the epi-project organisation to braneframework. The 3.0.0 release still expects everything to be located in the old repository.

At some point I will try to dedicate some time to getting the old 3.0.0 version to work again, but I cannot give that priority right now. The main "selling point" of Brane at the moment is the policy reasoner integration (or at least that is what I think). And that is not available in 3.0.0.

We are working on getting a nightly release with pre-built binaries (and a 4.0.0 release), but that is stuck behind a big refactor right now #270.

For the time being, I can only recommend compiling the latest commit on the main branch from source.

Note that brane the binary (also known as brane-cli) is "supported" on Windows. But running the worker and central node isn't. You could try on WSL though!

We understand that this is still a quite rough right now, but we are steadily working on getting Brane more mature and easy to work with.

[albatros13]

Thanks for a quick reaction! I have a Linux machine for experimentation with brane, and I have a GCP VM to run brane nodes, but I also hoped to wrap into containers a project I have set up on my local Windows laptop and just run it with "brane for Windows". It is frustrating that a system that positions itself as an easy solution for cross-platform workflow management is not able to do that - what is even the point of "brane for Windows" then? I can build and run this code in a container

1. Create Dockerfile
2. docker buildx build . --output type=docker,dest=image.tar
3. docker load < image.tar
4. docker tag brane/preproc:test
5. docker run --rm -it brane/preproc:test /bin/bash

[DanielVoogsgerd]

I understand your frustration, I have had my fair share of Brane troubles since I have started working on it. Believe it or not, we have already come a long way since then, but the work is definitely not done. Brane is currently not production ready software, it is research software that is in development. I disagree with the notion that we position ourselves as an easy solution. If there is any particular remark that gives that impression, feel free to point it out so I can add some clarification.

Brane is not meant for cross-platform workflow management. Its aim is to provide a cross domain (as in different users, e.g. hospitals, research institutes) federated/collaborative computing framework. Where people can write relatively simple workflows to orchestrate computations in those different domains. Since our latest version (3.0.0), we have been working on integrating the policy reasoner as a way of providing multiple different privacy enforcing mechanisms such that those domains can express what information can be shared under which circumstances.
The policy reasoner provides two examples of ways of encoding this information, but most of all exposes an interface where domains can add their own software for making those privacy concerning decisions.

I am unsure what you mean with "Brane for windows", if that means the brane (brane-cli) scientist interface, its purpose is for scientists (that can work on Windows, Mac or Linux), to connect to existing brane instances (administered by a sysadmin on Linux).

[albatros13]

Thanks for clarifying the current status. I am aware that this is a research software and it is work in progress (you do not need to be defensive about it, I can close this issue if this is a distraction from the major development), but no papers I read ever mentioned Windows users are out :), and I was told that it works on Windows so I gave it a try. It runs, but it does not build the student project for brane (neither locally nor by deploying it on a remote node which I have created), from the position of view of a data scentist - please check the project link in my initial message - it has no deployment requirements or anything, it's a pure ML project which I wanted to test as per instructions "brane build pipeline.py; brane run pipelene.bs"...

[DanielVoogsgerd]

No need to close it. What you ran into is an actual regression. It is just not because of Windows, the same thing would happen on macos or linux currently.
The issue does not distract from the development. It is just not at the top of the todo list for the reasons mentioned in my previous comment.

With regards to Windows support. What you tried to do when running into the bug is completely supported behaviour on Windows (I think building packages should be supported, @Lut99 would know for sure).

Windows might not be our main focus, but we do support it for clients (scientist role). We do not support it for domains (at least not currently), mainly because interfacing with docker on multiple OSes is a bit of a pain.

I will get back to it when I have some time available.

[Lut99]

Hi! Indeed, Daniel is right, what you're doing should work on Windows. The problem you're running into is that it's trying to download a little binary that will be injected in the package you're trying to build: branelet, which is responsible for receiving information like function input arguments from Brane, decoding them and then exposing them to the user code inside the container. It's a compiled program, so to avoid the time it costs, it attempts to download a prebuilt version from our repo; however, every since the 3.0.0 release, we moved it (from https://github.com/epi-project/brane to https://github.com/BraneFramework/brane) - and we're still in the process of releasing 4.0.0. So any download done by the current version 3.0.0 is, as you observe, broken :/

The easiest way, probably, is to manually head over to this repo's release page for 3.0.0 (https://github.com/BraneFramework/brane/releases/tag/v3.0.0), download branelet-x86-64, and then supply the --init flag when building your package:

brane package build --init <path_to_branelet> <...rest of command>

Like Daniel mentioned, branelet runs in a Linux container, so you always need a Linux binary regardless of your OS. You just have to make sure your architecture matches (if you're running e.g. a macbook with an M-chip and you want to run the package on your machine, download branelet-aarch64 instead).

I get that Brane is very confusing - it's a truly large, complicated and very organic project (as in, it changes a lot). As a result, docs are always out of date and users perpetually confused - and the devs too xD However, it's already improved massively, like Daniel says (and because of Daniel's hard work), and we're still improving as we speak - so we're getting there!

On that note, if you encounter more weird and/or unexplained things, please keep making issues! If nothing else, it documents this problem for the next user to come along. Also, if you want to help to update the docs directly, you can also always make issues or pull requests at https://github.com/BraneFramework/manual ;)

[DanielVoogsgerd]

We are working on getting a nightly release with pre-built binaries (and a 4.0.0 release), but that is stuck behind a big refactor right now #270.

This has been resolved. There is a nightly release right now. @albatros13, it might not be your priority right now, but would you mind trying to run a demo on Windows using the nightly release. It is not a priority, but I still am really interested to know if it works or not.