Separating out the Policy Store, Policy Reasoner and API

[Lut99]

This is the policy-reasoner counterpart to BraneFramework/brane#270

As discussed before (#42), the current co-dependency between the policy reasoner and the Brane repository is quite gnarly. This PR addresses this by completely refactoring the policy reasoner project into three parts:

1. The reasoner, i.e., the code that wraps around e.g. eFLINT, remains in this repository;
2. The store, i.e., the code that stores policies and manages versions, is moved to policy-store; and
3. The api, i.e., the interface to the reasoner, is moved to Brane itself.

As a side part of getting the whole project working again, implemented a fourth backend, the eFLINT Haskell interpreter. It is considered much more stable and complete than the previous "canonical" backend, the Go-implementation. It may be removed in a future release.

You can merge this PR first before you merge the counterpart at the Brane repo.

Fixes #39
Fixes #42

[Lut99]

Don't merge yet. Working on a final few fixes which you did but aren't completely taken over during the rebase. WIP until I fix this flippin' cannot-fetch-from-gitlab issue. It's really extremely weird that it's occurring here as well now. No clue what causes this behaviour. Did confirm it's Cargo somehow hanging on a git fetch, though.

[Lut99]

OK, should be mergable now! Fixed the gnarly git fetch error by avoiding the redirect in Gitlab. Seems like git2 has an issue with that now.

[codecov]

Codecov Report

Attention: Patch coverage is 36.31841% with 1024 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
lib/reasoners/eflint-haskell/src/trace.rs	76.21%	120 Missing and 17 partials ⚠️
lib/reasoners/eflint-haskell/src/hash.rs	26.63%	134 Missing and 1 partial ⚠️
lib/reasoners/eflint-haskell/src/reasonerconn.rs	0.00%	133 Missing ⚠️
lib/spec/src/auditlogger.rs	0.00%	99 Missing ⚠️
lib/reasoners/posix/src/reasonerconn.rs	0.00%	94 Missing ⚠️
lib/loggers/file/src/logger.rs	0.00%	75 Missing ⚠️
lib/reasoners/eflint-json/src/reasonerconn.rs	0.00%	68 Missing ⚠️
lib/share/src/lib.rs	0.00%	30 Missing ⚠️
lib/reasoners/eflint-haskell/src/reasons.rs	0.00%	29 Missing ⚠️
lib/eflint-to-json/src/lib.rs	0.00%	24 Missing ⚠️
... and 13 more

📢 Thoughts on this report? Let us know!

[Lut99]

Yay, finally passing all checks 🎉

[DanielVoogsgerd]

Pretty good, most of it is good stuff. Let's try to make it more idiomatic and maintainable.

[DanielVoogsgerd]

Is this you enjoying yourself or actually necessary? It is so error-prone and hard to read.

[Lut99]

Hahaha 😂 are they mutually exclusive?

No OK let me check what this actually is supposed to do xD

[Lut99]

Ah yes. I'd definitely say it's worth it's while. Remember that Rust has no way to abstract over tuples, so if you ever want to implement something for multiple tuples, you have to manually do the implementation for every tuple of size 0..n. This macro does all of those for you.

That said, this code is basically inspired by what I did over at Snack with the tuple-impls, which was a struggle at the time. However, as it happens, I was doing something like this again last night and accidentally ran into a MUCH SIMPLER way of doing this lmfao. For reference:

macro_rules! tuple_impl {
    // Private interface
    (__impl $($pi:literal),* : $fi:literal $(,$ni:literal)*) => {
        // Implementation goes here. Use $($pi,)* $fi to get 0, 1, ..., n, where
        // `n` is the size of the tuple - 1 (i.e., zero-indexed). You can also use
        // the `paste` crate to generate identifiers.

        // Recurse to implement size n + 1
        tuple_impl!(__impl $($pi,)* $fi : $($ni),*);
    };
    (__impl $($pi:literal),* :) => {};

    // Public interface
    ($fi:literal $(, $i:literal)* $(,)?) => {
        // We do this to avoid the user having to do as little as possible thinking;
        // we'll inject the `:` to separate between the "previous i's" and "future i's"
        // manually.
        tuple_impl!(__impl : $fi $(,$i)*);
    };
}

// Implement like:
// (Implements for tuples of size 1, 2, 3, ..., 8)
tuple_impl!(0, 1, 2, 3, 4, 5, 6, 7);

(yes that's the entire thing, two two macros are collapsed into one now)

[DanielVoogsgerd]

Ghmm, this gets me thinking: Should we optimize on the worker side or should the central node do the optimization and then force everybody to use that optimized workflow?

It feels a bit weird to optimize on here.

[Lut99]

That's a fair observation. I think you're right, in that in principle, the central node should do the optimization.

That said, the policy-reasoner version of a workflow is so much easier to optimize 😭 Plus, some of them (e.g., function inlining) done on the Brane one aren't just for fastness but also because that isn't supported in the checker version of a workflow.

Finally, note that eFLINT can be very slow, combinatorial-explosion-y. So there's incentive for workers to double-check that any workflows coming in are as small as possible. The question is, can they trust every workflow they get to be optimal.

Probably, the best way is: implement the optimizations as a library, then do it both at the central node and the worker node. If the central node does its job correctly, the worker node has little to do but scan the workflow once; if it doesn't, the worker can save time by investing some energy into optimization to relief (probably more) energy at the eFLINT step.

[Lut99]

Btw, the Brane version of a WIR is the b(r)ane of my existence. I wish I could do it all over again with what I learned, I think I can make it so much elegant, easier to work with and canonical with what I know from Rust now. Plus, we'll just ditch BraneScript harder than a coconut ~ we work with it because Brane started out that way, but really, it doesn't make a lot of sense. We're trying to express a graph, not a scripting language, so the whole compiler/workflow part of Brane has to bent over in a thousand different ways to make the script make an ounce of sense.

[Lut99]

Works on my machine :)

(The M-mac mini, that is)