Inlining

[countvajhula]

Summary of Changes

WIP inlining of separately-defined flows.

Public Domain Dedication

• In contributing, we relinquish any copyright claims on our contributions and freely release them into the public domain in the simple hope that they will provide value.

(Why: The freely released, copyright-free work in this repository represents an investment in a better way of doing things called attribution-based economics. Attribution-based economics is based on the simple idea that we gain more by giving more, not by holding on to things that, truly, we could only create because we, in our turn, received from others. As it turns out, an economic system based on attribution -- where those who give more are more empowered -- is significantly more efficient than capitalism while also being stable and fair (unlike capitalism, on both counts), giving it transformative power to elevate the human condition and address the problems that face us today along with a host of others that have been intractable since the beginning. You can help make this a reality by releasing your work in the same way -- freely into the public domain in the simple hope of providing value. Learn more about attribution-based economics at drym.org, tell your friends, do your part.)

[michaelballantyne]

This PR breaks programs like the following, where a macro is defined below a flow that uses it:

#lang racket

(require qi)

(define-flow f (~> (pare sqr +) ▽))

(define-qi-syntax-rule (pare car-flo cdr-flo)
  (group 1 car-flo cdr-flo))

(~> (3 6 9) f)

Without inlining, this works because the expansion of the RHS of the flow is delayed until a second pass of module expansion. With your new code, expansion is triggered eagerly in define-flow.

[michaelballantyne]

With a little further thought though, I don't think you can both support the full mutual recursion behavior as exhibited in this program and get full inlining in all definition contexts, given the limitations of Racket's expansion. You could perhaps at module level with enough hacking, but I'm pretty sure it isn't possible in local definition contexts even with effects tricks. We can discuss more next time I join a Qi meeting.

[eutro]

Yeah I was also thinking that there's no way our current solution works wrt mutually recursive definitions (thanks for the counterexample). I also had the thought, could it be better to store the surface syntax and then expand it only when we try to inline it? (Possibly caching expansions and, if we do them, compiler optimisations & analyses)? We can cut out the #%host-expression issues entirely and I don't see any semantics issues if the expansion of the form is otherwise pure enough. We settled on the approach we have now for reasons I don't recall (I think because we do want to store some processed version), but since that has fundamental issues, we can also just rederive the expansion when we need it from the surface syntax.

[michaelballantyne]

One possible problem with storing unexpanded syntax is that you may expand the same syntax more than once, and thus execute side effects contained in macros more than once. Some bits of Racket do this sometimes, but generally it could create trouble. But caching expansions as you say could avoid that problem.

I'll have to think more deeply about whether there are other problems.

[michaelballantyne]

Storing unexpanded syntax would also lead syntax to expand in a different module than where it is written in the cross-module inlining case.

One potential issue would be if that syntax includes host expressions with references to names with associated contracts, I think those references might expand in such a way that the associated blame would refer to the module into which they were inlined, which would be wrong.

[countvajhula]

It sounds like we either pre-expand the syntax and undercut two-pass expansion, which totally breaks if there are mutually recursive flows, or we expand at inlining time and run afoul of an improper source module being inferred by the expander (which would affect contract blame, etc.). I wonder how the Racket compiler handles this kind of cross-module inlining. Or does it just not do that?

I suppose, in the second solution, of expanding first as part of inlining, it would be fine if we restrict it to just the current module, as we talked about last time. Would be nice if we could avoid that restriction eventually though.

Is there any way we could set some kind of parameter that would tell our invocation of expand-flow to consider that the expansion is occurring in a specified buffer (i.e., the source buffer, where the flow-to-be-expanded-and-inlined is defined)? Or could expand-flow accept a syntax object as "context" which could allow it to spoof the context of expansion?

[michaelballantyne]

I wonder how the Racket compiler handles this kind of cross-module inlining.

The Racket compiler does it as part of an additional pass after expansion. The problem is not that making it all work is in general impossible, but specifically that it is impossible given the restriction to 2 passes. And we're restricted to two passes because that's how many passes the Racket expander takes over definition contexts, and it doesn't give us the opportunity to take a 3rd (at least in local definition contexts).

Another possibility would be to restrict the form of mutual recursion that is allowed, while still allowing some. You could have a form with syntax:

(define-flows [name floe] ...)

where the flows defined in a single define-flows form can mutually-recur, but can otherwise only refer to things already declared above the define-flows form. This approach is easy to accomplish in syntax-spec via the host-interface/definitions form.

[countvajhula]

That may be OK, as the benefit would outweigh the cost I think. Dominik also had a thought re: provide transformers as solving an analogous problem. It's mentioned in the most recent notes, in case you think we might be able to use something there. Any chance you can attend this Wednesday? I think Eutro will be there too. Happy Fourth! 🎆