New injections for forked elm packages

[lydell]

Summary

This adds https://github.com/lydell/elm-safe-virtual-dom to Lamdera, simplifies the internal implementation and allows for “proper” hot reloading during development in the future.

Details

I’m working on adding hot reloading and stopping of apps directly to elm/core. Read about it here:

https://github.com/lydell/core/blob/hot-reload-stop/javascript-interface.md#appstop

Branches:

• https://github.com/lydell/core/tree/lamdera (master -> safe -> hot-reload-stop -> lamdera)
• https://github.com/lydell/browser/tree/lamdera (master -> safe -> hot-reload-stop -> lamdera)
• https://github.com/lydell/virtual-dom/tree/hot-reload-stop (master -> safe -> hot-reload-stop)
• https://github.com/lydell/html/tree/safe (master -> safe)

As you can see above, elm/core has a lamdera branch:

• It makes it possible for Lamdera to inject some the things that differ between the four modes (NotLamdera, LamderaBackend, LamderaFrontend, LamderaLive).
• It includes the lamdera/containers patch directly so we don’t need to inject it.

elm/browser also has a lamdera branch:

• It includes three lines related to Browser.Navigation.Key.

Benefits of this PR:

• Everything is much simpler.
• We get the safe virtual DOM.
• We can add real hot reloading to lamdera live in a future PR.
• We can close Add support for lydell’s fork of elm/virtual-dom #40

TODO

• Install the forked packages as mentioned below. I’ve started this and done some good progress.
• Test.
• Maybe implement hot reloading as part of this PR after all. (See below.)

Installing the forked packages

Here’s the plan: Start out simple and make things more complicated in the future if needed.

1. Add package replacements as submodules.
2. Use TemplateHaskell to generate:
   • A map of packages that are replaced, to the versions that the replacements have. Used for validation.
   • A map of files to be replaced, to ByteString of alternate file content. Used to do the replacements.

Validate versions

In the TemplateHaskell-created map, each elm/* package that we replace specifies a version. That’s the version that the forked package was based on. In elm.json, there will still be lines like "elm/core": "1.0.5". What if the version in elm.json and the “fork version” do not match?

First, let’s go through the scenarios where the fork version is higher than the version specified in elm.json:

• elm.json major < fork major -> 🤷 This can’t happen. Only 1.x versions exists of all elm/* packages we replace. But we could code it to be a hard error.
• elm.json minor < fork minor -> 🤷 This can’t happen. Only 1.0.x versions exists of all elm/* packages we replace. But we could code it to be a hard error.
• elm.json patch < fork patch -> ✅ This is very likely to happen. Lots of people probably have "elm/virtual-dom": "1.0.3" in their elm.json, and haven’t bothered updating to the recent 1.0.4 version. If we made this a hard error, it would be annoying for lots of people. I think it’s better to simply allow the fork patch version to be greater than specified. If we want to, we could inform about this somehow (log message, or maybe the comments in the compiled JS shown below is enough?).

Then, let’s go through the opposite scenarios, where the fork version is lower than the version specified in elm.json:

• elm.json major > fork major -> ❌ It’s very unlikely that Evan will suddenly release 2.x of some package. But if that happens, this should be a hard error.
• elm.json minor > fork minor -> ❌ It’s unlikely that Evan will suddenly release 1.1.x of some package. But if that happens, this should be a hard error.
• elm.json patch > fork patch -> 🚨 This is somewhat likely. As mentioned, elm/virtual-dom 1.0.4 was recently released. If elm/virtual-dom 1.0.5 is released with a security fix, and a user tries to put "elm/virtual-dom": "1.0.5" in their elm.json, I think it should be a hard error, informing them that you can’t go above 1.0.4 with this release of the Lamdera compiler. (Silently using 1.0.4 anyway would be misleading, leading to a false sense of security.) They need to wait for a new Lamdera compiler version that has pulled in the security fix.

⚠️ We should make sure that lamdera init uses the known versions of elm/virtual-dom etc.

In the compiled JS, it might be nice adding comments for debugging:

// Lamdera vX.Y.Z-W: Using the following sources for some packages, rather than the official ones:
// https://github.com/lamdera/elm-core/commit/123
// https://github.com/lamdera/elm-virtual-dom/commit/456
// https://github.com/lamdera/elm-html/commit/789
// https://github.com/lamdera/elm-browser/commit/321

Replacements

When reading files from disk in ELM_HOME, check if the file is in the replacement maps. If so, go with that file content instead of the actual files.

This requires two changes to caches:

• In ELM_HOME there are artifacts.dat files, which Lamdera already call artifacts.x.dat. Now, they’ll be called for example artifacts.lamdera-1.3.3-0.19.1.dat. This is because we can’t trust an artifacts file made by a previous Lamdera version. It might have had different package replacements.
• In elm-stuff/ there’s usually a 0.19.1 folder with artifacts. Now there will be elm-stuff/lamdera-1.3.3-0.19.1/ instead. This is again because cached stuff from a previous Lamdera version cannot be trusted – they might contain the wrong package replacement code. (We don’ use the same approach for ELM_HOME – ~/.elm/lamdera-1.3.3-0.19.1/ – because IDE:s expect packages to be in ~/.elm/0.19.1/.)

Pros and cons

The benefit of having hard-coded (TemplateHaskell) maps of replacements is simplicity. We know that the provided packages are going to work with Lamdera and hot reloading.

The downside is that we need to make a new compiler release if there’s a hotfix in one of the replaced elm/* packages. However, that happens so infrequently that it feels safe to try out this approach.

How hot reload could work

Here’s how hot reload could work. I think we can do it as a follow-up PR, but I’m also open for doing it in this PR if somebody wants to pair on it.

The last commit of #29 adds two interesting things:

• A "j" WebSocket message, telling the client to hot reload. (It should only be used when the main type hasn’t changed.) It does that by adding a script tag pointing to the URL /_x/js.
• An endpoint for /_x/js.

If the endpoint returns JavaScript like this, hot reload should Just Work:

(function () {
    var scope = {};

    (function () {
        // Put the compiled Elm JavaScript code here. (Only the compiler output, no live.js or anything.)
    }).call(scope);

    Elm.hot.reload(scope);
})();

[lydell]

@supermario Has this ever happened in Bugsnag? Was it worth it? I have a feeling we don’t need this anymore, so I haven’t ported it. What do you think?

[supermario]

Just checked and nope, looks like we've never gotten this bugsnag yet... but this feels like one of those things that's not that big a deal until it happens, and then not having it will be very confusing. Perhaps just being overly paranoid. Happy to leave it for now 👍

[lydell]

I’ve pointed these to my repos while developing, but before merging I think it makes sense to have forks in github.com/lamdera/

[supermario]

I’m not super concerned to fork, but was there a practical reason for it? Like are you worried we might end up needing a Lamdera specific variant, or just a cheap hedge to make now just in case?