RFC: Execution Environments

[hone]

Readable

[buildpack-bot]

Maintainers,

As you review this RFC please queue up issues to be created using the following commands:

/queue-issue <repo> "<title>" [labels]...
/unqueue-issue <uid>

Issues

(none)

[jabrown85]

Can you add an example of a full project.toml that is used for producing a test image and a production image?

[hone]

Just for Project Descriptor?

[sambhav]

I'm having a difficulty trying to grasp how environment is being used by different toml files. @hone could you please provide examples of places where you imagine it being used and how that information will be leveraged by buildpacks/platform?

[jabrown85]

Can we perhaps get a conceptual example of what a buildpack author might do with this new execution environment information?

I have in my head a language buildpack author may...

• install test group dependencies
• create ruby-tests process / ruby-tests-verbose process
• Skip cleaning up things they might clean up otherwise for production images
• Set env vars to test mode operation (RAILS_ENV or similar)

[loewenstein]

Frankly, I don't get this proposal aligned with my mental model of what buildpacks do and how they help.

For me, container images are just another software artefact nowadays produced to ship software. Like a while back one was to produce JAR and/or WAR files that got delivered and finally deployed into some execution environment.
Buildpacks are a great tool to create those, but I don't see how or why I would create a test container that was separate from the production ones. I would want my tests to validate exactly the artefact that I am about to deliver. Otherwise, how do I know I checked the right thing.

From my POV buildpacks should focus on that task, creating the container image, and leave the other CI tasks to specialized tools.

Please help me to adjust my mental model, why's this added complexity in the buildpack spec worth it?

[jabrown85]

Please help me to adjust my mental model, why's this added complexity in the buildpack spec worth it?

I'm not the author but I can speak for my own mental model.

One use case is from the app developer side. Today they can pack build <img> to get a production artifact.

If we imagine the resulting image is ruby + nodejs, the developer has to setup both of those environments locally to develop and AGAIN on say GitHub Actions. The way those installations happen can differ. The GitHub action may install a different minor version of node or ruby to run the tests for instance.

Setting up CI with buildpacks could make this experience easier. pack test would use the same buildpacks and therefore test a more prod-like experience as the buildpack would install the same versions. This is especially important if your buildpack has options like DO_THIS_GARBAGE_COLLECTION_SETTING that you would have to know and replicate in your CI environment.

Another future-proofing thing here may be around future ARM support. Building and running an ARM test image via pack test seems tractable.

[natalieparellano]

How about "mode"?

[joshwlewis]

I like "context", but that's just as overloaded. How about "purpose = test" or "intent = test"?

[jkutner]

I think "env" was overloaded before we got here, and it's ok to use.

[natalieparellano]

Isn't this a lifecycle concern? i.e., when processing a group within an order, should it skip buildpacks that do not declare exec-env matching the desired env?

[hone]

When talking to @jabrown85, I thought the Platform/Builder provide the order.toml to lifecycle?

[hone]

From WG, lifecycle will decide which buildpacks to run based on the execution environment being passed along by the platform.

[natalieparellano]

Is this ever something we would want to spec? Having a consistent way for buildpacks to e.g., dump test output could help ensure portability across platforms.

[hone]

I could see us wanting to do that, but I wasn't sure how much we wanted to impose standards there.

[natalieparellano]

Are there any downsides to doing this? It would be more flexible and possibly avoid some duplication within orders (we could keep all as a special value)

[hone]

I think the only downside is what if a user wants to override in a way it wasn't intended? Is there a reason to lock stuff out?

[natalieparellano]

It would be nice to see the pack flow as well

[natalieparellano]

Should we add anything to the app image to designate it as being built for a particular environment? To avoid users accidentally deploying a test image in production...

I could see folks wanting to use the same tag when re-building a test image for production, in order to use previously cached dependencies.

[hone]

@natalieparellano that's a good question. Do you think cached dependencies should be shared b/t different execution environments?

[hone]

Is a label the best place to designate the execution environment?

[dmikusa]

IMO, it would be great if I could pack build --exec-env test image_foo and get image_foo:test automatically. If I then pack build --exec-env production image_foo, I'd get image_foo:production. If I want to change that, I could supply a tag and then pack should just use the tag I set, like pack build --exec-env test image_foo:my-test.

I would also, in a perfect world, like to see cache shared across different execution environments. It is extremely like that I'm using the same versions of Java, Node, etc... in both test & production so it would be a negative if we can't cache that and only download the language runtime once.

[hone]

For sharing cache, is it a problem if I run a test, get dev dependencies but my production build doesn't want them and prunes that cache. Next time I run a test I would need to install them. Do you just want to share cache for bootstrapping for every build?

[dmikusa]

I don't feel like that will be an issue for things like the JVM or Node.js runtime, but it sounds like it would be for other layers, like node modules or other things installed via package manager that would change between environments. That would certainly be annoying and not expected as a user.

Caching is per layer, what if the buildpack were to create different layers for cases like this? Maybe incorporate the exec env key into the layer name or something to make it unique? It wouldn't need to do this everywhere, but for things that are env dependent, it could have npm_modules-test and npm_modules-production layers.

If you ran a build for test and then a build for production, would that result in keeping both of those cached? or would it evict the test layer when you run the production build because the production image didn't use that layer?

[kanaksinghal]

I have a use-case for this.
In our company we use a variety of languages/stack and I wanted to write a generic CI pipeline where buildpacks can help us build the container image by detecting the stack but then to run unit test, I need to detect the stack again myself and run specific commands for nodejs/go/java etc.
In a typical CI pipeline we would do unit-test, run sonar quality checks, and build container image.

[jama22]

Really excited for this proposal, and I think there's a lot of interesting ways to make use of it.

In the context of a self-hosted PaaS
I've seen a common dynamic where the "operator" needs to establish fine grained controls over what the final containers look like. These configurations are also prone to change as the artifact itself moves into different environments (service connectors, certificates, environment variables, access to secrets managers, etc.)

On the developer side, most folks don't ever see production-like environments, so being able to debug their applications in production-like environments becomes incredibly difficult. In the past, I've seen some of these per-environment configurations being managed through GitOps and config files. This gives the user some control over managing the movement of the workload through the system, but even still they may not be able to build that container and interact with it directly

Moving the environment configuration opens up a lot of possibilities in this use case. I could image an operator encoding their per-environment configurations into the builder, and giving the developer the ability to simulate how their application will change in behavior across configurations.

For CI/CD
I think there have been some interesting ideas already discussed above and in the proposal. But I think a common use case that i can come up with is if you're building a library with buildpacks, you'll want to test on multiple OS-distros and architecture combinations (e.g. x86 vs win vs arm, ubuntu vs debian vs rhel). Those env configurations are commonly configured in CI itself or possibly externalized to some other config. Moving them back into the buildpack may provide for an improved UX

Changing what "prod-like" means
I think there's a pretty reasonable use case here where operators may want to give developers the ability to change what "prod-like" means. For example, the GCP buildpacks has GOOGLE_DEVMODE that can be enabled for faster changes. I can see it being used as a developer-oriented configuration while maintaining guardrails in prod-like envs

[jama22]

Similar to devmode, I just ran across live reloading in the Paketo python buildpacks https://paketo.io/docs/howto/python/#using-bp_live_reload_enabled

[jkutner]

Suggested change

	In addition, Builder Authors, Buildpack Authors, and App Developers will be able to specify various options to a specific execution enviroment using the `exec-env` key.
	In addition, Builder Authors, Buildpack Authors, and App Developers will be able to specify various options to a specific execution enviroment using the `exec-env` key in a `project.toml` file.

[hone]

There are now a few other files like builder.toml and launch.toml.

[jkutner]

Instead of defaulting to production, should we have a concept of all? Maybe this is similar to the exec-env having *?

[hone]

@jkutner do we need a concept of "all" for an individual lifecycle execution? In my head I liked the simplicity for a buildpack execution to know a single exec env you were targeting. I'm not sure how this would work in the current RFC. For instance, if I'm a buildpack author doing test execution environment I would install with test dependencies. If I'm building for production, I would likely prune those for that image. In order for us to make this work, we would need to have exec-env configuration in layer.toml then? How would lifecycle know what layers to export? If we're ok with a "fat" image that launcher would "construct" at runtime given a CNB_EXEC_ENV, we could make it work.

[hone]

Talking with @jabrown85 , I think compiled languages like Go might want to remove the source tree from /workspace for production but not test which would extra complicate things.

[hone]

Moving this to voting status with a close of next Thursday the 30th of Jan.

[jabrown85]

@hone the voting window is closed. There were no 👎 - feel free to approve and merge

[jabrown85]

@hone can you work on merging this? I think the scripts are broken..

[runesoerensen]

Happy to see this was approved and that the POC implementation is already well under way!

Just adding a few (somewhat pedantic) wording/spelling/phrasing suggestions for you to consider before merging :)

[hone]

Merged as part of 6ce039b