Automate CHANGELOG creation

[bwplotka]

Proposal

We have to automate the process of preparing CHANGELOG section for the latest release. It's currently very error prone and extremely tedious, as the release shepherd has to manually go through all the commits, check if they added CHANGELOG item, if not, traverse to PR, double check if it shouldn't be in CHANGELOG, otherwise add and item with correct PR number. This is is even worse with our recent change to "merge" instead of squashing which products epically convoluted git trees that only seasoned git hackers can untangle 🙈 .

Another challenge with CHANGELOG items is that it produces merge conflicts on CHANGELOG.md file.

Finally, it's a bit sad we have to manually copy some changelog section to release page.

All of this are common problems for other projects e.g. Thanos. In many small libraries and projects maintainers gave up with changelog and just use "auto-generated" commit log that prints all on GH release page.

This ticket is to find a way to improve it.

AC

• There is automation that generates changelog section, no need for human work or review other than e.g. improve language/typos.
• No merge conflicts from changelog reporting
• Ideally, we use only release page?

Ideas

• Mandatory PR labels, PR title prefix/suffix, Description item that specifies if commit title should be part of CHANGELOG and with what type?
• Remove CHANGELOG.md file and use release pages only.
• Create automation that produces release page section.

[jesusvazquez]

`Mandatory PR labels, PR title prefix/suffix, Description item that specifies if commit title should be part of CHANGELOG and with what type?

I like how this sounds but I know I'd forget sometimes, if we could enforce this with a bot that would make our lifes easier.

Remove CHANGELOG.md file and use release pages only.

Are the conflicts of updating the changelog that frequent? I was also thinking about having an unreleased section, where we could enforce that any change should be mentioned there so that then the shepherd has a well defined list of things to curate.

[kfdm]

While I have not used it personally, there are some tools like towncrier that allow you to put a changelog entry in a folder and then run a 'compile' step before release to merge the fragments before release 🤔

[beorn7]

We had this discussion in the past, and back then, the outcome was that most people dealing with the CHANGELOG.md file felt, automation wasn't really improving their quality of life, as the entries had to be curated, verified, and re-worded anyway. Also, with the 6-week cadence, the size of the problem wasn't considered large enough to warrant automation.

No harm, of course, in revisiting the decision. People have left and joined the project in the meantime, and the nature of the work might have changed.

Last time I was a release shepherd, I didn't feel updating the changelog was a huge burden. In contrast, repositories that require me to update some "unreleased" section of the changelog or specific file felt like a burden each time I create a PR. Plus, the release shepherd in the end has to vet all those entries anyway, see above.

[bwplotka]

I agree we have to curate language. That's fine and easy. If we had all items in that changelog (or generated by some script) this is easy. I just don't want release shepherds to spend 2-3 hours to carefully go through hundreds of commits (my experience). It's not the job for humans (: Easy to make mistakes and takes time. On top of that reviewers would have to also spend same amount of time to verify this (!).

I agree we can't expect contributors to do this, and when they do, merge conflicts arise.

The PR labels, description or title method feels low cost, but it has to be enforced, otherwise we will forget about items which will trigger the need to traversing history.

I think there is still a room to create enforced policy that will at minimum tell us what PRs has to be in changelog. We can optimize then message and type of items with manual review later on. WDYT?

[roidelapluie]

In general I am using prometheus/promu#170 and I think we should finalize that pull request. It does mostly what we need and it is very quick to create a changelog.

[beorn7]

Hello from the bug scrub!

Note that @machine424 created #15191 is a related approach.

As this discussion is still ongoing and relevant, we'll leave this open.

[jkroepke]

I would share my experience with GitHub native Changelog generation.

https://docs.github.com/en/repositories/releasing-projects-on-github/automatically-generated-release-notes

The best thing at this point is that almost zero code approach (except for the PR validation workflow)

I'm using that concept already in multiple project, and it work well in combination with goreleaser (another tool for automation build and releases). It also works fine in combination with release branches.

It concept based on PR labels and PR titles. It's the maintainer responsibility label the PR correctly and GitHub Workflow will take care of that, if the PR is labeled. PR titles must be contain user-facing changes, unless the PR is labeled as chore (which can be used to exclude an PR from changelog). It might be fine, if a maintainer directly edit the PR title if the title is not usable for a changelog entry.

Based on labels, GitHub can categorized the Release notes.

For windows_exporter, creating a tag (even via Web UI) creates a release including notes for me.

Here is an example:

• Release Notes: https://github.com/prometheus-community/windows_exporter/releases/tag/v0.31.0-beta.0
• Release Configuration: https://github.com/prometheus-community/windows_exporter/blob/master/.github/release.yml
• Validation Workflow: https://github.com/prometheus-community/windows_exporter/blob/master/.github/workflows/pr-check.yaml

[machine424]

While I agree that a no-code approach is generally preferable, But I don’t think a PR title is the right place for a release-note entry. Release notes can be long which would make the titles unreadable, sometimes a single PR can introduce changes spanning two categories requiring two different entries (Even if we try to limit each PR to one category).

That's why I went for a "release entry within the PR body", not sure if any simple-to-integrate existing tool supports that?

There is also the fact that we may want to keep a CHANGELOG.md around,.

Of course, if other maintainers are willing to give the gh native tool or goreleaser a try, I'd have no problem with that.

(Regarding my PR, if some want to go that way, I can split that PR to only add the logic to check for the existence and validate that release notes within the PR body and leave the generation logic for later...)

[bwplotka]

Just to reference the alternative: OpenTelemetry have changelog directory with file per changelog item (~per PR): https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/.chloggen (tool).

[marwin1991]

I don't know if you are looking for such a tool, but you can check out https://logchange.dev/tools/logchange/getting-started/

Jinja templates should meet your needs.

Apache Solr using it exactly because of wasting time for pull requests conflicts

https://lists.apache.org/thread/4dotf4qx4ss3qr3xonv2y63v7wdg40nt