Concerns regarding recipe repository "overlays"

[scymtym]

Paraphrasing discussion with semeyerz. @semeyerz please correct and/or expand as necessary
cc @rhaschke @swrede @LeroyR @warp1337 @nexero

Concerns

Keeping recipes in multiple unrelated (as in not forks of a single one) repositories is complicated and impractical:

• Hard to understand for non-expert users
• How to make/test changes to core recipes?
  • Clone core recipes, make the change, change my repository to work with my modified core recipes, make pull-request, undo temporary changes?
  • Overwrite recipes in my recipe repository, make and test changes, clone core recipes, make pull-request, undo overwrite?
• Harder to find things using e.g. grep

We don't have enough resources to develop, document and maintain tool-supported features like this. We should focus on solutions that can be implemented with standard tools (i.e. git, grep, etc.)

Semantics are unclear

• When including a template from a project recipe, in which repository should the search process start?
• How should overwriting recipes work with multiple parents/a hierarchy of ancestors (See Support repository "overlays" generator#30)

Would overlays instead of underlays work better?

Advantages of single-repository solution

• Standard workflows (fork, merge, pull-request)
• Works better with commandline/standard tools

Better for new external users as well current Unibi users

Simple Migration Proposal

1. Clean up templates in opensource.cit-ec
2. Put cleaned up templates into new rdtk/recipes-core repository
3. Fork rdtk/recipes-core into rdtk/recipes-unibi
4. Copy project and distribution recipes from opensource.cit-ec into rdtk/recipes-unibi
5. New external users as well as Unibi users work with rdtk/recipes-unibi
6. Synchronize (template) improvements between rdtk/recipes-core and rdtk/recipes-unibi using pull-requests

[rhaschke]

In the discussions, we all agreed that a modular, hierarchically-organized overlay mechanism is required to easily benefit from external repos. A modular structure has several advantages:

• All dependent repos can benefit from improvements in the core repo - without the need to update and merge. Obviously this also puts some burden on the quality of the commited code there, which could be seen as a con as well 😉
• We can easily define some semantically-grouped repos: core, robotics, vision, etc., which can be maintained by different groups of people.

How to make/test changes to core recipes?

• Checkout a fork of the core recipes, apply and PR your changes. Until they are merged, use your fork as the underlay. This temporary change of the parent isn't that complicated to handle, is it?

We don't have enough resources to develop, document and maintain tool-supported features like this.

Which kind of tool-support are you expecting? grep and git can still be used. If you place your various repos into an overarching folder, you can run grep across all repos at once.

Semantics are unclear.

The semantics was well-defined in RDTK/generator#30, wasn't it?

Would overlays instead of underlays work better?

Isn't that the same? Just looking from a different point of view?

[semeyerz]

In the discussions, we all agreed that a modular, hierarchically-organized overlay mechanism is required to easily benefit from external repos.

Do we really need external repos and composition of repos?
I think fragmentation would hurt the project right now.
If changes need to be done to the generator/templates, how can we possibly maintain compatibility?
The mightiest solution is not always the best solution -> KISS

We can easily define some semantically-grouped repos: core, robotics, vision, etc., which can be maintained by different groups of people.

We were not yet able to define semantically-grouped sub-folders.
How should that work out in reality across forks, underlays, overlays, etc.?

The semantics was well-defined in RDTK/generator#30, wasn't it?

It was a proposal that lead to a big discussion.
It was not even clear whether to treat templates in the same way as projects.

Isn't that[overlay/underlay] the same? Just looking from a different point of view?

From a theoretical point: yes.
From a practical point: It might cause a lot of confusion to do something from a different point of view.
Especially when we do not use the right terms from the beginning.

In my opinion (git) forks have everything we need right now and allow us to easily track changes.