Support bulk and partial redefinitions in the `\markdownSetup` plain TeX command

[Witiko]

Discussed in #121

^{Originally posted by Witiko March 5, 2022}
At the moment, the \markdownSetup LaTeX command allows the redefinition of individual renderers and renderer prototypes. This has several shortcomings:

• It is tedious to redefine several renderer or renderer prototypes at once, such as those for the six levels of headings.
• Keeping some parts of the previous definitions is difficult. Common use cases include prepending and appending code to previous definitions, undoing previous redefinitions, and calling the previous definition from the redefinition.

It makes sense to consider these shortcomings together rather than in isolation. For example, redefining all six levels of headings to behave the same is seldom useful, but prepending or appending the same code to all six levels of headings or all ordered, unordered, and definition lists is a common pattern.

Here are some possible desiderata for a syntax extension to \markdownSetup:

1. Backwards compatibility: Whatever is now valid in \markdownSetup should stay valid and keep its current meaning.
2. Bulk redefinitions: Allow several renderers or renderer prototypes to be redefined in bulk.
   • Enumeration: Allow the explicit specification of several renderers or renderer prototypes to be redefined in bulk.
   • Wildcards: Allow the implicit specification of several renderers or renderer prototypes to be redefined in bulk.
     Requires Add reflection to the TeX implementation #119.
3. Partial redefinitions: Instead of full redefinitions, allow the partial redefinitions of renderers and renderer prototypes.
   • Prepending and appending: Instead of replacement, the code would be prepended or appended to the previous definition.
   • Patching: Rather than replacing all of the previous definition, only a part of it could be replaced.
   • Matching: Rather than replacing the previous definition, the previous definition could be arbitrarily reordered and interleaved with code.

This thread will list some of the ideas for the syntax extensions and outline which desiderata they satisfy.

[Witiko]

Discussed in #121

^{Originally posted by Witiko March 5, 2022}

In the syntax extension we discussed today with @xvrabcov, the renderer and renderer prototype definitions would be specified as comma-separated lists of globs, where a glob is a name of a renderer or renderer prototype with optional asterisks that match zero or more characters. Here are some examples:

\markdownSetup{
  renderers = {
    headingOne = {...},                              # Redefines the top-level heading. (Backwards compatibility)
    {headingOne, headingTwo, headingThree} = {...},  # Redefines the first three levels of headings. (Enumeration)
    heading* = {...},                                # Redefines all levels of headings. (Wildcards)
  }
}

Between the list of globs and the equal sign (=), there would be an optional regex that would specify the part of the renderer or renderer prototype that should be redefined. Here are some examples:

\markdownSetup{
  renderers = {
    headingOne ^= {...},                            # Prepends code to the top-level heading. (Prepending and appending)
    headingOne $= {...},                            # Appends code to the top-level heading. (Prepending and appending)
    headingOne {^some code} = {},                   # Removes code from the beginning of a definition. (Patching)
    headingOne {^(.*)$} = {... \1 ... \1 ...},      # Repeats the definition twice and interleaves it with code. (Matching)
  }
}

The syntax extension satisfies all the desiderata. It is unclear whether a failure to patch or match should produce a warning or an error.

[Witiko]

Patching and matching seem both very situational and signlificantly more difficult to implement than the rest, especially since they involve specifying regular expressions ({^some code}) and replacement text ({... \1 ... \1 ...}). For regular expressions, the difficulty stems from parsing, because the l3regex module of expl3 has no notion of balanced text, so it's difficult to reliably separate the renderer globs ({headingOne, headingTwo, headingThree}) from the regular expression. For the replacement text, backwards compatibility dictates that we perform no substitutions except perhaps the \N capture groups. However, the l3regex has a special format of the replacement text, which requires that we translate between what we expect and what expl3 expects.

Dropping patching and matching for now, here is a simple parser of the extended syntax for definitions: https://www.overleaf.com/read/ztzjgtnkkwdx

[Witiko]

Support for enumeration makes things a bit more difficult in combination with prepending, appending, and matching, because the implementation must take care that we always change a renderer only once even if it is duplicated by the user. However given the lack of support for O(1) sets in expl3, this seems as a bit of a headache. Perhaps we can also drop enumeration for now, support just wildcards in 2.23.0, and revisit this in the future.