Consider splitting into multiple (packages|repo)

[adriaandegroot]

The Python tooling built on top of quick-der is not necessary for using the library itself. Since building the scripts and getting the dependencies right for generating the (C and Python) bindings for ASN.1 syntax files is a pain in the butt, perhaps it should be split out of the QD library (repo|package). Generated (C) headers for the RFC headers could be checked in, since they change very little.

As an alternative, the configure script should at least try to find a suitable Python interpreter and check for dependency modules (e.g. six) before the build starts.

[vanrein]

I suppose we have three layers:

• Python support in asn2quickder and asn1literate
• C language support related to generated header files
• Python support related to generated Python modules

I moved <quick-der/api.h> to <arpa2/quick-der.h> for C, but this is not as easy for Python, where packages are delivered complete, and cannot receive later injections, AFAIK. I don't know if a package can receive later additions of sub-packages.

Python turns out to be quite a beast when it comes to packaging... I'm already glad that we now have a working setup.py...

[vanrein]

As a first stab at this, I've made a branch asn2quickder-split. It is a pure Python package, so I removed CMakeLists.txt and kept (moved) python/setup.py with modified instructions.

I'm not completely sure how to proceed. This is just asn2quickder and asn1literate but it already seems to call out for the rest:

• Header files are useful as soon as you compile generated ASN.1 content that includes them;
• Processing the data formats involves using the library, including der_pack() and der_unpack() but also things like der_get_int32() and utilities such as der_cmp_integer() are valuable to have.

I am finding it difficult to split the code. The split I made is far from perfect, because I don't know where to draw the boundary. I suppose I am doubting your statement that the Python tooling built on top of quick-der is not necessary for using the library itself but I do agree that the build is painful, and that alone might be a reason for a split. Would we then end up with two or three mutually dependent packages, built separately?

Please inspect and comment :-)

Also, this can be an opportunity to move out of the vanrein namespace. It would even be possible to setup a compiling service based on CI/CD using precisely this package to construct a Git repo with the header files for C and, separately built, a module for Python.

[vanrein]

This might be what we could aim for:

• PIP module based on the above, less the API includes that are now included
• ASN.1 sources in quick-asn1.git -- with directories and/or branches if we need name spaces
• GitLab CI/CD on quick-asn1.git can be setup to produce a repository quick-der.git with asn2quickder immediately upon checkin
• We may include standard recipes for producing new versions, both in Python as PIP and in other distribution formats as we please. These would add the C library and Python module.

This probably removes the burden of keeping this package building... agreed?

asn2qd-infra.pdf

[vanrein]

I suppose package asn2quickder might depend on quick-der but not the reverse.

In fact, it won't be building with asn2quickder that depends on it, but using the output. And that would be language-specific.

I suppose this means that we can have quick-der for C and Python separately, even if asn2quickder can serve both.

And I suppose that is the split you are after... (?)

[vanrein]

Hmm, this is advantageous from a packaging viewpoint... but to make changes we'd have much more trouble. For instance, to modify a data structure it would require a concerted upgrade (and risk inconsistencies if it went awry).

[vanrein]

The wish to support imports of specs with a mere import quick_der.rfc4120 can be scaled up to dynamic plugins with the entry_points of setuptools as described here. This might help to avoid declaring every spec inline. Less useful to have a module quick-der-rfc but it's already more useful to install quick-der-arpa2 as a separate package. A lot of work I guess, and a hectic amount of Python juggling.

Much more appropriate, in terms of involving less code to simulate sanity, may be namespace packaging for quick-der, with contributions from various places or projects. We would say from quick_der import rfc4120 and it would be taken from the RFC package for the quick_der namespace. Other packages could add other modules.