Non-file-based (in-memory/streaming) usage?

[frankier]

This issue is part of my JOSS review. However it goes a bit beyond the remit, so feel free to treat this issue especially as "take it or leave it".

The high level/documented API of this module has an interface with a fixed pipeline where most of the "payload data" is deserialised/serialised to files. This interface may be convenient for some workflows, but other users may have their own workflows or may already have their data in a particular format, and might want to put it into the correct format "in memory" either as a single chunk or as multiple chunks using a generator.

While it is the case that ndl can already take a generator rather than an events files, it simply serialises the generator and reloads it. It would be nice to restructure things so there are two strata:

• File based api, which is implemented in terms of
• Generator/in-memory API

[derNarr]

I agree that it is somehow restricting and unelegant.

The whole parallelization and speed gain is (mainly) based on the binary, efficient, intermediate file format. All the (learning) events that are encoded in that file format are gone through as often as there are outcome chunks usually between 5 to 400 times. For most scientist working with pyndl for now the main concern was memory and therefore having the learning events NOT in memory was a big plus as this leaves more memory for the weights.

Still, I agree adding an "in memory option" would be a nice addition and if anyone has a need for it, please comment here so that I can prioritize this. It should make processing even fast as no disk-io is needed during learning, with the caveat of higher memory demand.

[frankier]

The normal way of handling a generic resettable stream like this in Python would be to provide multiple implementations which follow some kind of reader interface. This could be more functional style as a function which returns an iterable which is recalled every time we need to reset it, or it could be a implemented using a class (similar to file-like objects) that has reset() and next() methods. I think in terms of working with multiprocessing, the latter is likely to be a smoother experience. An in-memory interface and pyndl event format interface could be provided, and then the documentation could give details of how users can make their own reader if they want (noting that it will be used in a multiprocessing context).

This issue may be related: #130

[frankier]

At a high level, I see there being 4 layers/strata in the project from the lowest:

1. The NDL rule as independent of the file format used for the events and the weights matrix (does not seem to be directly exposed/documented -- see the comments of @jinhangjiang and Non-file-based (in-memory/streaming) usage? #213

• Exposing/documenting this would be a "nice to have" and could create appeal to those outside the cognitive linguistic audience

2. The NDL rule as implemented in terms of the library's custom file formats

• This is introduced in the documentation through a word form-semantics association (meaning recognition) example

3. Event file generation routines which can convert "END OF DOCUMENT" delimited corpora files into

• This is introduced in the documentation between a bigram-word form association (form recognition) example

4. Convenience functions to convert from other corpora formats (currently only OpenSubtitles) to stratum 3

Exposing and documenting Stratum 1 may be a good longer term goal.