optimize METs handling

[krvoigt]

Current situation

The more files are in a workspace, the larger a METS file becomes, the longer processing takes, especially when information needs to be searched in different places of the XML, such as getting page ID (which is in the mets:structMap[@TYPE=LOGICAL]) for mets:file (which are in the mets:fileSec/mets:fileGrp.

Access to the METS file is exclusive, there can be no workspace-wide parallelization because different processes will overwrite, not append to the METS file, causing data loss.

How it should be

The performance penalty of dealing with larger workspaces should grow linearly or logarithmically, not polynomally. Caching should be used to keep relevant information (such as pageID to file mappings) in memory.

A synchronization mechanism should be implemented, that allows multiple processes to add to the METS file without data loss.

[MehmedGIT]

This is a resume of the discussions in #39 and the potential next steps:

1. Regexes of input parameters are compiled inside a loop and not precompiled before the loop. Doing so wastes compute resources. Robert has already pushed a change that precompiles all regexes (commit) and fixed that.

2. Since mets:file and mets:fileSec/mets:fileGrp are used more frequently, a caching mechanism for these two would be most beneficial to improve the performance of add_file and find_files. The add_file function calls inside the find_files function with a single ID parameter.

https://github.com/OCR-D/core/blob/7604abbe367bc4b60a81aacc575ad8d1ff801f42/ocrd_models/ocrd_models/ocrd_mets.py#L313-L316

Next step: Instead of doing that the ID should be searched inside the ID cache that is produced after an OCR-D processor is started. Doing so would reduce the searching complexity to O(1).

A proof of concept implementation of the cache:
Initialization of the cache inside the OcrdMets constructor:
https://github.com/OCR-D/core/blob/8b18c149605de3f347ae319755d16be8fbeb00b6/benchmarks/test_mets2.py#L66-L76

Usage of the cache inside the find_files function:
https://github.com/OCR-D/core/blob/8b18c149605de3f347ae319755d16be8fbeb00b6/benchmarks/test_mets2.py#L212-L237

Get a fileGrp function that checks the cache first, if the fileGrp is not there searches the entire XML tree:
https://github.com/OCR-D/core/blob/8b18c149605de3f347ae319755d16be8fbeb00b6/benchmarks/test_mets2.py#L93-L120

3. Testing the building of the METS file and searches should happen separately. Otherwise, the search performance is obfuscated by the building performance.

4. Test scenarios should be diversified (by Robert):

• thousands of pages
• dozens of fileGrps
• fileGrps with hundreds of files per page

[MehmedGIT]

Here is the final version ready for review. Extreme test cases are added too.

OCR-D/core#875

[MehmedGIT]

#875 was merged into the master branch.