SEMLA does not allow parallel loading of a library

[niklwors]

As a result, it can take a very long time until a library is loaded

[hubertus65]

What do you mean by parallel loading, and are you sure that is the right cure? If you have an issue, it woudl be nice to have a description of the environment: what tool did the loading, etc. We have not seen issues with slow library loading.

[niklwors]

We have the problem that it takes a very long time to load our encrypted library in OpenModelica Connection Editor, up to one minute. As far as I understand it, it is not possible to split the loading of a library so that parts of the library can be loaded in parallel. Maybe @adeas31 can describe the problem better.

[hubertus65]

As far as I can tell, this supposes that parallel loading would be a solution to the experienced slowness. We don't use any parallelism, and loading of encrypted libraries is very fast. I suggest to investigate the true reason for the slow loading, instead of starting with a supposed solution of parallelisation.

[hubertus65]

@adeas31, please get back with details of I will close this with somehting like "unable to replicate"

[adeas31]

I will take a look at it and will get back with details.

[adrpo]

Decrypting several .moc files at the same time in parallel would speed up the library loading quite a lot for large libraries (with a lot of files).

Currently it seems that calling the lve tool can only be done on one thread in serial.
Attempting to start several lve tools via mlle_start_executable on several threads and do decryption via the resulted struct mlle_connections *lve results in a lot of errors:

adrpo33@ida-0030 UCRT64 /c/home/adrpo33/dev/modelica/SomeLib
# time omc -d=execstat e.mos
Error reading version. . reason: record layer failure
Thread 5 startLibraryVendorExecutableImpl failed on C:/home/adrpo33/dev/modelica/SomeLib
Error starting library vendor executable C:/home/adrpo33/dev/modelica/SomeLIb/.library/lve_win64. SSL: Handshake failed. Reason: SSL_ERROR_SSL. reason: unexpected eof while reading
Thread 7 startLibraryVendorExecutableImpl failed on C:/home/adrpo33/dev/modelica/SomeLib
Error reading version. . reason: record layer failure
Thread 7 startLibraryVendorExecutableImpl failed on C:/home/adrpo33/dev/modelica/SomeLib
Error starting library vendor executable Error starting library vendor executable Error starting library vendor executable. SSL: Handshake failed. Reason:
Thread 5 startLibraryVendorExecutableImpl failed on C:/home/adrpo33/dev/modelica/SomeLib
Error reading version. . reason: ssl/tls alert bad record mac
Thread 5 startLibraryVendorExecutableImpl failed on C:/home/adrpo33/dev/modelica/SomeLib
Error starting library vendor executable C:/home/adrpo33/dev/modelica/SomeLib/.library/lve_win64. SSL: Handshake failed. Reason: SSL_ERROR_SSL. reason: unexpected eof while reading
Thread 5 startLibraryVendorExecutableImpl failed on C:/home/adrpo33/dev/modelica/SomeLib
Error starting library vendor executable C:/home/adrpo33/dev/modelica/SomeLib/.library/lve_win64. SSL: Handshake failed. Reason: SSL_ERROR_SSL. reason: unexpected eof while reading
Thread 6 startLibraryVendorExecutableImpl failed on C:/home/adrpo33/dev/modelica/SomeLib

I guess this happens because of how you are using pipes with stdin/stdout. I will check later.
I'm not even sure if is OK to start the lve tool several times and how that affects licensing (especially with floating licenses).

Well, ideally would be to start the lve tool ONCE like a server and be able to connect to it several times from different threads and decrypt files in parallel.

[hubertus65]

@axelmartenssonmodelon, what is your take on this?

@hubertus65 We spawn several processes using the same LVE executable in parallel.

[adeas31]

@axelmartenssonmodelon that doesn't sound optimal. This means you are doing the handshake several times. Ideally there should be only one LVE executable running and the functions like mlle_tool_file should be threadsafe.

@adeas31 I can see what you mean, this solution would indeed give a limited improvement in performance. However, it would also require the addition of multiplexing in the LVE protocol and a more complex implementation. We prefer to wait with this kind of feature.

[adeas31]

BTW, I tried spawning several processes of LVE executable in parallel but that doesn't work for us. For each LVE process we do,

mlle_start_executable
mlle_tool_version
mlle_tool_libpath
mlle_tool_file
.....
.....
.....
mlle_connections_free

I don't understand how running several processes of LVE are suppose to work. According to SEMLA specification here https://github.com/modelica/Encryption-and-Licensing/blob/master/doc/SEMLA.md#semla---communication-protocol-between-a-tool-and-lve some actions are only performed once.

https://github.com/modelica/Encryption-and-Licensing/blob/33879fd5d5b71dab174e4d0d3fe35b45923b8de5/src/tests/test_tool.c is a test case that implements the call sequence you mentioned.
test_tool compares the decrypted files of the library that it recieves from the LVE against a set of plain text reference files, and fails (with nonzero exit status) if the files are different.

This testcase still passes when run in parallel. It can be run in parallel using the following script:

#!/bin/bash

# Run 5 'test_tool' processes in parallel, wait for them to finish, and check that their exit statuses are zero

run_test_tool_command() {
    ./test_tool --lve lve_linux64 --feature test_licensed_feature --no-feature test_not_licensed_feature
}

wait_and_exit_if_nonzero() {
    local PROCESS=$1
    wait ${PROCESS}
    local EXIT_STATUS=$?
    if [ ${EXIT_STATUS} -ne 0 ] ; then
        echo "$(basename $0): error: process exited with nonzero exit status: process: ${PROCESS}"
        exit ${EXIT_STATUS}
    fi
}


run_test_tool_command & PROCESS_1=$!
run_test_tool_command & PROCESS_2=$!
run_test_tool_command & PROCESS_3=$!
run_test_tool_command & PROCESS_4=$!
run_test_tool_command & PROCESS_5=$!

 wait_and_exit_if_nonzero ${PROCESS_1}
 wait_and_exit_if_nonzero ${PROCESS_2}
 wait_and_exit_if_nonzero ${PROCESS_3}
 wait_and_exit_if_nonzero ${PROCESS_4}
 wait_and_exit_if_nonzero ${PROCESS_5}

When I save this script as run_test_tool_in_parallel.sh in the build dir, and run it, the exit status is nonzero. Here is the captured terminal session (my input prompt is prefixed with $):

$ ./run_test_tool_in_parallel.sh 
#test_tool: Info: using lve: 'lve_linux64'.
#test_tool: Info: using lve: 'lve_linux64'.
#test_tool: Info: using lve: 'lve_linux64'.
#test_tool: Info: using lve: 'lve_linux64'.
#test_tool: Info: using lve: 'lve_linux64'.
ok 1 - connect to  LVE
ok 1 - connect to  LVE
ok 1 - connect to  LVE
ok 1 - connect to  LVE
ok 2 - Test protocol version [1, 1]
ok 2 - Test protocol version [1, 1]
ok 3 - Test set library path ('test_library')
ok 1 - connect to  LVE
ok 4 - Test valid feature ('test_licensed_feature')
ok 5 - Test invalid feature ('test_not_licensed_feature')
ok 6 - Test invalid feature error message ('test_not_licensed_feature')
ok 3 - Test set library path ('test_library')
ok 2 - Test protocol version [1, 1]
ok 4 - Test valid feature ('test_licensed_feature')
ok 3 - Test set library path ('test_library')
ok 4 - Test valid feature ('test_licensed_feature')
ok 5 - Test invalid feature ('test_not_licensed_feature')
ok 6 - Test invalid feature error message ('test_not_licensed_feature')
ok 5 - Test invalid feature ('test_not_licensed_feature')
ok 6 - Test invalid feature error message ('test_not_licensed_feature')
ok 2 - Test protocol version [1, 1]
ok 3 - Test set library path ('test_library')
ok 7 - FILE('package.moc')
ok 7 - FILE('package.moc')
ok 4 - Test valid feature ('test_licensed_feature')
ok 5 - Test invalid feature ('test_not_licensed_feature')
ok 6 - Test invalid feature error message ('test_not_licensed_feature')
ok 8 - open facit ('test_facit/package.mo')
ok 2 - Test protocol version [1, 1]
ok 7 - FILE('package.moc')
ok 8 - open facit ('test_facit/package.mo')
ok 7 - FILE('package.moc')
ok 8 - open facit ('test_facit/package.mo')
ok 8 - open facit ('test_facit/package.mo')
ok 3 - Test set library path ('test_library')
ok 4 - Test valid feature ('test_licensed_feature')
ok 9 - sizes match
ok 9 - sizes match
ok 5 - Test invalid feature ('test_not_licensed_feature')
ok 10 - identical contents
ok 10 - identical contents
ok 6 - Test invalid feature error message ('test_not_licensed_feature')
ok 9 - sizes match
ok 10 - identical contents
ok 7 - FILE('package.moc')
ok 8 - open facit ('test_facit/package.mo')
ok 9 - sizes match
ok 10 - identical contents
ok 9 - sizes match
ok 10 - identical contents
ok 11 - FILE('Module/package.moc')
ok 11 - FILE('Module/package.moc')
ok 11 - FILE('Module/package.moc')
ok 11 - FILE('Module/package.moc')
ok 11 - FILE('Module/package.moc')
ok 12 - open facit ('test_facit/Module/package.mo')
ok 12 - open facit ('test_facit/Module/package.mo')
ok 12 - open facit ('test_facit/Module/package.mo')
ok 12 - open facit ('test_facit/Module/package.mo')
ok 12 - open facit ('test_facit/Module/package.mo')
ok 13 - sizes match
ok 13 - sizes match
ok 13 - sizes match
ok 14 - identical contents
ok 14 - identical contents
ok 14 - identical contents
ok 13 - sizes match
ok 14 - identical contents
ok 13 - sizes match
ok 14 - identical contents
ok 15 - FILE('Module/testInPackage.moc')
ok 15 - FILE('Module/testInPackage.moc')
ok 15 - FILE('Module/testInPackage.moc')
ok 15 - FILE('Module/testInPackage.moc')
ok 15 - FILE('Module/testInPackage.moc')
ok 16 - open facit ('test_facit/Module/testInPackage.mo')
ok 16 - open facit ('test_facit/Module/testInPackage.mo')
ok 16 - open facit ('test_facit/Module/testInPackage.mo')
ok 16 - open facit ('test_facit/Module/testInPackage.mo')
ok 16 - open facit ('test_facit/Module/testInPackage.mo')
ok 17 - sizes match
ok 17 - sizes match
ok 17 - sizes match
ok 18 - identical contents
ok 18 - identical contents
ok 18 - identical contents
ok 17 - sizes match
ok 18 - identical contents
ok 17 - sizes match
ok 18 - identical contents
ok 19 - FILE('binary.gif')
ok 19 - FILE('binary.gif')
ok 19 - FILE('binary.gif')
ok 19 - FILE('binary.gif')
ok 20 - open facit ('test_facit/binary.gif')
ok 20 - open facit ('test_facit/binary.gif')
ok 20 - open facit ('test_facit/binary.gif')
ok 20 - open facit ('test_facit/binary.gif')
ok 19 - FILE('binary.gif')
ok 20 - open facit ('test_facit/binary.gif')
ok 21 - sizes match
ok 22 - identical contents
ok 21 - sizes match

tests: 22, ok: 22, fail: 0
ok 21 - sizes match
ok 22 - identical contents
ok 21 - sizes match
ok 22 - identical contents

tests: 22, ok: 22, fail: 0

tests: 22, ok: 22, fail: 0
ok 21 - sizes match
ok 22 - identical contents

tests: 22, ok: 22, fail: 0
ok 22 - identical contents

tests: 22, ok: 22, fail: 0
$ echo $?
0

[adrpo]

Of course this works as you only have several processes, each with its own SSL instance, but do it in parallel on threads from test_tool.c.

[adeas31]

How this is suppose to speed up the library loading? You are doing the same thing again and again.

What we want to do is that run LVE process once for a library and do these actions as defined in the SEMLA specification,

And then decrypt its files in parallel,

This section above USED AS MANY TIME AS NEEDED should be done in parallel.

@adeas31 If I understand the problem with using only one LVE, it's that the protocol against one LVE is blocking, so you can't ask for the next file until you have received the previous file.

If we spawn several instances of the same LVE and use them as workers in a process pool, then we can ask each worker for a different file in parallel.