Memory exception on armhf platform raspberry pi

[menschel]

Hi, I'd like to report a bug.

After compiling fmtlib and installing dltpy on a raspberry pi 3B , the module seems to be broken.

The same steps on a x86_64 platform worked without any problem.

$ dltpy-print example.dlt 
[2020-09-29 19:34:12,059] - Will print file example.dlt with filters: []
filter len: 0
fwd filter len: 0
Traceback (most recent call last):
  File "/usr/local/bin/dltpy-print", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/dist-packages/dltpy/dltpy_print.py", line 47, in main
    for dm in dltf:
  File "/usr/local/lib/python3.7/dist-packages/dltpy/transforms/__init__.py", line 10, in apply_transforms
    for msg in it:
  File "/usr/local/lib/python3.7/dist-packages/dltpy/dltfile.py", line 171, in __iter__
    ret = self.get_next_message()
  File "/usr/local/lib/python3.7/dist-packages/dltpy/dltfile.py", line 165, in get_next_message
    m = DltMessage(self.rdr)
  File "/usr/local/lib/python3.7/dist-packages/dltpy/dltfile.py", line 70, in __init__
    self._load(reader)
  File "/usr/local/lib/python3.7/dist-packages/dltpy/dltfile.py", line 90, in _load
    self._raw_payload = bytes(reader.get_payload())
SystemError: Negative size passed to PyBytes_FromStringAndSize

dlt-convert does work on the raspberry pi 3B.

$ dlt-convert -a example.dlt 
0 2020/09/12 19:05:40.644940    6244253 000 ECU1 DA1- DC1- control response N 1 [service(3842), ok, 02 00 00 00 00]
1 2020/09/12 19:05:40.645094    5701774 000 ECU1 DA1- DC1- control response N 1 [service(3842), ok, 01 00 00 00 00]
2 2020/09/12 19:05:40.645119    5701775 118 ECU1 DLTD INTM log info V 1 [Client connection #7 closed. Total Clients : 0]
3 2020/09/12 19:05:40.645151    5701906 119 ECU1 DLTD INTM log info V 1 [ApplicationID 'SINA' registered for PID 6063, Description=stdin adaptor application]
4 2020/09/12 19:05:40.645237    5701906 000 ECU1 DA1- DC1- control response N 1 [get_log_info, 07, 01 00 53 49 4e 41 01 00 53 49 4e 43 ff ff 15 00 73 74 64 69 6e 20 61 64 61 70 74 6f 72 20 63 6f 6e 74 65 78 74 19 00 73 74 64 69 6e 20 61 64 61 70 74 6f 72 20 61 70 70 6c 69 63 61 74 69 6f 6e 72 65 6d 6f]
5 2020/09/12 19:05:40.645297    5701907 000 ECU1 SINA SINC log info V 1 [Hello, world! ]

[Equidamoid]

Hi @menschel,

Thanks for the report!
I'll try to reproduce it in the coming days. Just in case, could you mail me ($login at gmail.com) your example.dlt?

[Equidamoid]

Memory corruption. Something writes some zeroes to a place it shouldn't touch. It happens somewhere between calls to dltpy's native code and not when new data is read from the dlt file.
Pretty interesting one. Will keep digging.

[menschel]

My first guess was a problem with endianess. There are some caveats in the protocol. The standard header extra is variable length. The message length in the standard header is big endian although its own endianess flag tells little endian. Those are the pitfalls I fell into while reverse engineering the protocol from the dlt-daemon repo.

[Equidamoid]

Yup, my first thoughts exactly. Either endianness or some other parser error. Alternatively, I worked with a nice system which corrupts dlt files like crazy sometimes (buffer overflow in dlt-daemon), so maybe a corrupted file... but nope!

As far as I see now, a certain variable changes from correct value to zero somewhere after I return from native code and before the native code is called again.
I don't believe in such a bug in python itself, so probably the native part of dltpy (or pybind11, I've had some troubles from it before) does something wrong.

[menschel]

Just to be sure, are you using the DLT_MSG_READ_VALUE macro somewhere. There is an int32_t cast which I believe should be uint32_t. That macro can return a negative length.

BTW I have a working reference implementation in pure python now, so no need to hurry. It is a quick hack though.
Found a caveat in windows IDEs by accident. They use the local (system) charset for printf debugging. The contents I'm scanning are CP1252.
dlt-convert treats them as ASCII and prints funny stuff, however python3's bytes.decode() gets it right somehow.
The example.dlt does use ascii only.
EDIT: c&p error should have meant DLT_MSG_READ_VALUE, not DLT_MSG_READ_ID