Is there any method to accelerate the speed of reading a large DAT file?

[lkxajd]

It took about half a minutes when I tried to read a 68MB DAT file, which contains 2212 analog channels, and 10000 samples for every channel.
Is there any method to accelerate the reading process?

[engineerjoe440]

Hi @lkxajd!

That's sort of a challenging question. It's difficult to say without being as familiar with @dparrini, or having the actual files to test against, but my suspicion is that the time you're running up against is sort of the "limit" of how fast it can be processed.

There are certain things that could potentially optimize the performance if the implementation were done in a lower level (say C, Rust, or even with GO) but that adds complexity, and may ultimately become more fragile.

Are you able to share the COMTRADE record in question so that we could evaluate it ourselves and observe the performance to see if there are any clear areas for improvement?

[lkxajd]

@engineerjoe440

I tested again, and I found the slowest process is in "parse" , which contain nested loop.

I am sorry that I can not share any file right now, because they belong to my company.

Maybe, I can create a similar file later and share it.

[dparrini]

Hi @lkxajd , I'll check how to optimize this but I was thinking on adding options/modes before changing the current/default way. I'll let you know. Do you need to use all channels (or several) at once or on your use case you only use a few of the 2000+ channels?

[bomei]

@lkxajd You may use go or c++ to rewrite the parse func, and build it to a library. This just reduces time cost from 10s to 0.5s in my test reading a 10MB file in the same PC. My coding on go hasn't finished yet, or I can share it with you.

[dparrini]

Hi @lkxajd please have a look on the 0.0.9 update. It improves loading times with the default settings. It might help to use use_numpy_arrays=True keyword argument on Comtrade's constructor for computations later.

[lkxajd]

Hi @dparrini
I am sorry that I did not replay till now, because I was dealing with something else.

I tried 0.0.9, but it seems not help much.

The parse time is not big trouble. So, if there is any thing more important, just ignore this one.

I uploaded my record files .

Thanks so much!
sample_files.zip

[bomei]

@lkxajd
Hi, you may try this https://github.com/bomei/comtrade-gopy, using go for parse part.

Here is my test result:

# 0ms

rec = comtrade_grpc.Comtrade(grpc_endpoint='localhost:50051')
rec.load(cfg_file,dat_file)

# 497.397ms

rec = comtrade_grpc.Comtrade()
rec.load(cfg_file,dat_file)

# 5809.528ms