MatX for element-wise tensor calculation

[CrreeL]

Hi! I have a cluster of expressions with the form like p=Aexp(-(E-Fx)/(kBT)) to deal with in my project. I tried both MatX (numpy and matlab like) and thrust (with functor and transform function). It turns out that MatX is about half as fast as thrust. I think it reasonable, since MatX provides more user-friendly syntax. However, I still want to verify whether this is consistent with developers' expectations.

[cliffburdick]

Hi @CrreeL , thanks for reporting this. We definitely do not expect thrust to be faster. We will test it out on our end. Can you please provide a sample of what your thrust syntax looks like so we can compare?

[CrreeL]

I'm pleasantly surprised to hear that MatX is designed to be faster than thrust. Sure, I would like to provide a code sample. It should be available tomorrow.

[CrreeL]

I think I've located what's wrong. I made a test code calculating expressions like p=Aexp(-(E-Fx)/(kBT)). Firstly, make_tensor and (tensors = initialvalue).run(), cudaDeviceSynchronize, and then performs expressions calculation, cudaDeviceSynchronize again. The calculation was repeated for 10 times using a for loop, and the time cost for the first time was substantially longer than the remaining 9 runs.

Maybe I missed some details in the documentation. So, the first execution of element-wise expressions will incur a one-time cost? And if only the element values of the tensors are updated, the one-time cost is not incurred again, right? For example, I define the tensors as class members, and update the tensors and evaluate the expressions through member function calls each time. The one-time cost of each expression will only be incurred when I call the member function for the first time?

[cliffburdick]

Hi @CrreeL , we do not necessarily think it should be faster than Thrust, but it shouldn't be slower. Both should use the same constructs under the hood, but there are cases where I would expect it to be the fastest.

As far as benchmarking, make_tensor by default uses managed memory, and your first iteration of the loop with page the memory in and out of the GPU, which has a large negative contribution to the runtime. There are other reasons for the slowness on the first loop too, and in general, you should never take the first iteration as part of your measurements. For simple element-wise computations like you're doing there are likely only two sources of long time penalties:

1. Managed memory migration
2. GPU Clocks

To fix 1 you can use MATX_DEVICE_MEMORY as an argument for the memory type to make_tensor. This will give the fastest speed possible, even faster than managed on further loop iterations, because it doesn't have any paging. The downside is that you cannot access the memory on the CPU on most systems since it's equivalent to a cudaMalloc. For #2 you can fix the clocks on the GPU by using nvidia-smi, or usually it's good enough to run many iterations in a loop, start your timing events, then run the rest.

A good pattern would be:

1. Run 100 iterations
2. Start events
3. Run 100 iterations
4. Stop events and record timing

If you do this and we're slower than thrust, please let me know.

[CrreeL]

I tried MATX_DEVICE_MEMORY, and achieved significant speed up. Indeed, MatX is faster than thrust (transform +hand-writen functor), at least for my case, which is calculating element-wise expressions like p=Aexp(-(E-Fx)/(kBT)).
Thank you so much for your patience with a GPU programming rookie!