Identifying Pain Points of Future Refactor

[dwrodri]

Greetings Authors

I have stumbled upon this project as I am working on collecting fine-grained traces of CUDA kernels, and I am currently attempting to perform a minimal refactor to get it working on LLVM 11.0 and CUDA 10.1.

Here is what I have identified so far as the pain points:

• Custom commands in CMakeLists.txt need to be updated, since clang is now in the LLVM GitHub repo
• llvm::GlobalVariable::setAlignment(1) call on lib/InstrumentDevice:79 no longer accepts an int, so probably needs to be replaced with a MaybeAlign?
• The process for inserting this function call has changed significantly and I am not sure the best way to go about fixing it yet as I am new to writing LLVM plugins.

[alexander-matz]

Good finds, especially the last one might be interesting! Unfortunately I haven't done any LLVM development recently, so I can't comment on how to best solve it.

But I'd like to add some additional potential pain point with regarding the CUDA portion of this project. Incompatibilities here can stem from both changed LLVM behavior (since LLVM is used as the CUDA front/middle-end) and changes in GPU architecture/CUDA programming model (the project seems to currently target CUDA 8).

I'd go about verifying compatibility something like this:

• Compile the .cu files in support/ to verify the intrinsics/APIs used still work
• If this works, try to set up the host-GPU queue it implements manually to verify that the queue itself works (check setupTraceInfo() lib/InstrumentDevice.cpp to see the device side setup, and findOrInsertRuntimeFunctions in lib/instrumentHost.cpp for host side setup). It relies on some behaviors regarding synchronization and warp execution that might have changed
• Compile the device code of a CUDA application to LLVM IR and verify that the location of kernel call sites still works (implemented in lib/LocateKCalls.cpp). It relies on the LLVM IR for device code being generated in a particular way, which might have changed.

Let me know if you have questions about any of this.

[dwrodri]

Just leaving a note here to say thanks for the reply! I have some research on GPU perf analysis that will benefit from this tooling, but given a lack of man-hours I can dedicate to submitting a PR for LLVM 11/CUDA 11 support, it's temporarily on hold. I will definitely report the results of these tests when I get the chance.

[alexander-matz]

Yeah, that's reasonable. Depending on how much control you have over the applications you're analyzing, you can try a semi-automated approach, where you do all the host transformations (including the very fragile location of kernel calls) manually and also manually the GPU support code to your CUDA files, leaving you only with the actual instrumentation of memory accesses implemented as an LLVM pass. That cuts down on the porting efforts of the LLVM passes at least, but of course doesn't might still leave you with compatibility issues with newer CUDA versions. If you can find some tested and reusable host-gpu-queue, however, you might be able to just plug that in and side step at least some more issues or subtle bugs. If you end up doing anything like that, we should try to integrate some of it back into this project to maybe save the next guy some efforts :)