[FEAT]: cuda kernel of deft

[Monstertail]

Description

refs:
1. Flashinfer: good prefetch
2. Thunder Kittens: good for H100 GPU

Additional Information

No response

[Monstertail]

@chiakicage You can update more info of cuda kernel design here.

[Monstertail]

ThunderKittens makes a few tricky things easy that enable high utilization on modern hardware.

Tensor cores. ThunderKittens can call fast tensor core functions, including asynchronous WGMMA calls on H100 GPUs.
Shared Memory. I got ninety-nine problems but a bank conflict ain’t one.
Loads and stores. Hide latencies with asynchronous copies and address generation with TMA.
Distributed Shared Memory. L2 is so last year.

@chiakicage I think distributed shared memory could be a potential optimization for prefix sharing.

[Monstertail]

[milestone]

1. minimal demo: (a) attention is the overhead; (b) kv manager should be okay: ours, radix tree, hashed seq group(vLLM);
2. kernel: (a) ref: flashinfer ; (b) GEMV/GEMM; (c) different implementations: TK, Cute, Triton, etc.

[Monstertail]

refs:

1. api wrapper in FA；
2. kernel code in flashinfer, especially cascade.