[Bug]: Misleading Data in related paper.

[jianfei-wangg]

For instance, in 70B models, a single request with a 16K-token prompt generates 640 MB of KV cache, split into 2048 disjoint 4KB blocks for
a single GPU out of eight. (Page 4)

2048 * 4KB != 640MB

16×128×2B = 8192B. (Page 6)

maybe 16 x 256 x 2B?

cache[B][KV][L][H][D]

how deal with paged attention such as k/v: [max_num_pages, num_layers, num_kv_heads, page_size, head_dim]? Whether Block coalescing can work with paged kv_cache?

[cctry]

Thanks for pointing this out. I will fix it soon

The block in the paper refers to the page. It can support arbitrary KV cache layouts, but we only tested those in vllm/attention/backends. In your layout, you can slice the tensor like cache[:,L:,:,:,:] to create a view for each transformer layer. However, the current Python-based block coalescing will not work because it only merges consecutive blocks.