d-Matrix Blog - Tag: Data Science

Keyformer: KV Cache reduction through attention sparsification for Efficient Generative Inference

TL;DR: Generative AI inference is often bottlenecked by growing KV cache. There have been several numerous strategies proposed to compress the KVCache to allow longer inference-time context lengths. However, most of…Read More