KVpop -- Key-Value Cache Compression with Predictive Online Pruning

KVpop is a novel key-value cache compression method that uses predictive online pruning to achieve high performance while reducing memory usage. It learns a fixed-budget KV eviction policy by directly supervising the keep-or-drop decision. KVpop outperforms established eviction baselines on AIME and HMMT mathematical reasoning benchmarks.

1sec.ai