News
NVIDIA lets AI learn to "copy homework": KV cache can be migrated across models, and inference speed increases by 25 times. NVIDIA's research team made a breakthrough: they found a way to migrate KV cache between different models
2 min read
Source: Telegram AI频道
NVIDIA lets AI learn to "copy homework": KV cache can be migrated across models, and inference speed increases by 25 times. NVIDIA's research team made a breakthrough: they found a way to migrate KV cache between different models. The target model can directly skip the pre-population stage, and the conversion speed is 2. 7 to 25 times faster than reprocessing the context. To understand the significance of this breakthrough, you need to first understand the role of KV cache. When using ChatGPT or Claude, you will find that the first word is always generated significantly slower, and then the subsequent content pours out almost instantly. There is a deliberately designed mechanism behind this - before the model generates the first word, it needs to process the entire input context and store the intermediate results as a KV cache. Each subsequent word can reuse these caches, so it's much faster. The problem is that KV cache has always been a "one-at-a-time model". If you switch to another model, or upgrade the model version, all previously calculated caches are lost, and the new model must process the entire context from scratch. For long text scenarios, this means a lot of double calculations and wasted time. NVIDIA's solution makes the KV cache portable. The cache calculated by one model can be used directly by another model after conversion, and the target model completely skips the pre-population stage. The transformation process itself is also much faster than reprocessing the context - the speedup ranges from 2.7x to 25x, depending on the complexity of the model and context. The impact of this on the AI industry is profound. A considerable part of the current cost of using the LLM API comes from context processing, especially long conversation and long document scenarios. If the KV cache can be migrated between models, it means that users do not have to bear the cost of repeated calculations when switching models, and they do not have to discard existing conversation contexts when upgrading models. This research may change the way AI inference infrastructure is designed, making switching between models more fluid and economical. via AI News (author: AI Base)