Nvidia finds that simple linear math can replace costly AI model handoffs

When an agentic AI system hands a task from a small model to a larger one — or back down again — it pays a steep tax: the receiving model has to recompute the entire conversation from scratch, driving up compute costs and latency. This is a major bottleneck for enterprises building long-horizon, multi-LLM workflows. To solve this challenge, researchers at Nvidia have introduced a cross-model KV cache transfer technique that directly maps the prefilled KV cache from a source model into the target model. This technique aligns with real-world agentic applications where large contexts accumulate across many turns. For real-world AI applications, cross-model KV cache transfer can reduce compute costs and latency on long-running, multi-LLM workflows — and it does so with simple linear math, not an expensive deep learning model. Experiments show that, on compatible model pairs, this linear mapping process runs 2.7 to 25 times faster than recomputing the conversation while retaining up to 98% of the target model's standalone accuracy. Why swapping models mid-session is so expensive Examining how LLMs handle memory helps understand why multi-model workflows hit a performance wall in production. When an LLM receives a prompt, it must first execute the “prefill” stage, which is the initial forward pass that computes the keys and values for all input tokens and populates the Key-Value (KV) cache. After that, it enters the “decode” phase, where it computes and generates the next tokens in the sequence. During this phase, the model reads from this KV cache to predict new tokens one by one, bypassing the need to re-evaluate the entire history of the conversation for each new token. In multi-turn conversations or long-horizon agentic sessions, the context gradually becomes longer. Beca

When an agentic AI system hands a task from a small model to a larger one — or back down again — it pays a steep tax: the receiving model has to recompute the entire conversation from scratch, driving up compute costs and latency. This is a major bottleneck for enterprises building long-horizon, multi-LLM workflows. To solve this challenge, researchers at Nvidia have introduced a cross-model KV cache transfer technique that directly maps the prefilled KV cache from a source model into the target model. This technique aligns with real-world agentic applications where large contexts accumulate across many turns. For real-world AI applications, cross-model KV cache transfer can reduce compute costs and latency on long-running, multi-LLM workflows — and it does so with simple linear math, not an expensive deep learning model. Experiments show that, on compatible model pairs, this linear mapping process runs 2.7 to 25 times faster than recomputing the conversation while retaining up to 98% of the target model's standalone accuracy. Why swapping models mid-session is so expensive Examining how LLMs handle memory helps understand why multi-model workflows hit a performance wall in production. When an LLM receives a prompt, it must first execute the “prefill” stage, which is the initial forward pass that computes the keys and values for all input tokens and populates the Key-Value (KV) cache. After that, it enters the “decode” phase, where it computes and generates the next tokens in the sequence. During this phase, the model reads from this KV cache to predict new tokens one by one, bypassing the need to re-evaluate the entire history of the conversation for each new token. In multi-turn conversations or long-horizon agentic sessions, the context gradually becomes longer. Beca