Capex is a promise; revenue is the receipt — and inference cost is the line in between that decides whether the promise pays. Every time a deployed model answers a query, it does inference, and inference cost scales with usage in a way that training does not. So the most economically interesting AI patents are often not the flashy model architectures but the unglamorous methods that shave the per-query bill. The key-value cache is exactly that kind of plumbing.

Here is the mechanism, in business terms. As a language model generates a response token by token, it stores intermediate state — the key-value cache — so it does not recompute everything for each new token. That cache grows with the length of the conversation and the size of the model, and it lives in expensive accelerator memory. Optimize it and you serve more queries per chip; fail to, and you buy more chips. The published application US20250390703A1, "Optimizing key value cache for large language model inference," lists inventors including Noam Shazeer and Myle Ott and claims methods aimed squarely at that cost.

“An input sequence is received from a client device. Large language model inference is performed by processing the input sequence through a series of transformer layers to generate one or more tokens including by performing hybrid attention, multi-query attention, and cross-layer key value sharing.”— U.S. Patent Application 2025/0390703 A1 source

What the independent claim actually recites is three named cost levers stacked on top of each other, and each one attacks the same scarce resource: accelerator memory. The first is hybrid attention. Claim 1 describes enabling local attention for a plurality of consecutive transformer layers and then "injecting global attention at regular intervals" between two blocks of local-attention layers — a dependent claim pins those blocks at five layers each. Local attention is cheap because each token only looks at nearby tokens; global attention is expensive but occasionally necessary. Interleaving them is a deliberate trade of a small amount of modeling reach for a large reduction in the memory the cache has to hold.

The second lever is multi-query attention. Standard attention keeps a separate set of key and value vectors for every attention head; the claim instead "shares the corresponding key vectors and value vectors across a plurality of heads." Fewer distinct key/value sets means a smaller cache for the same model width — a direct cut to the dominant term in the inference memory bill. The third lever is cross-layer key-value sharing, recited in a dependent claim as sharing those key and value vectors "across two or more transformer layers." Where multi-query attention shrinks the cache horizontally across heads, cross-layer sharing shrinks it vertically across depth. A further dependent claim adds that the attention weights can be represented in Int8 precision — eight-bit integers instead of wider floats — which shrinks each stored number as well. Stack hybrid attention, multi-query sharing, cross-layer sharing, and low-bit weights together and you are attacking cache size on four axes at once.

Show me the line item: there is not a standalone "KV-cache savings" entry in anyone's 10-K, and there never will be. But the aggregate shows up where it always does — in the technical-infrastructure investment hyperscalers disclose. Alphabet's most recent annual report describes continued investment in servers and data centers to support growth (Alphabet Form 10-K, FY2025, filed 2026-02-05). Every dollar of that capex is, in part, a bet that demand outruns the cost of serving it. Cache-efficiency IP is the engineering answer to the question the capex line poses.

The claims also describe the serving loop the cache lives inside, which is where the cost actually accrues. The method tokenizes the input, converts tokens to embeddings, adds positional embeddings, and runs them through the transformer stack to predict the next token from "an output vector associated with a last token in the input sequence." That token is selected by highest probability, a probability distribution, or beam search, appended to the sequence, and the loop repeats until an end condition — an end-of-sequence token, a maximum token limit, or a user interrupt. Two claims add that generated tokens can be streamed "as the one or more generated tokens are being selected," which is the technical basis for the typewriter-style streaming users see. Every iteration of that loop touches the cache; that is precisely why shrinking the cache compounds across an entire generation.

The distinction this desk cares about: a published application is not a granted patent, and a method is not a margin. US20250390703A1 is an application — published, not yet issued — and even a granted claim describes a technique, not a guaranteed cost curve. The patent does not disclose how many queries per chip any of these techniques actually buys, and no filing attributes a dollar figure to them. Treat the document as a signal of where the efficiency effort is pointed, not as a disclosed financial outcome.

Why does this belong on a business site rather than an engineering one? Because the AI investment debate ultimately reduces to unit economics: does the marginal query earn more than it costs to serve? Inference cost is the denominator of that question, and the KV cache is one of the largest terms in it. When you read a hyperscaler's rising infrastructure spend, the offsetting story — the one management hopes plays out — is told in applications like this one, where the engineering is aimed squarely at the memory term.

The reader takeaway is disciplined, not breathless. The capex is disclosed and large. The efficiency IP is real, dated, and accumulating, with named techniques that map cleanly onto the cost of serving. Whether the two net out to durable AI margins is not something any single filing or patent settles — but the KV-cache work is precisely where to watch the cost side of that equation move.