KV Eviction
Long-context AI, without the GPU tax.
We build inference infrastructure that compresses the KV cache of large language models at serving time — so teams run longer contexts on smaller GPUs, without losing answer quality.
The market problem
Context is the new bottleneck.
The KV cache grows linearly with every token in the context and scales with model depth. At long contexts it — not the model weights — becomes the binding constraint on GPU memory, capping batch size and forcing teams onto larger, costlier instances than the model itself requires.
Existing compression methods trade quality for memory in ways that are poorly characterised: results are reported on different models, budgets and protocols, and rarely reproduce. Teams cannot tell which method is safe to ship.
What we’re building
A serving layer that makes long context affordable.
A Python inference library that scores, protects and evicts KV-cache entries during generation under a fixed memory budget — plus the benchmark suite that proves, per workload, how far you can compress before quality moves.
-
01
Compression under a fixed budget
You set the KV budget in cache entries. The policy keeps attention sinks and a recency window, scores the rest by learned importance, and evicts the remainder — either once at prefill or at every decode step.
-
02
Quality you can audit
Every policy ships with paired evaluations against an uncompressed anchor on reasoning, multi-document QA and long-context retrieval, with bootstrap confidence intervals. Savings without a measured quality bound are not a product.
-
03
Drop-in at the serving layer
Delivered as an API and a library that wraps the generation loop of open-weight models. No retraining, no change to model weights, no change to the prompt.
Technology
What we have running today.
A validated evaluation harness and eviction runtime, exercised end to end on 7B–8B open-weight models across four public benchmarks and KV budgets from 128 to 2048 entries.
Models
Llama-3.1-8B-Instruct as the primary target; Qwen-7B for cross-architecture validation.
Benchmarks
GSM8K and MATH-500 for reasoning, LongBench multi-document QA for retrieval, RULER for long-context recall.
Policies
Per-layer and cross-layer attention scoring, OV-contribution signals, StreamingLLM and H2A-style baselines, with a random control as the floor.
Regimes
Dynamic eviction that re-compresses at every decode step for reasoning, and static prefill compression for retrieval workloads.
Protection
Configurable attention-sink, recency-window and task-span protection, so the spans a workload depends on are never evicted.
Evidence
Paired bootstrap confidence intervals against literature-matched protocols, with every run pinned to a commit, environment and model revision.
Why AWS
We need GPU scale to ship this.
A single benchmark sweep — one model, four datasets, six policies, five budgets, paired against an uncompressed anchor — is roughly 14 GPU-hours. Validating a policy before we let a customer put it in front of production traffic takes many such sweeps.
Category
AI infrastructure · LLM inference efficiency
Customers
Businesses serving open-weight LLMs at long context
Delivery
API and Python library at the serving layer
Company
Building the memory layer for LLM serving.
KV Eviction is an AI infrastructure company founded in 2026 and bootstrapped to date. We are building the compression runtime and the evidence base that lets engineering teams cut long-context serving cost without guessing at the quality risk.