Temporal Correlation Meets Sparse Attention: Guess-Verify-Refine Top-K for Blackwell#

By NVIDIA TensorRT LLM team

Table of Contents#

Introduction#

Long-context and code-centric LLM workloads routinely push decode-time context lengths into the 100K+ regime. Sparse attention reduces the quadratic attention bottleneck, but it does not remove the need to rank a long vector of indexer scores at every decode step. In practical sparse-attention deployments such as DeepSeek DSA, exact Top-K selection remains on the critical path, and its cost still grows with sequence length.

TensorRT-LLM already ships a highly tuned production radix-select Top-K kernel for DeepSeek Sparse Attention (DSA), as described in Tech Blog 15. This blog introduces the next step: Guess-Verify-Refine (GVR), a data-aware exact Top-K algorithm that uses the previous decode step’s Top-K result as a prediction signal.

The core observation is simple:

consecutive decode steps tend to query highly similar KV neighborhoods,
so the previous step’s Top-K is a strong warm-start for the current step,
which lets GVR reduce full-row passes from 3-4 to 1-2 in the common case,
while still preserving exact Top-K outputs.

On real DeepSeek-V3.2 decoding workloads running on NVIDIA Blackwell GPUs, GVR delivers:

1.88x average single-operator speedup over the production radix-select baseline,
up to 2.42x per layer per decode step,
and end-to-end TPOT reduction of up to 7.52% in fixed-OSL TEP8 min-latency deployment.

This blog explains the motivation, algorithm, TensorRT-LLM integration, enablement path, and measured operator-level and end-to-end gains. For full derivations, correctness arguments, iteration statistics, and supplementary methodology, see Cheng, Long, et al. “Guess-Verify-Refine: Data-Aware Top-K for Sparse-Attention Decoding on Blackwell via Temporal Correlation”, arXiv:2604.22312 (2026).

Why Decode-Time Top-K Matters in DSA#

Lightweight Indexer and Top-K Selection#

As described in the DeepSeek-V3 technical report, DSA computes index scores with a lightweight MQA-style indexer:

$$I_{t} = \sum_{j=1}^{h}W_j^I \cdot \text{ReLU}(Q_{t, j}^I (K_t^I)^T)$$

The index score tensor $I_t \in \mathbb{R}^N$ (where $N$ is the current sequence length) estimates which past KV tokens matter most for the current query token. A Top-K operator then selects the top 2048 positions, and only those selected entries participate in sparse MLA.

_{Figure 1. Indexer Top-K selection in DeepSeek Sparse Attention. The indexer scores all historical tokens, and Top-K keeps only the most important ones for sparse MLA.}

This design is powerful because it preserves the benefits of fine-grained token sparsity. But it also means that decode-time latency still contains a full-row ranking problem whose cost grows with context length.

Why Long Context Makes Top-K a Bottleneck#

The DSA decode step has three main components:

Component	Scaling	Total Memory Traffic	Trend as $N$ Grows
Indexer MQA	$O(N)$	$N \cdot d_i \cdot 2B$	Linear growth
Top-K (radix-select)	$O(R \cdot N)$	$R \cdot N \cdot 4B$	Linear growth
Sparse MLA	$O(K)$	$K \cdot d \cdot 2B$	Constant ($K$ fixed)

Here $K = 2048$ is fixed, while $N$ grows with the request’s decode-time context length.

This creates a very specific long-context effect:

sparse MLA stays roughly constant because it always attends to only $K$ tokens,
but the decode-stage Top-K still has to inspect all $N$ scores,
so the Top-K fraction of DSA latency grows with context length.

That is why Top-K becomes worth optimizing as a standalone operator rather than being treated as a small side cost. The production radix-select kernel is already very strong, but it still pays for multiple full-row passes and shared-memory atomic serialization. GVR keeps exactness, but uses temporal structure in the workload to reduce those passes.

Why Temporal Correlation Exists#

What We Observe in Real Decode Traces#

A key empirical observation is that the Top-K index sets in DSA have strong temporal stability across decode steps: the important tokens at step $t$ often overlap substantially with the important tokens at step $t-1$.

We measured this on real DeepSeek-V3.2 decode-stage indexer logits using a long-context coding prompt from SWE-bench-derived LongSeqTasks. Concretely, the prompt used here is entry 1 of the 64K bucket, built from repository files of a single SWE-bench task; its actual prompt length is 68,665 tokens, followed by 2,025 decode steps.

_{Figure 2. Raw Top-K overlap between consecutive decode steps across layers. Layers 20-60 show 35-50% average raw overlap, while Layers 0 and 1 are much lower.}

Two details help interpret the figure:

Figure 2 reports raw overlap, i.e. exact set intersection between consecutive steps.
A related but different view is shifted overlap, where the previous-step indices are shifted by +1 before comparison. Shifted overlap is useful for intuition, but Figure 2 itself uses the stricter raw metric.

The practical takeaway is straightforward: for most layers, the previous step’s Top-K is already a useful prediction set for the next step.

Why RoPE and YaRN Make This Predictable#

The empirical observation is not accidental. It is tied to how the DSA indexer uses RoPE-encoded query and key tensors.

At a high level:

the positional score contribution depends on relative position $\Delta = n - m$,
so the positional score landscape is Toeplitz-like,
and advancing the query by one decode step produces a smooth translation rather than a random reshuffling.

Temporal Correlation and Toeplitz Structure

_{Figure 3. Intuition for temporal correlation in decode-stage Top-K. The previous step's Top-K is not exact ground truth for the next step, but it remains a strong local predictor because the score landscape moves smoothly.}

YaRN matters here too: by preserving meaningful long-range peaks, it makes the Top-K set span both nearby and more distant positions, which strengthens the prediction signal available from the previous step.

For the full theoretical derivation and the formal connection between RoPE/YaRN structure and temporal Top-K stability, see the GVR Top-K arXiv paper.

How GVR Works#

GVR turns the previous step’s Top-K into a prediction-guided exact selection pipeline.

At a high level, the algorithm does four things:

estimate a threshold from the predicted indices,
verify that threshold with a small number of full-row passes,
collect the resulting candidates efficiently,
finish exact selection in shared memory.

_{Figure 4. Guess-Verify-Refine (GVR) pipeline. The previous step's Top-K provides the warm-start; exact selection is preserved by verification and in-shared-memory refinement.}

Phase 1: Guess from Previous-Step Top-K#

The previous step’s Top-K indices are passed in as preIdx. GVR reads those positions and computes:

pmin
pmax
pmean

The most important quantity is pmean, which gives a much better initial threshold estimate than the unconditional mean of the full score vector. In real decode workloads, the hit ratio of preIdx is much larger than the trivial baseline $K/N$, so pmean is already biased toward the Top-K tail.

Phase 2: Verify with Secant Threshold Search#

Starting from pmean and the bracket [pmin, pmax], GVR runs a secant-style threshold search over the monotone count function:

$$f(T) = |{i : x_i \geq T}|$$

The target is to find a threshold $T^\ast$ such that the candidate count falls into a safe refinement range:

$$K \leq f(T^\ast) \leq C$$

where $C$ is a bounded candidate capacity.

_{Figure 5. Secant-style threshold search in GVR. With a good starting estimate, the search typically converges in 1-2 iterations on real decode data.}

This is where temporal correlation directly turns into speedup: instead of blindly narrowing the search space with distribution-agnostic radix passes, GVR starts close to the relevant tail and needs only a small number of global scans in the common case.

Phase 3: Ballot-Free Candidate Collection#

Once a valid threshold is found, GVR collects all elements above threshold into shared memory.

This stage is optimized around two ideas:

count-cache reuse: Phase 2’s per-thread counts are reused so Phase 3 can avoid a redundant full-row count pass,
ballot-free write layout: candidates are written via prefix-sum-derived offsets rather than per-element ballot/atomic machinery.

This reduces synchronization overhead and avoids the compiler barriers associated with __ballot_sync, which is especially important on Blackwell where streaming memory behavior matters.

Why This Is Still Exact#

GVR is not an approximate pruning method. The heuristic is used only to get a good threshold estimate quickly; exactness is preserved by the verification step and the final shared-memory refinement.

In practice:

the output Top-K set matches torch.topk on tested sequence lengths,
ties remain non-deterministic in the same way as the production kernel,
and end-to-end model behavior remains unchanged within observed variance.

For the full correctness argument, the exact candidate-set condition, detailed pseudocode, and the GPU performance model, see the GVR Top-K arXiv paper.

TensorRT-LLM Integration#

Where GVR Fits in the DSA Stack#

GVR is integrated into the existing TensorRT-LLM DSA decode path rather than as a separate code path. The high-level dispatch is:

Python-level DSA code decides whether a small-batch CuTE DSL path is appropriate,
otherwise the request falls through to the C++ indexer_topk_decode operator, carrying the previous-step Top-K indices and a graph-safe scratch buffer when GVR is enabled,
inside that operator, the GVR Top-K dispatcher selects the fast path only when the heuristic prerequisites and hardware-aware (batch size, sequence length) thresholds are met,
otherwise the original insertion/radix-select fallback chain is used.

_{Figure 7. Full decode-stage Top-K dispatch in TensorRT-LLM. GVR takes priority only when `preIdx`, scratch buffers, and hardware-aware thresholds are satisfied; otherwise dispatch falls back to the original insertion/radix-select pipeline.}

From a system point of view, this fits naturally into the broader TensorRT-LLM sparse attention stack described in Tech Blog 17, while specifically accelerating the DSA Top-K selector discussed in Tech Blog 15.

How to Enable It#

GVR is controlled by the DSA configuration and is enabled through the existing YAML config path used by trtllm-serve, trtllm-bench, and trtllm-eval.

Example:

sparse_attention_config:
  algorithm: dsa
  index_topk: 2048
  enable_heuristic_topk: true

The same option is also available through the Python LLM API:

from tensorrt_llm import LLM
from tensorrt_llm.llmapi import DeepSeekSparseAttentionConfig

llm = LLM(
    model="<model>",
    sparse_attention_config=DeepSeekSparseAttentionConfig(
        index_topk=2048,
        enable_heuristic_topk=True,
    ),
)

The current GVR implementation supports index_topk=2048. Support for index_topk=512 and index_topk=1024 is planned but not enabled yet, so those configurations continue to use the production insertion/radix Top-K path.

Looking ahead, DeepSeek V4-style DSA workloads are expected to make this extension more important: the indexer Top-K can move to smaller index_topk settings such as 512 or 1024 while still operating over long decode-time sequences. GVR Top-K is being extended to cover those configurations so the same temporal-correlation idea can accelerate both today’s 2048-wide selector and upcoming smaller-K long-context selectors.

Pass that config with the usual DSA deployment flow, for example:

trtllm-serve <model> --config config.yaml

trtllm-bench --model <model> throughput --config config.yaml

At the operator boundary, the C++ path receives both the current logits and the previous-step Top-K indices:

torch.ops.trtllm.indexer_topk_decode(
    logits,
    kv_lens,
    indices,
    next_n,
    topk,
    pre_idx=pre_idx,
    heuristic_scratch=heuristic_scratch,
)

GVR Top-K does not add another user-facing API beyond this configuration flag. The public option remains enable_heuristic_topk, and the public launcher/operator names keep the existing heuristic_* spelling for compatibility. Internally, the single-CTA micro-kernel is named after the algorithm (gvrTopKJob / gvrTopKKernel).

Two environment variables are useful for benchmarking and debugging:

TRTLLM_HEURISTIC_NMIN=<int> overrides the small-sequence lower bound (12288 by default, valid range [1024, 200000]).
TRTLLM_SCHEMEX_DEBUG=1 prints the per-launch GVR dispatcher decision, including the SM/L2-derived batch threshold and the small-N route marker.

When It Falls Back to Radix Select#

The GVR fast path is only taken when the required conditions are satisfied. Notable fallback cases include:

prefill (no previous-step Top-K yet),
missing or invalid preIdx / scratch feedback buffers,
non-contiguous logits layout, index_topk values other than the currently supported 2048, or unsupported hint size,
short rows below the small-sequence threshold,
large batches above the SM/L2-derived batch threshold,
very long rows where the implementation still prefers the existing split-work radix path,
unsupported architectures.

In other words, enabling GVR is designed to be low-risk: if the heuristic path is not applicable, TensorRT-LLM simply continues to use the production radix-select implementation.

Today, the heuristic path is targeted at Blackwell (sm_100+). On older architectures, the flag is effectively ignored.

Results#

Single-Operator Performance#

On synthetic data, GVR shows the expected scaling behavior: there is fixed overhead at short sequence length, but the reduced pass count wins as $N$ grows. Production dispatch uses a hardware-aware GVR gate rather than a single hard crossover: the default small-sequence lower bound is 12288, while a hardware-aware batch threshold routes large-batch or L2-pressure cases back to radix.

_{Figure 8. On random synthetic data, standalone GVR reaches parity around 16K sequence length and then increasingly outperforms the production radix-select kernel as context grows. In production, the GVR dispatcher starts from the validated 12,288-token lower bound and falls back when the `(batch size, sequence length)` cell is better served by radix.}

The more important result is real decode data.

We evaluate on real DeepSeek-V3.2 decode-stage indexer logits captured from a 68,665-token SWE-bench-derived long-context coding prompt, using 17 sampled decode steps across 9 representative layers. Under those conditions:

the average speedup is 1.88x,
the per-layer range is 1.57x-2.02x,
and the best per-step measurement reaches 2.42x.

_{Figure 9. GVR consistently outperforms the production radix-select baseline across all evaluated layers on real DeepSeek-V3.2 decode traces.}

These real-data gains line up with the main design intuition:

high-correlation layers tend to converge in fewer Phase 2 iterations,
bounded distributions (for example beta-like layers) give cleaner thresholds,
but even the harder layers still remain meaningfully faster than the baseline.

For the full tables, per-layer breakdowns, and replay statistics of Phase 2 / Phase 4 iteration counts, see the GVR Top-K arXiv paper.

Ablation: Architecture vs Prediction Quality#

One of the most important questions is how much of the gain comes from better prediction versus better exact-kernel structure.

The ablation below answers that directly:

preIdx Source	Overlap ($\alpha$)	Kernel Latency	vs. Radix Baseline
No preIdx (radix fallback)	0%	43.7 us	1.00x
Random indices	~2.9%	30.9 us	1.44x
Prev-step Top-K (L20-60)	~44%	22.7 us	1.94x
Prev-step Top-K (L0-1)	~1.5%	27.2 us	1.65x

This is a useful way to think about GVR:

architecture alone already matters: even random preIdx gives 1.44x,
temporal prediction adds another layer of gain: high-correlation layers reach 1.94x,
and the kernel degrades gracefully instead of collapsing when prediction quality is weak.

End-to-End Accuracy#

We validated that enabling GVR does not introduce measurable end-to-end quality regression in the TensorRT-LLM stack. Using trtllm-eval on DeepSeek-V3.2 NVFP4 on B200, all observed deltas stay within run-to-run variance across:

MMLU
GSM8K
GPQA-Diamond
LongBench V1

The high-level conclusion is the important one for this blog: GVR accelerates the decode-stage Top-K path without introducing measurable end-to-end accuracy regression.

For the detailed benchmark table and discussion of why long-context evaluation is the most representative stress test for this path, see the GVR Top-K arXiv paper.

End-to-End TPOT Reduction#

To isolate the length-scaling behavior of the decode-stage optimization, we measured fixed-OSL min-latency serving on DeepSeek-V3.2-Exp NVFP4 with TensorRT-LLM on B200 x8 under TEP8 parallelism.

The benchmark is intentionally set up so that decode-time Top-K is a meaningful fraction of end-to-end latency:

batch size = 1 for min-latency serving,
TEP8 with tp=8, ep=8,
chunked prefill enabled with max_num_tokens=8192,
FP8 KV cache with tokens_per_block=64 and free_gpu_memory_fraction=0.8,
CUDA Graph padding enabled,
real SWE-bench-derived prompts tokenized into trtllm-bench dataset format,
and interleaved A/B execution so the radix baseline and GVR path are run back-to-back under the same GPU conditions.

For each configuration we use 5 requests and repeat the A/B pair 3 times. The fixed-OSL figure and table below focus on the OSL=1K slice because it is a clean way to compare how the benefit scales with input length and speculative decoding depth without mixing in very short or very long output tails.

_{Figure 10. End-to-end TPOT reduction at fixed OSL=1K. GVR helps more as context length grows, and the gain is diluted as MTP becomes more aggressive.}

Context Bucket	MTP = 0	MTP = 1	MTP = 3
64K	5.47%	4.36%	2.40%
100K	7.52%	6.30%	3.45%

The fixed-OSL=1K slice shows the two expected trends. First, GVR becomes more valuable as context grows: at MTP=0, TPOT reduction rises from 5.47% at 64K to 7.52% at 100K. Second, speculative decoding dilutes the per-step saving, but does not remove it: at 100K, the gain remains positive from MTP=0 through MTP=3.

The broader 16K-128K sweep shows the same pattern: gains increase with context length, and MTP reduces the magnitude through Amdahl’s law while preserving the direction. This is consistent with the single-operator story: the longer the context, the larger the decode-stage Top-K burden, and the more valuable the heuristic warm-start becomes.

Conclusion#

GVR shows that data-aware exact selection can outperform even highly tuned distribution-agnostic GPU Top-K kernels when the workload contains a strong temporal signal.

In DeepSeek Sparse Attention decode, the previous step’s Top-K is exactly such a signal:

it gives a much better initial threshold estimate,
reduces full-row passes in the common case,
lowers synchronization overhead in candidate collection,
and still preserves exact Top-K outputs.

That combination is what makes GVR interesting: it is not an approximate pruning trick and it is not just a micro-optimization of radix select. It is a workload-aware exact algorithm that fits naturally into TensorRT-LLM’s DSA path and converts temporal stability into real operator-level and end-to-end gains. As DSA-style models evolve toward long-sequence index_topk=512/1024 use cases, the same GVR Top-K direction is expected to remain useful beyond the current index_topk=2048 deployment.