Cache a copy of the conv tail, not a view of in_proj_qkv (semantic-router#5891)

#2

Fixes https://github.com/vllm-project/semantic-router/issues/5891.

The state pass cached each Gated DeltaNet layer's convolution tail as a slice of that layer's whole in_proj_qkv output, so the cache kept the full [1, conv_dim, S] projection on the GPU until the request ended. This caches a copy of the tail instead, in both paths:

  • StateOnce._delta: tail = mixed[:, :, -window:].clone()
  • FusedStateOnce._fused_delta: entry = (tail.clone(), final). The .contiguous() it replaces is a no-op on the batch-1 slice, so it returned the same view.

MODEL_MANIFEST.json updates files_sha256, runtime.files_sha256 and files_bytes for d3_runtime.py; identity is unchanged. The repeated K/V on the fused path is left for its own issue.

Memory, measured on d3-ultra BF16 through d3_server.py (unfused StateOnce path, one 96 GB RTX PRO 6000): the cached state drops from 1 MiB to 64 KiB per state token (16x). Unpatched, a 34,090-token state block with 19 questions ran out of memory; patched, state blocks up to the 131,072-token limit run, peaking at 35.0 GiB at 129,897 tokens. Probabilities and answers were bitwise identical on 21 requests.

This package ships the same d3_runtime.py as vllm-sr/d3-ultra, which gets the same change.

adaamko changed pull request status to merged

Sign up or log in to comment