Qwen3-VL DeepStack

Every image Qwen3-VL looks at becomes visual tokens, and DeepStack injects them into the language decoder at three different depths — roughly 2,752 tokens per depth. Those tokens are most of what the model spends its context and its compute on, yet nobody had published how much of that budget actually carries weight.

This is not a KV-cache trick or generic token trimming. The question is whether the three injection groups matter differently — whether a shallow injection tolerates aggressive pruning while a deep one cannot, so that a per-depth token budget should beat a uniform one. That hypothesis turned out to be wrong, and why it is wrong became the most useful thing the study found.

1. 01

   Probe the mechanism

   First, establish what is actually there. Injection happens at the first three decoder layers, drawing from vision-encoder layers 5, 11 and 17. Every group carries the same number of tokens, but per-token strength (L2 norm) grows with depth — 14.6, then 17.0, then 23.6 — the first sign that the groups are not interchangeable. Measured against the overall median strength, 68% of the shallow group's tokens are low-value, 51% of the middle group's, and only 22% of the deep group's.

   All of this is measured on one model — Qwen3-VL-2B-Instruct in fp16 on a Colab T4. Big enough to have real DeepStack behaviour, small enough to sweep; whether the pattern holds at 8B and beyond is untested.

   

2. 02

   Instrument without touching the model

   A non-invasive instrumentor captures feature distributions, latency and memory around the injection points, so every later measurement runs against a model whose behaviour has not been altered by the measurement. Coefficient of variation across per-token norms comes out at 0.61, 0.45 and 0.42 — the shallow group is the most prunable, the deep one the most fragile.

   The model never runs locally; the machine has no CUDA. Code moves local to Colab through GitHub and results come back the same way, which forces every experiment to be a committed, re-runnable artifact rather than a notebook state.

   

3. 03

   Ablate group by group

   Zeroing one injection group at a time across four tasks shows sensitivity is task-dependent rather than uniform. Single-group removal costs at most 1.3% on general VQA and counting, but the deep group is worth 4.0% on text recognition. And dropping the shallow group improves text recognition by 1.3% — on that task it was contributing noise.

   Dropping a group means zeroing its refinement while the base token stays in the sequence — the model loses the information, not the length. That keeps the comparison clean, and it also means these numbers claim nothing about speed. Four benchmarks is a screen, not a survey.

4. 04

   Score tokens within a group

   Five selection methods — random, spatial-uniform, activation magnitude, diversity, and a hybrid — evaluated at three keep-ratios. Activation magnitude wins overall and the hybrid is the most robust. Separately, attention-based selection loses to plain feature magnitude on every task and decays toward random as pruning gets aggressive, which is worth stating precisely because it is the method most people would reach for first.

   That negative result is kept in the paper rather than dropped — a method that sounds principled and does not work is more useful to the next person than one more confirmation of the method that does. The ranking itself comes from n=100 sweeps: enough to order the scorers, not to split the close ones.

   

5. 05

   Budget across groups, and reject the hypothesis

   A water-filling allocator distributes a fixed token budget across the three depths by measured sensitivity. Compared at equal retained-token count, it is equal to or worse than uniform allocation in every cell tested, and global cross-group selection merely ties uniform. The cause is that the depth groups are mutually redundant: single-group ablations look informative because importance is not additive across them, and an entire injection group turns out to be droppable for under 1% accuracy.

   The practical recommendation the paper lands on is therefore the boring one, which is the point: uniform keep-ratios, feature-based scorers, no per-depth tuning, and the ratio chosen by task difficulty — roughly 15% for general VQA, 30% for text, 50% for documents. Two honest caveats ride along: at n=300 the validation confidence intervals all overlap, so the claim is flat, not better — and a per-group win may still exist on OCR-heavy text tasks, which this sweep cannot rule out.

   

• Designed and ran the full five-phase study, on a 2B model in fp16 across four vision-language benchmarks.
• Built the non-invasive instrumentation, the five token-scoring methods, and the water-filling budget allocator.
• Established that attention-based token selection underperforms feature magnitude, and kept it in the paper as a negative result.
• Wrote the paper draft around the rejected hypothesis rather than around a result that would have been easier to sell.

Decoder vs vision-encoder time6,689 ms vs 1,459 ms per baseline generation — why decoder-side tokens dominate

4.6×

Prunable below median strengthper injection group, shallow → deep

68 / 51 / 22%

General VQA−0.2% accuracy at n=300; confidence intervals overlap the baseline

keep 15%

Text recognition−0.3% accuracy

keep 30%

Documents+0.2% accuracy

keep 50%

Whole group droppableat ≤1% accuracy

1 of 3

Per-group budgetingequal or worse than uniform at every ratio — n=100 sweep, n=300 validation

no gain

Measured asnot wall-clock — the base token stays in the sequence; no speedup is claimed until rows are actually cut

feature zeroing

Transcribed from the project’s own documentation and not yet re-checked against the source runs. Marked until it is.

The whole study: instrumentation, five scoring methods, the allocator, and the paper draft.

PyTorch · Transformers · Qwen3-VL · CUDA · Colab

• Repository