Qwen3-VL DeepStack

Every image Qwen3-VL processes becomes visual tokens, and DeepStack injects them into the language decoder at three different depths, with about 2,752 tokens at each. Those tokens make up a large part of the model’s context and compute, but nobody had measured how much of that visual information actually carries weight.

The key question was whether the three injection groups matter differently: whether shallow features can tolerate aggressive pruning while deeper features cannot, and whether giving each depth its own token budget would outperform a uniform one. The experiments showed that the three depths were more interchangeable than expected, pointing to a simpler compression strategy.

1. 01

   Probe the mechanism

   First, I measured what is actually happening inside the model. DeepStack injects vision features into the first three decoder layers, using vision-encoder layers 5, 11, and 17. Each group contains the same number of tokens, but the average token strength (L2 norm) increases with depth: 14.6, 17.0, and 23.6. That was the first sign that the three groups were not contributing equally. Compared with the overall median token strength, 68% of shallow tokens are low-value, compared with 51% in the middle group and just 22% in the deep group.

   All of this was measured on one model: Qwen3-VL-2B-Instruct in fp16 on a Colab T4. It was large enough to show real DeepStack behavior, but small enough to run the full set of experiments. Whether the same pattern holds at 8B and larger models remains untested.

   

2. 02

   Instrument without touching the model

   I built an instrumentor that measures feature distributions, latency, and memory around the injection points without changing the model. This keeps every measurement independent of the instrumentation itself. The coefficient of variation of per-token norms was 0.61, 0.45, and 0.42 from shallow to deep, suggesting the shallow group has more room for pruning while the deep group is more consistent.

   The model never runs locally because the machine has no CUDA. Code is pushed to Colab through GitHub, and results come back the same way. This makes every experiment a committed, reproducible run instead of relying on notebook state.

   

3. 03

   Ablate by group

   I removed each injection group one at a time across four tasks to measure how much each mattered. The effect depended on the task: removing one group changed general VQA and counting by at most 1.3%, while removing the deep group reduced text recognition accuracy by 4.0%. Removing the shallow group actually improved text recognition by 1.3%, meaning those features were hurting performance on that task.

   Dropping a group zeros its refinement but keeps the base token, isolating information loss from sequence length. These results say nothing about speed, and four benchmarks are not enough to establish generality.

4. 04

   Score tokens within a group

   We compared five different selection strategies -random, spatial-uniform, activation magnitude, diversity, and a hybrid approach- across three different keep ratios. Overall, activation magnitude performed the best, while the hybrid method was the most consistent across different pruning levels. We also found that attention-based selection consistently performed worse than simply using feature magnitude on every task. As pruning became more aggressive, its performance increasingly approached that of random selection. This is an important result to highlight because attention-based selection is arguably the most intuitive approach one might try first.

   We report the negative result because it shows that attention-based selection does not outperform simpler methods. The ranking is based on 100 sweeps, which is enough to compare the methods overall, but not to distinguish between very close results.

5. 05

   Budget across DeepStack groups

   We used a water-filling allocator to distribute a fixed token budget across the three depths based on their measured sensitivity. At the same number of retained tokens, it performed as well as or worse than uniform allocation in every case. Global selection across groups also only matched uniform allocation. This suggests that the depth groups contain substantial redundancy: their individual ablation results do not translate into additive importance. In fact, an entire injection group can be removed with less than a 1% drop in accuracy.

   The practical recommendation is to use uniform keep-ratios, feature-based scoring, and no per-depth tuning. The keep-ratio should instead be chosen based on task difficulty: roughly 15% for general VQA, 30% for text, and 50% for documents. One important caveat is that with n=300, the validation confidence intervals overlap across the methods, so we can say they perform similarly, not that one is better. A per-group advantage may still exist for OCR-heavy text tasks, which this sweep does not rule out.

• Designed and ran the complete five phase study using a 2B-parameter model in FP16 across four vision-language benchmarks.
• Built the non-invasive instrumentation, the five token scoring methods, and the water filling budget allocator.
• Showed that attention-based token selection underperforms feature-magnitude selection, and reported this negative result in the paper.

Decoder vs vision-encoder timeBaseline generation takes 6,689 ms compared with 1,459 ms for the baseline setup, showing that decoder-side tokens are the main source of generation cost.

4.6×

Prunable below median strengthper injection group, shallow → deep

68 / 51 / 22%

General VQAAccuracy drops by only 0.2% at n=300, with confidence intervals overlapping the baseline.

keep 15%

Text recognition−0.3% accuracy

keep 30%

Documents+0.2% accuracy

keep 50%

Whole group droppableat ≤1% accuracy

1 of 3

Per-group budgetingPerformed as well as or worse than uniform allocation at every ratio, based on an n=100 sweep and n=300 validation.

no gain

Measured asThis is not a wall-clock speedup: the base token remains in the sequence. We only claim a speedup once tokens are actually removed.

feature zeroing

The whole study: instrumentation, five scoring methods, the allocator, and the paper draft.

Forthcoming · Paper in preparation — publication expected later in 2026.

PyTorch · Transformers · Qwen3-VL · CUDA · Colab

• Repository