DeepSeek：2025 DeepSeek-V3.2-Exp技术报告（英文版）（6页）.pdf

1、DeepSeek-V3.2-Exp:Boosting Long-Context Efficiencywith DeepSeek Sparse AttentionDeepSeek-AIAbstractWe introduce DeepSeek-V3.2-Exp,an experimental sparse-attention model,which equipsDeepSeek-V3.1-Terminus with DeepSeek Sparse Attention(DSA)through continued train-ing.With DSA,a fine-grained sparse at

2、tention mechanism powered by a lightning in-dexer,DeepSeek-V3.2-Exp achieves significant efficiency improvements in both trainingand inference,especially in long-context scenarios.The model checkpoints are available athttps:/huggingface.co/deepseek-ai/DeepSeek-V3.2-Exp.1.ArchitectureCompared with De

3、epSeek-V3.1-Terminus,the last version of DeepSeek-V3.1,the only architec-tural modification of DeepSeek-V3.2-Exp is the introduction of DeepSeek Sparse Attention(DSA)through continued training.Prototype of DSA.The prototype of DSA primarily consists of two components:a lightningindexer and a fine-gr

4、ained token selection mechanism.The lightning indexer computes the index score,between the query token h Rand apreceding token h R,determining which tokens to be selected by the query token:,=1,ReLU?q,k?,(1)wheredenotes the number of indexer heads;q,Rand,Rare derived from thequery token h;and k Ris

5、derived from the preceding token h.We choose ReLU as theactivation function for throughput consideration.Given that the lightning indexer has a smallnumber of heads and can be implemented in FP8,its computational efficiency is remarkable.Given the index scores,for each query token h,our fine-grained

6、 token selectionmechanism retrieves only the key-value entriesccorresponding to the top-k index scores.Then,the attention output uis computed by applying the attention mechanism between thequery token hand the sparsely selected key-value entries c:u=Attn?h,?c?,Top-k?,:?.(2)concatenateInput Hidden,Mu