Skip Softmax Attention is a drop-in post-training sparse attention method and the productized implementation of BLASST in NVIDIA TensorRT-LLM. It is available for accelerating both LLM workloads and visual-generation workloads.

The original article was published in NVIDIA TensorRT-LLM and is embedded here for convenience.

As context lengths grow from thousands to hundreds of thousands of tokens, attention computation becomes a major bottleneck in long-context LLM inference. TensorRT-LLM provides a sparse attention framework that supports techniques like KV cache compression and sparse pattern prediction, featured in RocketKV and DSA. However, these require framework-level support—additional token selection steps, auxiliary data structures, and kernel modifications are required, compared to the vanilla attention architecture. This complexity introduces runtime overhead that can offset performance gains, particularly when context lengths are not long enough to amortize the extra work.

In this blog, we introduce Skip Softmax Attention, a drop-in sparse attention technique that is designed to accelerate the existing pretrained models that use standard attention mechanisms like MHA, GQA, or MLA. Skip Softmax Attention is built on top of the Flash Attention algorithm and only requires modifying the existing attention kernels. Due to this simplicity, the end-to-end performance gain is more predictable. In addition, it is only an approximation method of the attention kernel computation, making it compatible with nearly all the other features, such as FP8 attention, KV cache reuse, chunked prefill etc.

Method Overview

The idea of Skip Softmax Attention is to compare the local maximum $\tilde{m}_i^{(j)}$ of $Q \cdot K^T$ with the running global maximum $m_i^{(j)}$, and skip the softmax (exp) and BMM2 calculation for blocks that are below a certain threshold $\lambda$:

In this way, we can indirectly control the sparsity via the threshold. The threshold is set to be inversely proportional to the context length, i.e., the longer the context, the smaller the threshold is needed to achieve the same sparsity.

The method is fully dynamic, and can be applied to both the prefilling and decoding. The algorithm of Skip Softmax Attention is described in the paper BLASST: Dynamic Blocked Attention Sparsity via Softmax Thresholding. We have also published a Developer Blog for explanation. Please refer to these resources for in-depth dive into the algorithm details. We will focus on the application of Skip Softmax Attention in TensorRT-LLM to accelerate long-context inference.

Example Usage

Enabling Skip Softmax Attention is pretty simple: we only need to configure the SkipSoftmaxAttentionConfig and pass it to the LLM API:

1
2
3
4
5
6
7
8
9
10
11
12
13

from tensorrt_llm import LLM
from tensorrt_llm.llmapi import SkipSoftmaxAttentionConfig

sparse_attention_config = SkipSoftmaxAttentionConfig(threshold_scale_factor=1000.0)

# Additionally, the threshold_scale_factor for prefill and decode could be separately configured.
sparse_attention_config = SkipSoftmaxAttentionConfig(threshold_scale_factor={"prefill": 1000.0, "decode": 500.0})

llm = LLM(
   model="Qwen/Qwen3-30B-A3B-Instruct-2507",
   sparse_attention_config=sparse_attention_config,
   # Other LLM arguments...
)

The configuration could also be specified through the extra LLM API options YAML file. An example to launch an OpenAI-compatible endpoint is shown below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

cat >extra_llm_api_options.yaml <<EOF
sparse_attention_config:
    algorithm: skip_softmax
    threshold_scale_factor: 1000.0
EOF

# Additionally, the threshold_scale_factor for prefill and decode could be separately configured.
cat >extra_llm_api_options.yaml <<EOF
sparse_attention_config:
    algorithm: skip_softmax
    threshold_scale_factor:
        prefill: 1000.0
        decode: 500.0
EOF

trtllm-serve Qwen/Qwen3-30B-A3B-Instruct-2507 --extra_llm_api_options extra_llm_api_options.yaml

The actual threshold value equals the threshold_scale_factor divided by the context length. NVIDIA Model Optimizer will support the calibration to automatically determine the required value given the target sparsity. We will use Qwen3-30B-A3B-Instruct-2507 as the example model for testing, and the calibrated threshold scale factors are listed below:

Accuracy Evaluation

We evaluate the accuracy of Skip Softmax Attention using LongBench V1 and V2. LongBench V1 is a comprehensive benchmark for medium-to-long context understanding, with average sequence length of 10k tokens. LongBench V2 is a harder benchmark that contains longer sequences, and we pick its medium subset and truncate the prompt length to 256k due to the limit of the native context window of the model. The average sequence length of LongBench V2 is 130k tokens.

The evaluation results (on H200) are summarized in the table below:

Note that the number of samples in LongBench V2 is very small (200), so the result is subject to large variance. You will see non-monotonic relationship between sparsity and accuracy. We recommend to look at LongBench V1 (5000 samples) for inspecting the accuracy loss trend.

Performance Benchmark

Skip Softmax Attention is supported on both Hopper and Blackwell GPUs and builds on top of the SoTA performance of the TensorRT-LLM’s attention kernels. Hopper prefilling is implemented in fmha_v2, Hopper decoding is implemented in XQA, and Blackwell is implemented in trtllm-gen.

Kernel Performance

We provide the performance data of the attention kernels under different achieved sparsity by specifying the threshold. The micro-benchmarking is performed under these configs: q_heads=64, kv_heads=4, head_dim=128, seqlen=16k/64k. Both BF16 and FP8 attention are supported. For prefilling, batch size is set to 1; for decoding, batch size is 64.

As a reference, the baseline performance data without Skip Softmax Attention are listed below (you can fill in the numbers).

Prefill Baseline (we report compute performance in TFLOP/s):

Decode Baseline (we report memory bandwidth in TB/s):

The following figures plot speedup vs. achieved sparsity, on top of the baseline performance:

Hopper (H200) Prefill Decode

Blackwell (B200) Prefill Decode

Skip Softmax Attention could further boost the performance of FP8 attention, though the gain is less significant compared to BF16.

End-to-end Performance

We benchmark the end-to-end performance to demonstrate the benefit of Skip Softmax Attention. Due to the quadratic complexity of the attention, the TTFT in long-context scenarios is often a severe blocker for real-world usage. Skip Softmax Attention can significantly reduce the TTFT by accelerating the prefilling kernel, and the TPOT can also be reduced if the context length is long enough. The experiment is conducted on a single H200 or B200 GPU, using the exact same dataset as the accuracy evaluation.

LongBench V1, avg ISL=10k, OSL=6:

LongBench V1 results are reported with concurrency 64. Due to the nature of in-flight batching, the decoding requests might be piggybacked with the prefilling requests, so the TPOT is relatively high.

LongBench V2, avg ISL=130k, OSL=200:

Due to the extremely long context length, we only run LongBench V2 with concurrency 1. In this scenario, the prefilling/decoding is better separated and we can observe how is TTFT/TPOT affected by the sparsity. Note that the speedup for decoding is less pronounced under small batch size. Small batch size and small number of heads (with TP) are more close to real-world usage for long-context serving due to the limit of SLO, and we are actively optimizing the decoding performance under such scenarios.

Reproduction

We provide the commands to reproduce the results in the previous context, as a showcase of how to evaluate the accuracy and benchmark the performance for Skip Softmax Attention.

Accuracy evaluation (LongBench V1/V2)

Both LongBench V1 and V2 are integrated into the TensorRT-LLM accuracy test suite, trtllm-eval. Here are the example scripts to run the accuracy evaluation:

1
2
3
4
5
6
7
8
9
10

# Dump the extra LLM API options YAML file.
cat >extra_llm_api_options.yaml <<EOF
kv_cache_config:
    free_gpu_memory_fraction: 0.8
sparse_attention_config:
    algorithm: skip_softmax
    threshold_scale_factor:
        prefill: ${thr_prefill}
        decode: ${thr_decode}
EOF

1
2
3
4
5
6

# Evaluate LongBench V1.
trtllm-eval --model Qwen/Qwen3-30B-A3B-Instruct-2507 \
 --max_batch_size 64 --max_num_tokens 100000 \
 --extra_llm_api_options extra_llm_api_options.yaml \
 longbench_v1 \
 --output_dir ${OUTPUT_DIR}  # Dump dataset for perf benching

1
2
3
4
5
6
7
8

# Evaluate LongBench V2.
trtllm-eval --model Qwen/Qwen3-30B-A3B-Instruct-2507 \
--max_batch_size 1 --max_num_tokens 262144 \
--extra_llm_api_options extra_llm_api_options.yaml \
longbench_v2 \
--length medium \  # Medium subset of LongBench V2
--max_input_length 256000 \  # Truncate the prompt length to 256k
--output_dir ${OUTPUT_DIR}  # Dump dataset for perf benching

End-to-end performance (TTFT/TPOT)

The option --output_dir in trtllm-eval will dump the dataset in the format required by trtllm-bench as dumped_ids.json. After getting the data, we can perform end-to-end benchmarking:

1
2
3
4
5
6
7

# Benchmark on LongBench V1.
trtllm-bench --model Qwen/Qwen3-30B-A3B-Instruct-2507 \
throughput --dataset ${OUTPUT_DIR}/dumped_ids.json \
--concurrency 64 --max_batch_size 64 --max_num_tokens 100000 \
--extra_llm_api_options extra_llm_api_options.yaml \
--warmup 0 --streaming \
--report_json ${OUTPUT_DIR}/report.json

1
2
3
4
5
6
7

# Benchmark on LongBench V2.
trtllm-bench --model Qwen/Qwen3-30B-A3B-Instruct-2507 \
throughput --dataset ${OUTPUT_DIR}/dumped_ids.json \
--concurrency 1 --max_batch_size 1 --max_num_tokens 262144 \
--extra_llm_api_options extra_llm_api_options.yaml \
--warmup 0 --streaming \
--report_json ${OUTPUT_DIR}/report.json

Conclusion

Skip Softmax Attention is a kernel-based solution for accelerating the attention. Due to the design that BMM1 ($Q \cdot K^T$) in the attention kernel is not skipped, the performance gain is capped to 1.8x at kernel level. Nevertheless, it excels at achieving high sparsity with minimal accuracy degradation, and is especially effective in the medium-to-long context scenarios where previous methods like MInference cannot well handle, because the introduced runtime overhead may not pay off the speedup of the attention kernel. The drop-in nature of Skip Softmax Attention makes it a flexible, easy-to-use method for accelerating long-context inference. The Skip Softmax Attention kernels will also be available in FlashInfer for adoptions by the open-source community.

This one-sided AlltoAll design for MoE EP communication originated in NVIDIA TensorRT-LLM and has since been ported to FlashInfer. It is currently the fastest available MoE EP AlltoAll implementation for inference on NVLink and has been integrated into vLLM and SGLang. A presentation (in Chinese) is also available.

The original article was published in NVIDIA TensorRT-LLM and is embedded here for convenience.

Large-scale Mixture-of-Experts (MoE) models have become the dominant architecture for open-source LLMs. For inference, a common parallel strategy is Attention Data Parallel (DP) + MoE Expert Parallel (EP): each GPU handles a portion of the requests (thus tokens) for attention and a portion of the experts for MoE. Since a token’s routed experts may reside on other GPUs, efficient communication is crucial for performance.

In this blog, we introduce NVLinkOneSided AlltoAll, the MoE communication kernels in TensorRT LLM designed for NVLink-connected systems (DGX B200, GB200 NVL72, etc.). We describe the key design choices, implementation details and performance benchmark results.

Background

There are two communication patterns that can be used between attention DP and MoE EP:

1. AllGather + ReduceScatter. After attention, AllGather brings all tokens to every rank. Each rank computes MoE on the full token set using its local experts. After MoE, ReduceScatter sums the partial results back to each rank.

2. AlltoAll (Dispatch + Combine). AllGather is redundant: if a token is not routed to any expert on a given rank, it does not need to be communicated there. The waste is especially significant when the EP size is larger than top_k, since a token is routed to at most top_k experts. AlltoAll sends each token only to the ranks that own its routed experts, eliminating the redundancy. The AlltoAll after attention is called dispatch and the AlltoAll after MoE is called combine.

Concretely, the total message size of AllGather/ReduceScatter scales as ep_size × num_tokens_per_rank × hidden_dim (including data movement to self), while AlltoAll scales as top_k × num_tokens_per_rank × hidden_dim, since each token is dispatched to at most top_k target ranks. When ep_size > top_k, AlltoAll communicates strictly less data.

Design Overview

NVLinkOneSided AlltoAll rests on three design ideas:

• One-sided data movement — each kernel only reads OR only writes peer memory, eliminating cooperative send/recv and the extra copies it implies.
• Rank-major recv buffer — slots are indexed by source rank with no top_k duplication, shrinking buffer size and saving redundant communication.
• Flexible interface — quantization-agnostic data path, with a small API that exposes the recv buffer directly to the MoE module.

The remainder of this section walks through the NVLink symmetric memory substrate that the kernels are built on, then each of the three ideas in turn.

NVLink Symmetric Memory

Communication across GPUs is enabled by CUDA’s Virtual Memory Management (VMM) APIs. Each GPU allocates a piece of physical memory and exports a shareable handle for its allocation. These handles are then exchanged across all participating GPUs, allowing each GPU to import the remote handles and map the remote memory into its own virtual address space. The result is symmetric memory: any GPU can read from or write to any other GPU’s portion of it via NVLink using standard instructions in kernels, as if they were normal global memory pointers. This works within a single NVLink domain — 8 GPUs on a DGX B200, or all 72 GPUs on a GB200 NVL72 rack.

Symmetric memory registration across GPUs. (Image courtesy: NVIDIA NCCL 2.27 Blog)

One-Sided Communication

NVLinkOneSided eliminates the send/recv pairing typical of collective communication.

Two-sided communication uses a send/recv model: the sender and receiver must cooperate. With direct NVLink put, the sender writes into the target rank’s symmetric memory, but the receiver still has to perform an additional data movement before the data lands in its final recv buffer. Common reasons for this extra step include:

• Layout reconciliation. The data lands in symmetric memory in an intermediate layout; the receiver re-permutes or repacks it into the final layout the downstream kernel expects.
• FIFO drain. Symmetric memory may be too small to hold the entire recv buffer, so it is used as a fixed-capacity FIFO; the receiver continuously drains it into local memory.

One-sided communication drops the cooperation entirely. Symmetric memory is the recv buffer, and the dispatch op returns tensor views directly into symmetric memory rather than allocating new tensors, so the MoE module consumes the dispatched data in place. In this way, extra local data movement on the receiver side is eliminated.

Rank-Major Buffer Layout

NVLinkOneSided uses a rank-major recv buffer: a tensor of shape [ep_size, max_tokens_per_rank, ...] split by source rank, where each rank’s slice holds the tokens that source sent to this rank. Each token is sent at most once per (src → tgt) pair — even if multiple of its top_k experts land on the same target rank — so within a slice, no token appears more than once. Some slots may be empty after dispatch, depending on routing.

The figure below contrasts this layout against the expert-major layout used by some other implementations (e.g. DeepEP Low Latency) on a small example: ep_size = 2, top_k = 2, 3 tokens per rank, 2 experts per rank. T2 is routed to one expert on each rank, so the rank-major recv buffer (left) holds it once, while the expert-major recv buffer (right) duplicates it across both expert sub-rows.

Rank-major layout has two advantages:

• Smaller buffer. The pre-allocated recv buffer is 1 / num_experts_per_rank of an expert-major buffer.
• Save duplicated communication. When top_k > ep_size it is impossible to avoid duplication in an expert-major layout.

The MoE module consumes the rank-major recv buffer directly, and it is up to the MoE module to decide whether explicit expert permutation is needed for GroupGEMM. For example, trtllm-gen MoE can efficiently load tokens from the raw rank-major buffer without additional permutation.

Interface Between Communication and MoE

Dispatch feeds the MoE module; combine collects its outputs. In TensorRT LLM, the MoE module takes 4 payloads as input. The dispatch kernel communicates all of them to target ranks at once; the combine kernel performs the reverse — gathering results back and reducing them with router weights:

1
2
3
4
5
6
7
8
9

# Dispatch: scatter local tokens to all ranks
def dispatch(
    hidden_states: Tensor,          # [local_num_tokens, hidden_size]
    hidden_states_sf: Tensor,       # [local_num_tokens, sf_size]
    token_selected_experts: Tensor, # [local_num_tokens, top_k]
    token_final_scales: Tensor,     # [local_num_tokens, top_k]
    ...
) -> Tuple[Tensor, ...]:
    # Returns same 4 payloads, each reshaped to [ep_size * max_tokens_per_rank, ...]

Here hidden_states is the (possibly quantized) activation tensor with hidden_size elements per token (or packed elements for certain quantization schemes — e.g., FP4 packs two elements per byte). hidden_states_sf carries the optional blockwise quantization scales. token_selected_experts contains the top_k routed expert ids per token, and token_final_scales holds the corresponding router weights.

The dispatch outputs are tensor views into symmetric memory — no allocation, no copy. The recv buffer is pre-allocated with ep_size * max_tokens_per_rank slots to accommodate the maximum number of tokens from all ranks, as described in Rank-Major Buffer Layout. The MoE module then performs GroupGEMM on the received payloads. Two points are worth noting:

• The MoE module only computes the experts on the local rank. For example, if a token’s token_selected_experts is [0, 1, 4, 7] and only experts [0, 1, 2, 3] reside locally, the MoE output for that token is the weighted sum of experts [0, 1] only.
• Some slots are empty after dispatch. The dispatch kernel sets token_selected_experts of empty slots to an invalid expert id (-1), so the MoE module knows to skip them. (For instance, trtllm-gen MoE consumes the raw-token recv buffer directly without re-permutation.)

The MoE module writes its output for each token to the same slot in the recv buffer. To obtain the final result, each rank combines the partial results from peer ranks.

1
2
3
4
5
6

# Combine: gather results back to local tokens
def combine(
    final_hidden_states: Tensor,    # [ep_size * max_tokens_per_rank, hidden_size]
    ...
) -> Tensor:
    # Returns [local_num_tokens, hidden_size]

Quantization-Agnostic Communication

NVLinkOneSided is quantization-agnostic. The AlltoAll kernel transports opaque bytes — quantization happens before dispatch (post-quant dispatch), and any recipe (FP8 block scales, MXFP8, NVFP4, BF16, …) plugs into the same code path. This has two consequences:

• Flexibility. The kernel does not need to change when the quantization format changes; the model’s recipe drives the dispatch payload size directly.
• Bandwidth savings. Communicating quantized data (e.g., NVFP4 reduces dispatch payload to ~28% of BF16) directly reduces the bytes transferred over NVLink, unlike implementations that must communicate in BF16 and quantize afterward.

Pre-dispatch quantization is naturally aligned with the model’s quantization recipe — the activations need to be quantized anyway before the MoE GEMMs. The combine path, by contrast, communicates in BF16 by default to match the kernel’s reduction behavior; pre-combine quantization would alter the model’s numerical output. Additionally, an optional low-precision combine path is available: the sender quantizes to FP8 before pushing into combine and the receiver dequantizes after pull, recovering most of the bandwidth saving with a small numerical impact from the round-trip quantization.

Implementation Details

Dispatch Kernel

The dispatch kernel pushes tokens from local memory into the target rank’s symmetric memory. For each payload tensor, the recv buffer on each rank has shape [ep_size, max_tokens_per_rank, ...]. The slice [src_rank, :, :] stores data received from src_rank, so different source ranks never race when writing to the same peer.

Each CTA handles one token. For each of the top_k routed experts, the kernel computes:

• target_rank — the rank owning that expert, derived from token_selected_experts;
• target_index — the slot index within the src_rank slice of the destination’s recv buffer.

To obtain a unique target_index, the kernel maintains atomic counters send_counters[ep_size] that track how many tokens have been sent from the local rank to each target rank — an atomic increment yields the next available slot. The slot assignment is local (decided by the sender’s atomic state) and does not require knowing how tokens on other ranks are routed.

If multiple of a token’s top_k experts map to the same rank, only the first gets a valid entry; duplicates are assigned (target_rank, target_index) = (-1, -1) and the kernel skips transmission for those entries. For each payload, the kernel loads the token data from local memory once and stores it to up to top_k peers’ symmetric memory — one per unique target rank.

Combine Kernel

The combine kernel pulls MoE outputs from peers’ symmetric memory in the reverse direction. Each CTA handles one (reduced) token. The combine kernel reuses the (target_rank, target_index) pairs recorded during dispatch to read from the exact same slots — no routing computation is repeated. For each payload element, it loads from up to top_k peers’ symmetric memory into registers, performs the reduction, and stores the result to local memory once. The reduction uses a pairwise tree pattern rather than sequential accumulation, keeping the dependency chain shallow.

The MoE module writes its output directly into symmetric memory, so that the combine kernel can read peer ranks’ MoE output without an intermediate copy. This zero-copy path avoids a staging copy before the combine kernel starts; it requires small modifications to the MoE ops to accept a provided output buffer instead of allocating their own in PyTorch.

The diagram below illustrates the complete dispatch → MoE → combine flow with ep_size = 2, top_k = 2, and batch_size = 3. Dispatch computes target_ranks and target_indices to push tokens into the correct slots in the recv buffer; combine reuses the same routing to pull results back. Dashed arrows indicate deduplicated entries in the dispatch phase.

Synchronization

One-sided communication requires two producer-consumer barriers, scoped to the operations they belong to:

• Tail of dispatch. Before the MoE module can consume the dispatched data, all source ranks must have finished writing to symmetric memory.
• Head of combine. Before pulling MoE outputs from peer symmetric memory, every rank’s MoE module must have finished writing its output.

Both barriers share a common write+poll structure built on flag-based synchronization. Each rank holds an array of ep_size flag slots in its symmetric memory and a monotonically increasing epoch counter. To enter a barrier:

1. Each rank writes its current epoch into every peer’s flag slot for itself.
2. Each rank polls its own local flag slots until all ep_size peers have written the current epoch.

The two barriers differ only in their memory-fence semantics:

• The dispatch barrier issues a release membar BEFORE writing the flags, ensuring that all preceding peer-bound stores (token data written into peer symmetric memory) are globally visible by the time the flag store lands.
• The combine barrier issues an acquire membar AFTER the polling loop completes, ensuring that subsequent loads (reading MoE output from peer symmetric memory) observe the data written before each peer’s release membar.

Together the release/acquire pair gives the producer-consumer ordering required between dispatch writes and MoE reads (and, symmetrically, between MoE writes and combine reads).

Saturating NVLink Bandwidth

The key to high throughput is keeping bytes in flight across NVLink at all times. We apply both data-level parallelism (DLP) and instruction-level parallelism (ILP):

• Vectorized load/store. The kernel uses the widest vector type (up to 128-bit) allowed by the payload data’s alignment.
• Dispatch: Load from local memory once, store to remote memory top_k times. The top_k loop is unrolled via template instantiation.
• Combine: load from top_k peers’ symmetric memory (unrolled, kept in registers), reduce pairwise in a tree fashion rather than sequentially, and store to local memory once.

Performance Benchmark

We benchmark the dispatch+combine communication kernels on GB200 NVL72 using the DeepSeek-V3 model profile: hidden_size=7168, top_k=8, 256 total experts.

Methodology

For each (ep_size, batch_size) configuration we report the dispatch and combine kernel latency (µs) and achieved NVLink bandwidth (GB/s). The achieved bandwidth is computed as:

1

bandwidth = batch_size × min(ep_size, top_k) × bytes_per_token / latency

where bytes_per_token covers the activation payload plus its scaling factors when present. We neglect the smaller payloads (expert IDs and router weights) since they are an order of magnitude smaller than the hidden states. The min(ep_size, top_k) factor reflects the deduplication described in the Rank-Major Buffer Layout section. Following the convention used by other MoE communication libraries (e.g. DeepEP), the reported bandwidth is logical — it includes the local-rank fraction of the traffic that does not actually traverse NVLink.

Scaling With Batch Size and EP Size

We sweep ep_size ∈ {8, 16, 32, 64} with BF16 dispatch and combine:

ep_size=8:

ep_size=16:

ep_size=32:

ep_size=64:

The uni-directional NVLink Bandwidth of GB200 NVL72 is 900GB/s. We plot the achieved bandwidth below:

Key observations:

• Given sufficient batch size, both dispatch and combine reach ~80% of the peak NVLink bandwidth at ep_size=8. Dispatch lands slightly higher than combine at large batches despite carrying the routing/slot-index work, because combine does a top_k-way reduction on top of the same payload.
• Bandwidth slightly degrades as EP grows, because of the increased synchronization overhead inside the larger communication domain.

Post-Quant Dispatch

We test the perf benefit of post-quant dispatch, with FP8 and FP4 respectively:

At bsz = 2048:

Key observations:

• Quantization mainly helps at large batch size, where the communication is bandwidth bound.
• Dispatch speedup tracks the byte-ratio asymptote — quantization translates almost directly into faster dispatch.

Reproduction

The benchmark is available in the TensorRT LLM repository. Example command:

1

python tests/microbenchmarks/bench_moe_comm.py --backend NVLINK_ONE_SIDED --profile deepseek_v3 --perfect_router --kernel_breakdown --iter_stats --ep_size 8 -b 1 -e 2048 -f 2 --output_file nvlink_one_sided.json

srun is required for multi-node NVLink benchmarking. See python tests/microbenchmarks/bench_moe_comm.py --help for the full set of options.

Conclusion

NVLinkOneSided AlltoAll is a symmetric memory-based MoE communication kernel that combines a simple one-sided design with minimal overhead and a modular, quantization-agnostic interface. It is the default communication strategy within a single NVLink domain in TensorRT LLM and delivers the best performance among known MoE communication implementations for inference. NVLinkOneSided is also available in FlashInfer.