AllReduce Speedup With In-Network Compute: How INC Accelerates AI Collective Operations

At a glance:

• INC reduces the transfer time component of latency for collective operations.
• The tensor parallelism that drives modern inference means INC is most impactful at small message sizes.
• AllReduce is an increasing percentage of inference time as cluster sizes grow.

The inflection point facing AI today as inference workloads overtake training workloads in volume¹ has created an industry-wide need for the execution of collective operations within network switches, or In-Network Compute (INC). Today’s inference workloads utilize massive models distributed across scale-up fabrics comprising numerous accelerators, and as a result rely heavily on operations like AllReduce to orchestrate the matrix multiplications that make them possible across this fabric.

AllReduce is conventionally implemented in a “Ring” algorithm which occurs over two sequential steps:

1. ReduceScatter: Aggregation of data is performed across all involved accelerators. A single “slice” of total data is reduced and stored in each GPU.
2. AllGather: Local copies of the scattered slices from the previous step are distributed back to each accelerator in the cluster.

The resulting distribution of data means that every accelerator in the cluster has an identical copy of the final result, which drives the generation of a token.

Understanding exactly how INC speeds up AllReduce operations provides crucial insight into its potential impact across the AI industry. INC is best suited for the precise kinds and sizes of data transfers that are driving the inference explosion we see today.

How Does In-Network Compute Accelerate Inference by Cutting AllReduce Latency?

GPU- to-GPU communication latency has two components:²

• Fixed overhead time (Tf): The cost of coordination, packetization, and load and store, which is independent of message size
• Transfer time (Tt): the cost of moving data, which is proportional to message size

This means total AllReduce latency is dependent on message size plus a fixed amount. Understanding where INC delivers its highest return (and where it does not) is the difference between a well-architected INC deployment and a disappointing one – and it all depends on how an INC-enabled fabric switch reduces the time cost of moving data.

Ring AllReduce without INC

In a standard ring AllReduce with N participants, the collective operation is typically carried out in two phases, visualized below:

Figure 1: AllReduce Without INC

Each phase requires N – 1 communication steps, as shown in the two diagrams above. So the total number of steps is:

At each step, two things happen:

• A fixed startup or coordination cost Tf is paid
• Only 1/N of the total data is transferred, contributing Tt/N

That means the time per step is:

Multiplying by the total number of steps gives the overall ring AllReduce time without INC:

This expression captures an important property of ring AllReduce: Even though each step only moves a fraction of the data, the algorithm must repeat the communication pattern many times. As a result, the fixed overhead is incurred over and over again throughout the AllReduce steps – but as we will see, this changes with In-Network Compute.

AllReduce with INC

With an INC-enabled switch like the Scorpio™ X-Series 320 Lane Smart Fabric Switch, each GPU sends its local copy of the data slice to the switch, the slice is reduced and distributed, while the next slice is already being streamed to the switch.

Figure 2: AllReduce with INC

INC effectively collapses the many-step ring behavior into a much more direct operation. Instead of paying the fixed-overhead term (Tf) at every stage of the ring, the system pays it once. Likewise, instead of decomposing the communication into many small, serialized transfers of size 1/N, the operation behaves more like a single collective transfer with total data-movement cost Tt.

In other words, INC removes the repeated per-step overhead that dominates ring execution, and the time saved by doing so scales as the number of participants grows.

Figure 2 illustrates the AllReduce data flow with INC, showing how upstream and downstream traffic can be overlapped. For simplicity, each colored message represents the amount of data that can be transmitted upstream at line rate during half of a round-trip time. As the GPUs stream one data chunk upstream to the switch, the switch computes the AllReduce result for the preceding chunk and sends that result downstream. Because transmitting one data chunk does not depend on the completion of the previous chunk, the entire AllReduce message can be streamed continuously. This pipelined process allows computation and bidirectional data movement to proceed concurrently, improving link utilization and reducing overall communication time.

With INC, the cost model becomes much simpler (assuming the value of Tf and Tt are the same both with and without INC):

What’s the Actual Speedup?

Speedup is simply the ratio of the old execution time to the new execution time:

Substituting the two expressions gives:

We estimate INC speedup using the collective latency models derived above. The system parameters were selected based on our understanding of representative systems and observed behavior. Because these parameters can vary across system architectures, the results should be interpreted as indicative trends rather than absolute performance values. Accordingly, Figures 3 and 4 focus on how INC speedup changes with cluster size and message size, rather than predicting a specific speedup for any particular system.

Figure 3: Roofline INC Speedup for typical decode message sizes

As we see above, the largest potential benefit of INC is for AllReduce with small message sizes, with speedup of 2x possible with 8-GPU clusters and scaling almost linearly with the cluster size. This is the region of operation for the decode phase of inference workloads.

Figure 4 zooms in on INC speedup for these small AllReduce messages, 1 MB and below, where the potential speedup is substantial. Small messages also bring secondary benefits to the fabric: They enable finer-grained load balancing and improve tail latency. Taken together, these effects give model and system architects strong reasons to design toward smaller message sizes — potentially a new paradigm in AI model development.

Figure 4. Roofline INC Speedup for small message sizes

Why Is INC Best Suited for Inference Today? Tensor Parallelism and AllReduce Message Sizes

Modern LLM inference uses Expert Parallelism (EP) to split model layers across multiple GPUs within a single scale-up domain. Each transformer layer has two sub-blocks, each requiring an AllReduce:

• AllReduce #1: Multi-Head Attention (Attn), in which each GPU holds 1/TP of attention weights and computes a partial output
• AllReduce #2: Feed-Forward Network (FFN), in which the FFN applies two linear transformations with a non-linear activation (FFN(x) = activation(x × W1) × W2), where W1 expands hidden dimensions to FFN dimensions and W2 compresses back; each GPU holds 1/TP of FFN weights and computes a partial output

This means that there are 2xL+1 AllReduces per generated token for an L-layer model. This is demonstrated in Figure 5.

Figure 5: AllReduce in Decode phase of Qwen3.5 35B, TP = 2, Batch = 1

Below, you can see the number of layers in some popular open-source LLMs as well as the number of AllReduce calls per token for those models.

Table 1: Number of AllReduce calls in popular open-source inference applications

As we saw earlier, the size of the AllReduce messages plays a critical role in the INC speedup. The size of AllReduce messages in tensor-parallel inference linearly grows with the aggregation of partial hidden-dimension activations and batch sizes:

The example in Figure 5 shows 4 KB AllReduce message sizes for Qwen3.5-35B with Batch Size =1, and the table below shows the size of the hidden dimension for different models, which confirms the decode message sizes are in the several-KB range.

Table 2: Decode AllReduce message sizes grow linearly with hidden dimension and batch sizes

These decode messages are exactly the kind of latency-bound small messages for which switch INC provides maximum benefit. The inference that now represents the majority of AI workloads¹ thus maps perfectly onto the message size regime where INC delivers its highest ROI.

Contrast this with data-parallel training gradient synchronization: AllReduce messages of hundreds of megabytes to gigabytes that are deeply bandwidth-bound. Ring AllReduce is bandwidth-optimal by design at these sizes; the latency advantage growth provided by switch INC, while non-negligible, is lower in this case than for smaller message sizes.

AllReduce Is a Significant Percentage of Inference Time…and Growing

The inference workload study and measurements done independently by Jian et al.³ and Singhania et al.⁴ show that AllReduce communication accounts for 47% and 25% of end-to-end latency in prefill and decode phases respectively in 8-GPU clusters. These percentages increase with the number of accelerators in a cluster, further extending the communication time as the systems scales to larger nodes.

To make matters worse, the growth in AllReduce operations is outpacing bandwidth expansion in each successive hardware generation: According to Jian et al, “Peak server FLOPS have scaled at approximately 3.0× every two years, while interconnect bandwidth has grown by only about 1.4× every two years.”³

The call to action is thus clear for the AI industry: Adoption of In-Network Compute across inference scale-up fabrics will be a vital tool for meeting the demand for more, more complex, and faster inference workloads. Read our companion blog for more details about the industry’s readiness for INC and what steps need to be taken before it can be deployed at scale.

References:

1. McKinsey & Company. The Future of AI Workloads.” Analysis of global data center demand by workload type through 2030. [Online] 2026. https://www.mckinsey.com/featured-insights/week-in-charts/the-future-of-ai-workloads.

2. Thakur, R., et al. Optimization of Collective Communication Operations in MPICH. International Journal of High Performance Computing Applications, vol. 19, no. 1, 2005, pp. 49–66.

3. Du, Aojie Jiang and Kang Zhu and Zhiheng Zhang and Zhengxu Su and Juntao Liu and Yuan Du and Li. A Switch-Centric In-Network Architecture for Accelerating LLM Inference in Shared-Memory Network (SCIN). arXiv. [Online] 2026. https://arxiv.org/abs/2603.28239.

4. Singhania, Prajwal, et al. Understanding and Improving Communication Performance in Multi-node LLM Inference. Proceedings of the ACM Conference on AI and Agentic Systems (CAIS ’26). Association for Computing Machinery, New York, NY, USA, 2026, pp. 1070–1083. https://doi.org/10.1145/3786335.3813165.