光合及目
1lux.xyz
← 报道
行业报道x2robot· 2026-07-15

X Square Official Site X Square Robot WALL-A Large Operating Model Robot

X Square Robot has released DMuon, an open-source distributed implementation of the Muon optimizer that brings its per-step cost close to AdamW while preserving Muon's convergence

X Square Robot has released DMuon, an open-source distributed implementation of the Muon optimizer that brings its per-step cost close to AdamW while preserving Muon's convergence behavior.

Optimizers are a foundational component of large-model training, determining how model weights are updated at each step. In recent years, Muon has emerged as a promising alternative to AdamW. Rather than updating parameters element by element, Muon applies Newton-Schulz iterations to momentum-smoothed gradient matrices, producing matrix-orthogonalized updates that can improve training convergence.

However, Muon's algorithmic benefits are difficult to realize in distributed environments. Conventional distributed implementations introduce substantial computation and communication overhead, potentially offsetting the time saved through faster convergence.

DMuon addresses this systems bottleneck. Compared with a vanilla gather-then-compute implementation of distributed Muon, it delivers a 1.48-3.01x speedup in end-to-end step time and a 6.85-163x speedup in optimizer-step time. Across the evaluated workloads, its average per-step overhead remains within 2% of AdamW.

Why Muon Needs a Distributed Redesign

Modern distributed training systems divide model parameters, gradients, and optimizer states across multiple GPUs. ZeRO-3-style sharding and PyTorch FSDP2 are designed primarily around element-wise optimizers such as AdamW.

This arrangement works well for AdamW because each GPU can update its local parameter shard without accessing other shards.

Muon breaks this assumption. For each two-dimensional parameter managed by Muon, the Newton-Schulz update requires the complete momentum-smoothed gradient matrix rather than an individual shard.

A straightforward distributed implementation therefore reconstructs the full matrix on every GPU and repeats the same orthogonalization computation across all devices. This creates additional communication and large amounts of duplicated computation.

The problem is particularly significant in embodied-model training. Vision-language-action models often use shorter temporal contexts than contemporary language-model pretraining workloads. Their forward and backward passes therefore occupy a smaller share of each training step, making optimizer overhead more difficult to amortize.

Three Core Improvements

DMuon reduces this overhead through three coordinated system optimizations.

1. Owner-Centric Execution and Communication Optimization

DMuon assigns each Muon-managed matrix parameter to a unique owner rank. Only the owner maintains the authoritative optimizer state and performs the Newton-Schulz update. Other GPUs materialize the parameter temporarily when it is needed for forward or backward computation.

During backpropagation, gradients are routed to the corresponding owners. Once an owner completes the update, the new parameter values are published asynchronously for subsequent computation.

DMuon also separates inter-node and intra-node communication. Cross-node broadcasts are issued ahead of use, while local broadcasts are delayed until shortly before a layer needs its parameters. During the backward pass, parameter broadcasts and gradient reductions are interleaved across adjacent layers.

This design eliminates replicated orthogonalization while allowing much of the remaining communication to overlap with model computation.

2. A Shape-Adaptive Execution Stack

Neural-network weight matrices vary significantly in size and shape, so a single execution strategy cannot use the GPU efficiently across an entire model.

DMuon adopts a Gram-space formulation of the Newton-Schulz iteration. For an (m \times n) matrix where (m < n), this reduces the dominant computational complexity from (O(m^2n)) to (O(m^3)).

Because the resulting Gram matrix is symmetric, DMuon computes only one triangular half during the main operation and reconstructs the other half afterward. This nearly halves the arithmetic required by the dominant Gram computation.

Small matrices that cannot fully occupy a GPU are grouped by shape and processed in batches. DMuon also uses automatic tuning to benchmark different kernel configurations on the target hardware and cache the fastest configuration for each recurring matrix shape.

3. Computation-Aware Load Balancing

The cost of a Muon update varies substantially across matrix shapes. A simple round-robin assignment can overload some owner ranks while leaving others underutilized, causing the entire optimizer step to wait for the slowest GPU.

During initialization, DMuon measures the actual execution cost of each matrix shape and candidate batch configuration on the target hardware. It then formulates owner assignment as a mixed-integer linear programming problem that minimizes the predicted completion time of the slowest owner.

This measured-cost approach captures the effects of batching, kernel selection, and hardware-specific performance more accurately than estimates based only on parameter counts or theoretical FLOPs.

Near-AdamW Performance at Scale

DMuon was evaluated on four production workloads spanning embodied and language-model training: WALL-OSS, π0, WALL-WM, and Qwen2.5-7B.

The experiments were conducted in bf16 on an A800-SXM4-80GB cluster. Each node contained eight GPUs connected through NVLink, while nodes communicated over 200 Gb/s InfiniBand.

Across the full evaluation, DMuon kept its average end-to-end step-time overhead within 2% of AdamW. At 256 GPUs, its overhead relative to AdamW ranged from 0.2% to 3.3% across the four workloads.

Its optimizer computation also decreased as the number of GPUs increased. By contrast, vanilla distributed Muon continued to perform the same orthogonalization redundantly across ranks, preventing its optimizer time from scaling effectively.

DMuon does not introduce a new optimization algorithm or modify Muon's parameter-update rule. Its techniques are mathematically equivalent reformulations and systems optimizations that preserve exact Muon semantics while removing distributed execution overhead.

Three-Line Integration

DMuon is packaged as a standalone Python module and integrates with existing PyTorch FSDP2 training pipelines through three DMuon-specific lines:

import dmuon

dmuon.dedicate_params(model, mesh)

optimizer = dmuon.Muon(model)

No changes to the model architecture or PyTorch FSDP2 itself are required. DMuon automatically manages parameter ownership, gradient routing, asynchronous parameter publication, load balancing, and owner-only optimizer execution.

The system also supports hierarchical sharding and composition with tensor parallelism, allowing it to operate within more complex distributed training configurations.

Embodied AI Moves Deeper Into the Infrastructure Stack

Much of the recent momentum around Muon has come from the large-language-model community. DMuon extends this work from the perspective of embodied foundation-model training, where optimizer overhead can account for an unusually large share of each step.

By bringing Muon's distributed execution cost close to AdamW, DMuon allows its convergence benefits to translate more directly into wall-clock training gains. It offers teams a practical way to adopt matrix-aware optimization without restructuring their models or distributed training frameworks.

成为付费用户可以阅读 自变量机器人 所有资料

了解更多 →
阅读原文 ↗x2robot

首发未确认;采集时间不代表发布时间。