Scaling Language Models with Mixture-of-Experts Architectures

Scaling Language Models with Mixture-of-Experts Architectures Scaling dense transformer models raises training and inference costs because every token passes through every layer, whereas mixture-of-experts architectures reduce computation by activating only a small subset of subnetworks per token. Dense transformer architectures process every token through every layer, making scaling computationally expensive for both training and inference. Mixture-of-experts architectures scale by employing multiple subnetworks, or experts, and activating only a small subset for each token.

语言模型扩展中的稠密架构与 MoE 架构对比 随着语言模型规模扩大，稠密架构的扩展成本高昂，而混合专家（MoE）架构通过对每个 token 仅激活少量专家子网络，提供了不同的扩展路径。 在稠密 Transformer 中，每个 token 都会经过所有网络层，导致增加模型能力的同时增加训练与推理的计算开销。 混合专家（MoE）架构采用多个子网络（专家），每个 token 仅激活其中一小部分来参与计算。

As language models grow, scaling dense architectures becomes increasingly expensive. In a dense transformer, every token passes through every layer, so adding capabilities increases computation for both training and inference. Mixture-of-experts (MoE) architectures take a different approach to scaling by using many subnetworks, or experts, while activating only a small subset for each token.