Study Finds Non-Monotonic Scaling Behavior in Language Model Sequence Weighting

Study Finds Non-Monotonic Scaling Behavior in Language Model Sequence Weighting A study across in-house and open-weight language models examined how sequence weighting during training impacts loss reduction as model scale increases, uncovering a non-monotonic trend. Researchers observed that models transition from learning general patterns independent of data weights at small scales to learning data-specific patterns proportional to weights at medium scales. At large scales, models are able to learn all patterns present in the data, returning to a state where learning is independent of sequence weights.

研究发现语言模型数据加权规律随规模呈非单调变化 Jane Street研究了自研及开源语言模型在不同规模下的数据加权标度律，发现训练中序列损失下降对权重的依赖关系随模型规模呈现非单调变化。 从小规模过渡到中等规模时，模型从学习独立于权重的通用模式，转变为学习与序列权重成比例的数据特定模式。 从中等规模过渡到大规模时，模型具备学习数据中所有模式的能力，对模式的学习再次变得独立于数据权重。

*TL;DR: We study the scaling laws of data weighting across in-house and open-weight LMs, finding non-monotonic behavior across scales. We vary the weight assigned to sequences during training and measure how strongly the model’s loss reduction on a sequence depends on the sequence’s weight. Taken together, our results are consistent with a general trend: as models transition from small to medium scale, they transition from learning general patterns independent of data weight to learning data-specific patterns proportional to the data weights. As models then transition from medium to large scale they are able to learn all patterns present in the data, once again independent of data weight.*