Multiverse Computing Models LLM Block Removal as an Ising Glass Optimization Problem

Multiverse Computing Models LLM Block Removal as an Ising Glass Optimization Problem Multiverse Computing has introduced a depth-pruning method that frames transformer block removal as a constrained binary optimization problem mapped onto an Ising glass, accounting for inter-block interactions that traditional pruning heuristics ignore. A second-order Taylor expansion of model loss yields a Hessian where off-diagonal entries represent pairwise couplings between block removal decisions. Low-energy states of the spin system act as a proxy for benchmark performance, solvable through GPU search, Tabu search, or quantum-inspired algorithms. At 50% block removal on Llama-3.3-70B-Instruct without retraining, the method scores 76.9 on MMLU, beating the leading baseline by nearly 23 percentage points. The formulation transfers directly to heterogeneous architectures like NVIDIA-Nemotron-3-Nano-30B-A3B-FP8, improving upon baselines on AIME25 and GPQA. Formulating Block Removal via Many-Body Physics Traditional depth-pruning techniques evaluate transformer blocks independently or prune contiguous sequences, treating block impacts as isolated mean-field effects. Because the effect of deleting any block depends on which other blocks are simultaneously removed, Multiverse Computing assigns a binary variable to each block and computes a second-order Taylor expansion of model loss. The resulting Hessian diagonal reflects individual block importance, while off-diagonal terms capture pairwise couplings. This maps block selection to finding low-energy states in an Ising glass with conserved magnetization. The Hessian is computed once using forward and backward passes on a small calibration dataset, allowing candidate configurations to be evaluated cheaply across various compression targets without running the model. Solving the Energy Landscape and Leveraging Excited States For moderate configuration spaces, the energy calculation can be evaluated directly on a single GPU, such as checking roughly 29 billion configurations for removing 8 of 80 blocks in Llama-3.3-70B over about two days. For larger spaces, the problem converts into quadratic unconstrained binary optimization (QUBO) solved via quantum-inspired methods and classical tools like an open-source Tabu solver, which reliably reaches low-energy states in seconds. Because the energy function is a proxy rather than an exact performance score, the lowest-energy ground state is not always the best model. Extracting the spectrum of low-lying excited states provides alternative prunings; for example, the 17th excited state of Llama-3.1-8B-Instruct with 16 blocks removed targeted an early block and outperformed the ground state across several benchmarks after light retraining. Empirical Gains in Deep Compression The advantage of accounting for pairwise couplings increases as compression deepens. Evaluated without retraining on Llama-3.3-70B-Instruct, the method matches block influence baselines up to 24 removed blocks, but diverges significantly under deeper cuts. When removing 32 of 80 blocks, it achieves an MMLU score of 76.6 compared to 59.3 for the baseline. At 40 of 80 blocks removed (50% depth compression), it scores 76.9 against 54.0 for the baseline, representing a nearly 23 percentage point lead while the unpruned model scores 82.2. On Qwen3-14B with 12 of 40 blocks removed, the method leads the baseline on MMLU by approximately 10 points. Extension to Heterogeneous Architectures The mathematical formulation does not assume identical block types, enabling it to prune non-uniform models without retraining. Tested on NVIDIA-Nemotron-3-Nano-30B-A3B-FP8—a hybrid model interleaving Mamba2, attention, and mixture-of-experts (MoE) layers—the approach successfully removed two to three MoE layers or two attention layers, outperforming block influence heuristics on both AIME25 and GPQA benchmarks. These tests confirmed that redundancy is unevenly distributed across hybrid architectures and that top-performing configurations in these models also frequently correspond to excited states rather than the ground state.

将大模型层级剪枝重构为物理学伊辛自旋玻璃优化问题 Multiverse Computing 提出一种通过受约束二进制优化进行大语言模型网络块剪枝（Block Removal）的新方法。该方案通过二阶泰勒展开捕获层与层之间的配对耦合相互作用，将其映射为全连接的伊辛自旋玻璃能量最小化问题，在无需高昂评测成本的前提下显著提升了深层剪枝后的模型表现。 传统剪枝方法将各层视为独立个体（类似物理学中的平均场理论），而新方法通过损失函数的 Hessian 矩阵显式保留层间配对耦合，将块选择转化为约束二进制优化（CBO）问题。 系统只需在前向与后向传递中小规模校准集上计算一次 Hessian 矩阵，便可重复用于多种剪枝目标；候选配置可通过单 GPU 暴力枚举或开源禁忌搜索（Tabu）等经典/量子启发式求解器在数秒至数天内完成评估。 能量最低的基态并非总是最优模型，低能激发态同样提供了高价值候选方案；实验发现 Llama-3.1-8B 的第 17 个激发态经轻度微调后综合性能超越基态，打破了以往仅能连续剔除中后层块的惯例。 在未重新训练的 Llama-3.3-70B-Instruct 深度压缩测试中，剪去 50% 层数时该方法 MMLU 得分达到 76.9，较现有最优基线领先近 23 个百分点，并成功扩展至混合 Mamba2、注意力与 MoE 的异构模型。 理论重构：从平均场方法到多体耦合 现有的网络块剪枝方案多基于单块敏感度或影响力的启发式评分，属于平均场近似，即假定每层的贡献独立于其他层；或仅局限于删除单段连续网络块。然而模型各层间存在耦合交互效应，删除某一层的影响取决于是否同时移除了其他层。研究人员为每个 Transformer 块赋予二元变量（0 代表保留，1 代表删除），通过对模型损失进行二阶泰勒展开构建 Hessian 矩阵。对角线反映单块影响，非对角线则直接对应块之间的相互耦合，从而将层选择形式化为一个具有固定磁化强度的伊辛自旋玻璃能量最小化问题。系统只需在小型校准集上计算一次 Hessian 耦合项，即可复用并快速计算任意组合的能量代理指标。 求解机制与低能激发态探索 单次能量评估的计算开销极低，这使得在单张 GPU 上穷举数十亿种自旋配置成为可能（例如评估剪掉 Llama-3.3-70B 80 层中 8 层的约 290 亿种配置耗时约两天）。对于更大维度的搜索空间，该问题可转换为二次无约束二进制优化（QUBO）形式，输入到禁忌搜索等经典或量子启发式求解器中并在数秒内收敛。与常规优化追求真实基态不同，该方法关注整个低能谱系。在 Llama-3.1-8B-Instruct 剪除 16/32 层的测试中，前序低能态普遍移除靠后的层，但第 17 个激发态首次提议移除靠近模型前端的层，经轻度重训练后其在多项基准上的表现全面超越基态，表明低能激发态能有效打破局部最优经验偏见。 深度压缩评测与基准表现 在 Llama-3.1-8B、Qwen3-14B 和 Llama-3.3-70B-Instruct 的测试中，CBO 表现均优于或看齐现有的块移除基线，且剪枝比例越深优势越明显。在未做任何重训练的 Llama-3.3-70B-Instruct 评估中，移除不超过 24 层时该方法与块影响力基线相当；当剪除层数增至 32 层和 40 层时，CBO 优势大幅拉开。在 50% 深度压缩（移除 40/80 层）的极端场景下，CBO 将 MMLU 维持在 76.9 分，而传统块影响力基准骤降至 54.0 分，差距接近 23 个百分点。此外在 Qwen3-14B 剪去 12/40 层时，CBO 亦在 MMLU 上取得了约 10 个百分点的领先。 在异构模型上的泛化与技术组合 该伊辛表述不依赖同质网络堆叠假设，因而可直接泛化到现代异构架构。研究团队将其应用于交替包含 Mamba2、注意力层与混合专家（MoE）层的 NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 模型中。在未微调的情况下，无论移除 2 至 3 个 MoE 层还是 2 个注意力层，CBO 选出的配置在 AIME25 和 GPQA 基准上均超过块影响力基线，验证了异构模型中专家层冗余度高度非均质的特性。该层级剪枝算法能与量化、低秩分解、宽度剪枝及基于知识蒸馏的修复技术无缝堆叠，融入更广泛的大模型压缩流程。

One of the cheapest ways to make a large language model faster is also one of the bluntest: delete whole transformer blocks. Because the model literally gets shorter, block removal (also called depth pruning) buys predictable inference speedups on top of the memory savings, and it stacks cleanly with quantization, low-rank compression, and other techniques. The hard part is deciding which blocks to cut. Remove the wrong ones and the model collapses; and the effect of removing any one block depends on which others you remove alongside it, so the choices interact. That makes it a combinatorial problem, not a ranking problem, and combinatorial problems with interacting binary variables are exactly what the physics of spin systems was built to describe. Our latest paper, LLM Compression by Block Removal with Constrained Binary Optimization , takes that correspondence literally. We reformulate block selection as a constrained binary optimization (CBO) problem that maps directly onto an Ising glass , a disordered spin system with all-to-all interactions and a fixed number of "up" spins. The energy of that spin system turns out to be a strong, cheap proxy for how well the pruned model will actually score on benchmarks, which means we can rank a huge number of candidate configurations without benchmarking any of them, and hand the hard instances to the same classical and quantum-inspired solvers we use elsewhere at Multiverse. The payoff in the deep-compression regime is large: at 50% compression of Llama-3.3-70B-Instruct, we gain almost 23 percentage points on MMLU over the best competing block-removal method. Why picking blocks is a many-body problem Most existing block-removal methods score each block on its own, then remove the ones that look least important, using magnitude, sensitivity, or "block influence" heuristics. In physics terms these are mean-field methods: they treat each block as if its contribution were independent of the others, the way mean-field theory replaces a spin's neighbors with a single averaged field. A related shortcut is to only ever remove a single consecutive run of blocks, which keeps the problem small but throws away most of the search space. The trouble is that blocks are not independent, any more than spins in a real magnet are. Whether removing block 20 hurts the model depends on whether you also removed block 19 or block 24, an interaction, or coupling , between the two decisions. As models get deeper and more heterogeneous, ignoring those couplings leaves quality on the table, especially when you want to remove a lot of blocks at once. What you really want is to search over combinations of blocks while accounting for how they interact, but the number of combinations grows exponentially, so brute force looks hopeless. This is precisely the regime, exponentially large configuration spaces with pairwise couplings, where the tools of statistical physics earn their keep. The idea: turn block selection into an energy-minimization problem We attach a binary variable to each transformer block: 0 means keep it, 1 means remove it, just like a spin that can point down or up. Then we do a second-order Taylor expansion of the model's loss with respect to those variables, which produces an (approximate) Hessian matrix. The diagonal of that Hessian is how much each block matters on its own; the off-diagonal entries are exactly the pairwise couplings between blocks, the many-body physics that mean-field methods throw away. That reformulation turns "which blocks should I remove?" into a clean optimization: find the set of M blocks whose removal minimizes the energy xᵀH⁰x , subject to removing exactly M of the N blocks. Mathematically this is a constrained binary optimization problem; physically it is an Ising glass, an all-to-all coupled spin system with conserved magnetization (the fixed number of removed blocks plays the role of a fixed total spin). The key property we establish is that this energy is a strong proxy for downstream quality: low-energy states of the spin system correspond to high-performing pruned models. Minimizing energy and maximizing benchmark score become the same search. Block selection becomes a constrained binary optimization problem, equivalent to finding low-energy states of an Ising glass; each solution says which M of N blocks to delete. Right: the coupling variable α we insert into each block's residual path to build the Hessian. Source: paper Figure 1. The reason this is practical is cost. The Hessian, i.e. the full set of couplings, is computed just once, from forward and backward passes on a small calibration dataset. After that, evaluating any candidate configuration is a single cheap energy calculation, no need to run the actual model, let alone benchmark it. And because the couplings don't depend on the compression target, the same Hessian can be reused to solve for many different values of M . Solving it: exact when you can, quantum or quantum-inspired when you can't For most models the configuration space is large but still checkable. Because computing one energy is so cheap, we brute-force it on a single GPU, checking up to tens of billions of spin configurations. A few million take seconds; the hardest tractable case here, removing 8 of Llama-3.3-70B's 80 blocks (about 29 billion configurations), took roughly two days. Beyond that the exact approach breaks down, and this is where casting the problem as an Ising glass pays off a second time. In its equivalent QUBO form (the constraint absorbed into a penalty term), the exact same task can be handed to the highly optimized classical, quantum, and quantum-inspired solvers built for this class of Hamiltonian, the machinery of quantum annealing, QAOA, tabu search, and specialized branch-and-bound. We find that an open-source tabu solver reliably reaches the lowest-energy states in seconds , even on the hardest cases we can verify against brute force. So the method scales to models where enumerating configurations is out of the question, using solvers that are squarely in Multiverse's domain. There's a subtle but important point here, and it runs against the usual grain of optimization. Normally a CBO or annealing solver is judged by whether it finds the true ground state. We don't actually need the ground state. What we need is a fast way to generate a handful of good low-energy states, and that is a far easier bar, which is why lightweight solvers work so well for us and why we can afford to run several of them. Why the whole low-energy spectrum matters The energy is a strong proxy for quality, but not a perfect one, so the single lowest-energy state isn't always the best model. This turns out to be a feature, not a bug: once the Hamiltonian is set up, reading off the ground state and the low-lying excited states is essentially free, giving a spectrum of high-quality candidate prunings to try rather than one fragile answer. Exploring excited states, not just the ground state, is itself an area of active physics research, and it maps neatly onto what practitioners actually need here. A concrete example: for Llama-3.1-8B-Instruct at 16/32 blocks removed, most of the top states cut blocks toward the end of the model, as prior work would expect. But the 17th excited state is the first to propose removing a block near the beginning of the model, and after light retraining that configuration outperforms the ground state across several benchmarks. That directly disproves the common assumption that the best pruning is one consecutive chunk of middle-or-late blocks, and it shows why respecting the full many-body structure of the problem pays off. Left: which blocks each of the 20 lowest-energy states removes (red = removed). Right: the 17th excited state, which removes an early block, beats the ground state on several benchmarks after retraining. The best model is an excited state, not the ground state. Source: paper Figure 2. Results Across Llama-3.1-8B-Instruct, Qwen3-14B, and Llama-3.3-70B-Instruct, our method (CBO) is on par with or better than state-of-the-art block-removal baselines, and the gap widens as compression gets more aggressive. The clearest win is deep compression of Llama-3.3-70B-Instruct, evaluated without retraining. Up to 24 of 80 blocks removed, CBO is roughly on par with block influence. But at 32/80 and 40/80, it pulls decisively ahead, with an almost 23-point MMLU advantage at the deepest setting, where it beats the baseline on every benchmark we tested. For Qwen3-14B at 12/40 removed, CBO leads MMLU by about 10 points. At lighter compression the methods are comparable, which is expected: the couplings matter most when you're cutting deep. Llama-3.3-70B-Instruct, no retraining Blocks removed MMLU Original 0 82.2 CBO (ours) 32 / 80 76.6 Block influence 32 / 80 59.3 CBO (ours) 40 / 80 76.9 Block influence 40 / 80 54.0 At 40/80 (50% depth), CBO holds MMLU near 77 while the strongest baseline falls to the mid-50s. Source: paper Table 2. It generalizes beyond dense transformers Block removal gets much harder on modern heterogeneous architectures, where different block types are interleaved, and the Ising formulation doesn't care: a coupling is a coupling regardless of what kind of block sits at each site. To stress-test that, we applied the method to NVIDIA-Nemotron-3-Nano-30B-A3B-FP8, a hybrid model that interleaves Mamba2, attention, and mixture-of-experts (MoE) layers in a non-uniform pattern, without any retraining. Nothing about our formulation assumes a homogeneous stack, so it transfers directly. Removing 2–3 MoE layers or 2 attention layers, CBO finds configurations that beat block influence on AIME25 and GPQA. The results also confirm that redundancy in these hybrid models is real but unevenly distributed: some expert layers are far more disposable than others, and the method's ability to search the coupled configuration space is what locates the good cuts. Even here, the pattern from the dense models holds, the best configuration is often an excited state rather than the ground state. Why this fits Multiverse Computing Reframing a messy machine-learning problem as an Ising Hamiltonian, then solving it with the classical and quantum-inspired optimization machinery built for physics, is squarely in Multiverse's wheelhouse, it's the same instinct that runs through our compression stack. And block removal composes with the rest of that stack, quantization, low-rank/SVD compression, width pruning, and knowledge-distillation-based healing, so it slots into a larger pipeline rather than competing with it. Want the full technical details, including the Taylor-expansion derivation, the QUBO mapping, the solver benchmarks, the calibration-dataset ablations, and the complete results tables? Read the full paper on Hugging Face , or get in touch with our team to talk about applying this to your own models. The code is open-sourced at github.com/CompactifAI/Block_removal_through_constrained_binary_optimization .