AINEWS
A new Amazon AGI paper says the field's FLOP based budgeting rule can pick a design that costs several fold more GPU hours to train than the math predicted, sharpest for sparse mixture of experts (MoE) models.
AI labs pick how big a model to train by solving a math problem denominated in FLOPs. The cluster that actually trains the model is billed in GPU-hours. A new paper from Amazon AGI Foundations shows the gap is large enough to change the answer.
Two architectures with the same predicted loss can cost several-fold different amounts of GPU-hours to actually train. That gap is what makes the field's standard recipe fragile, and it is the load-bearing comparison in a paper posted to arXiv on 11 August 2026. The authors also published a companion blog post explaining the framework for a technical audience. A Hacker News thread is the only public discussion so far.
The standard convention has a specific failure mode. It splits the work into two stages. A scaling-law stage picks an architecture and a token budget using a formula denominated in FLOPs, the standard measure of compute work. A systems stage then tunes the implementation around that architecture. The two stages do not share a cost model. The scaling law treats "model FLOP utilization" (MFU), the share of theoretical hardware throughput the design actually achieves, as a constant. MFU is not a constant. It is a property of the architecture under its best feasible parallel layout, and that property is invisible to the formula that picks the architecture.
The paper's response is a framework called MOSAIC. MOSAIC folds the systems stage into the scaling-law stage. One optimization picks architecture, token budget, and the distributed execution layout together, for a specific cluster and a specific training window. The objective is no longer predicted loss per FLOP. It is the cheapest design that hits a target loss on a given cluster, denominated in the GPU-hours that cluster actually bills.
The break is sharpest for sparse mixture-of-experts, or MoE, designs. In a sparse MoE model, only a small fraction of the model's total parameters activates for any single token, so the per-token compute is much lower than the total parameter count would suggest. That is exactly the design knob a systems-aware scaling law should price, because the choice of expert count, top-k routing, and parallelism strategy can move MFU by a wide margin. The paper finds that "optimal" sparsity is not a property of the math. It is a property of the cluster. Two clusters with the same predicted-loss budget can pick two different MoE designs, and the one the FLOP-based rule would have picked can cost several times the GPU-hours of the cluster-aware one.
The Kimi K2 training report, cited by the authors as a case of the standard two-stage workflow, describes sparse-MoE scaling laws and systems tuning as separate stages. MOSAIC's claim is that the join between those stages is where the cost is set, and that the join is not optional.
The benchmarks in the paper are self-reported on one Amazon fleet. The model-and-cluster coupling is structural, but the absolute size of the cost gap will move with the cluster. A lab running on a different topology, with different interconnect bandwidth, different failure rates, and different goodput (the share of wall-clock time a job is actually making progress rather than checkpointing, failing, or waiting on input), will not see the same number. The framework generalizes. The figure does not.
The first major sparse-MoE training report that prices its design in GPU-hours rather than FLOPs will show whether the field has started to fold the systems stage into the scaling-law stage. The convention is portable. The bill is not.