physical-misgeneralization
🚨New paper! 📃Mechanisms of Misgeneralization in Physical Sequence Modeling
Planners for the physical world produce motions that look safe, but quietly change quantities the demonstrations are meant to control.
When does this happen? Why? Can we predict it before training?👇🧵
Imagine training a navigation model on demonstrations chosen so path lengths are uniformly spread over a safe range.
After training, it reaches the goal. But its generated paths are too long too often. The individual paths look fine; the aggregate physical behavior is wrong.
We call this physical misgeneralization. The issue is broader than maze navigation: the same kind of mismatch appears for travel distance, mechanical energy, and parameters of dynamical systems. We observe it across diffusion models, Transformers, VAEs, and MLPs.
Why does this happen? We design synthetic tasks to investigate.
The answer: the model is trained to imitate trajectories, not to preserve the physical quantity distribution. Thus, small errors in trajectory space can turn into large shifts of the quantities we care to measure.
Our main result is that this shift is predictable. We build a data deviation kernel: a mathematical proxy for the local errors a model class tends to make. Push those errors through the physical measurement, and we can forecast which values will be over- or under-produced.
This prediction is surprisingly detailed. Across smooth, chaotic, and applied physical systems, the kernel predicts not only the direction of drift, but also the peaks and troughs of the shifted distribution. The model’s aggregate physical error is not arbitrary.
This also tells us why obvious fixes fail. Reweighting the dataset does not stop local errors from moving probability mass after generation. Conditioning on the quantity helps in simple cases, but gets tricky when distance, energy, endpoints, and dynamics must all be compatible.
Our best mitigation changes the coordinate system the model sees. The model remains unconditional; it is not handed the physical quantity at inference time. Instead, we use the kernel to make nearby points more balanced with respect to the recovered quantity, reducing the drift.
Takeaway: for physical generative models, “the samples look valid” is not enough. If a model will control moving systems, we should also test whether it preserves aggregate physical quantities. Overall, physical misgeneralization is predictable, measurable, and reducible.
Huge thanks to @ralph_tang, @karun_kumar_, @corefpark, and @Hidenori8Tanaka! It was amazing collaborating across Harvard, NTT, Comcast, and Microsoft. Excited to hear thoughts from those in generative modeling, physical AI, and mechanistic understanding.
arxiv.org/abs/2605.20299