Zhihao Cen, Chuhua Xian, Hailin Sun, Yuliang Liufu, Zhen Zhang, Xiangyu Chu, Hongmin Cai, Yunbo Zhang, Guoxin Fang
Observing a deformable object reveals its surface motion, but many applications also need its internal stress, material properties, and interaction forces. RealSimLoop combines differentiable physical simulation with vision feedback to keep a simulated object aligned with its real-world behavior. A neural reduced-order model makes both simulation and gradient computation more efficient. Coupled with differentiable 3D Gaussian Splatting, it allows camera observations to guide material-parameter updates within a sliding time window. This continuous adaptation tracks changing stiffness and compensates for simulation mismatch. We demonstrate the framework through robotic manipulation, multi-material structures, temperature-dependent stiffness tracking, force prediction, and stress visualization.