Fokker-Planck analysis and invariant laws for a continuous-time stochastic model inspired by Adam

Abstract

We study a degenerate stochastic system inspired by bias-corrected Adam. The system arises as the continuous-time limit of an explicitly modified recursion in which the squared-gradient update is replaced by a prescribed second-moment closure. We construct an entrance solution at the singular initial time and prove convergence of the modified recursion on finite time intervals away from zero. Bias correction enters through time-dependent coefficients which converge to constants as time tends to infinity. For a smooth objective with globally Lipschitz gradient and radial dissipativity at infinity, we prove existence and uniqueness of an invariant probability measure for the autonomous system under either $b<4a$ or $γL_f\le a(σ/\sqrt2+\varepsilon)/4$. Here $L_f$ is the Lipschitz constant of the gradient, $a,b$ are the relaxation rates, $γ$ is the learning-rate parameter, and $σ,\varepsilon$ are the noise and regularization parameters. The proof combines an exponential Lyapunov function with compact-set minorization obtained from explicit control paths and a regular endpoint map. Under an exponential Lyapunov moment condition, we obtain exponential convergence in weighted total variation and in the squared $2$-Wasserstein distance of the complete state. The invariant law has finite moments of every order, and qualitative total variation convergence holds for every initial probability law. The time-inhomogeneous process converges weakly to the same invariant law. We formulate the associated Fokker-Planck equation for probability measures and establish local hypoellipticity and smoothness of the invariant density where $\operatorname{Diag}(\nabla f)D^2f$ is invertible. A quadratic example shows that a uniform exponential convergence estimate requires a prefactor growing exponentially in the initial position along a specified family of states.