Stochastic Hessian Fitting on Lie Group

This paper studies the fitting of Hessian or its inverse with stochastic <PRE_TAG>Hessian-vector products</POST_TAG>. A <PRE_TAG>Hessian fitting criterion</POST_TAG>, which can be used to derive most of the commonly used methods, e.g., BFGS, Gaussian-Newton, AdaGrad, etc., is used for the analysis. Our studies reveal different convergence rates for different Hessian fitting methods, e.g., sub<PRE_TAG>linear rates</POST_TAG> for gradient descent in the Euclidean space and a commonly used closed-form solution, linear rates for gradient descent on the manifold of symmetric positive definite (SPL) matrices and certain Lie groups. The Hessian fitting problem is further shown to be strongly convex under mild conditions on a specific yet general enough Lie group. To confirm our analysis, these methods are tested under different settings like noisy <PRE_TAG><PRE_TAG>Hessian-vector products</POST_TAG></POST_TAG>, time varying <PRE_TAG>Hessians</POST_TAG>, and low precision arithmetic. These findings are useful for stochastic second order optimizations that rely on fast, robust and accurate <PRE_TAG>Hessian estimations</POST_TAG>.