text: 229,136,1301792837214797827,561899047,Aki Vehtari,"Akash Dhaka, @AleexCatalina, @Michael_riis, @MansMeg, @jhhhuggins, and I have a new paper ""Robust, Accurate Stochastic Optimization for Variational Inference"" <LINK> <LINK> tl;dr We combine Polyak–Ruppert averaging with MCMC convergence diagnostics to make stochastic optimization in variational inference more robust or get a warning when it performs badly. These help to make automated use of VI safer in probabilistic programming frameworks. <LINK> Many VI methods use stochastic optimization either due to using random mini-batches of data or Monte Carlo to estimate expectations of the divergences. For example. For example, autodiff VI (ADVI) in Stan has stochastic target and gradients due to the latter. To fulfill Robbins-Monroe condition of reaching eventually the optimum, the step size is usually gradually decreased. Although this guarantees asymptotic convergence, it may take unfeasible amount of time and the last iteration in finite time can be far from the optimum. Under certain conditions, stochastic optimization with a fixed step size converges to a finite variance stationary process around the optimum. Average of the iterations converges towards the optimum faster. This is an old but under-used idea, aka Polyak–Ruppert averaging. Recently iterate averaging has been used also with name stochastic weight averaging (SWA) in context of deep learning. What we add is 1) a diagnostic for detecting when we have reached stationarity and can start averaging, and 2) a standard error estimate to decide when we can stop averaging or give up if the standard error is not decreasing (indicating violation of conditions). The diagnostics are familiar from MCMC convergence literature (e.g. Rhat, MCSE, and autocorrelation) and VI diagnostics literature (e.g. Pareto k). @lauretig Oops, forgot that. The repo will be public next week.",https://arxiv.org/abs/2009.00666,"We consider the problem of fitting variational posterior approximations using stochastic optimization methods. The performance of these approximations depends on (1) how well the variational family matches the true posterior distribution,(2) the choice of divergence, and (3) the optimization of the variational objective. We show that even in the best-case scenario when the exact posterior belongs to the assumed variational family, common stochastic optimization methods lead to poor variational approximations if the problem dimension is moderately large. We also demonstrate that these methods are not robust across diverse model types. Motivated by these findings, we develop a more robust and accurate stochastic optimization framework by viewing the underlying optimization algorithm as producing a Markov chain. Our approach is theoretically motivated and includes a diagnostic for convergence and a novel stopping rule, both of which are robust to noisy evaluations of the objective function. We show empirically that the proposed framework works well on a diverse set of models: it can automatically detect stochastic optimization failure or inaccurate variational approximation ","Robust, Accurate Stochastic Optimization for Variational Inference",9,"['Akash Dhaka, @AleexCatalina, @Michael_riis, @MansMeg, @jhhhuggins, and I have a new paper ""Robust, Accurate Stochastic Optimization for Variational Inference"" <LINK> <LINK>', 'tl;dr We combine Polyak–Ruppert averaging with MCMC convergence diagnostics to make stochastic optimization in variational inference more robust or get a warning when it performs badly. These help to make automated use of VI safer in probabilistic programming frameworks. https://t.co/3RUKPPdzfs', 'Many VI methods use stochastic optimization either due to using random mini-batches of data or Monte Carlo to estimate expectations of the divergences. For example. For example, autodiff VI (ADVI) in Stan has stochastic target and gradients due to the latter.', 'To fulfill Robbins-Monroe condition of reaching eventually the optimum, the step size is usually gradually decreased. Although this guarantees asymptotic convergence, it may take unfeasible amount of time and the last iteration in finite time can be far from the optimum.', 'Under certain conditions, stochastic optimization with a fixed step size converges to a finite variance stationary process around the optimum. Average of the iterations converges towards the optimum faster. This is an old but under-used idea, aka Polyak–Ruppert averaging.', 'Recently iterate averaging has been used also with name stochastic weight averaging (SWA) in context of deep learning.', 'What we add is 1) a diagnostic for detecting when we have reached stationarity and can start averaging, and 2) a standard error estimate to decide when we can stop averaging or give up if the standard error is not decreasing (indicating violation of conditions).', 'The diagnostics are familiar from MCMC convergence literature (e.g. Rhat, MCSE, and autocorrelation) and VI diagnostics literature (e.g. Pareto k).', '@lauretig Oops, forgot that. The repo will be public next week.']",20,09,1850