Sharper Bounds for Regression and Low-Rank Approximation with Regularization.
arXiv: Data Structures and Algorithms(2016)
摘要
The technique of matrix sketching, such as the use of random projections, has been shown in recent years to be a powerful tool for accelerating many important statistical learning techniques. Research has so far focused largely on using sketching for the vanilla un-regularized versions of these techniques. Here we study sketching methods for regularized variants of linear regression, low rank approximations, and canonical correlation analysis. study regularization both in a fairly broad setting, and in the specific context of the popular and widely used technique of ridge regularization; for the latter, as applied to each of these problems, we show algorithmic resource bounds in which the {em statistical dimension} appears in places where in previous bounds the rank would appear. The statistical dimension is always smaller than the rank, and decreases as the amount of regularization increases. In particular, for the ridge low-rank approximation problem $min_{Y,X} lVert YX - A rVert_F^2 + lambda lVert YrVert_F^2 + lambdalVert X rVert_F^2$, where $Yinmathbb{R}^{ntimes k}$ and $Xinmathbb{R}^{ktimes d}$, we give an approximation algorithm needing [ O(mathtt{nnz}(A)) + tilde{O}((n+d)varepsilon^{-1}k min{k, varepsilon^{-1}mathtt{sd}_lambda(Y^*)})+ tilde{O}(varepsilon^{-8} mathtt{sd}_lambda(Y^*)^3) ] time, where $s_{lambda}(Y^*)le k$ is the statistical dimension of $Y^*$, $Y^*$ is an optimal $Y$, $varepsilon$ is an error parameter, and $mathtt{nnz}(A)$ is the number of nonzero entries of $A$. We also study regularization in a much more general setting. For example, we obtain sketching-based algorithms for the low-rank approximation problem $min_{X,Y} lVert YX - A rVert_F^2 + f(Y,X)$ where $f(cdot,cdot)$ is a regularizing function satisfying some very general conditions (chiefly, invariance under orthogonal transformations).
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络