线性回归最小二乘法闭式解

发表于 分类于

w^∗=arg min⁡w^(y−Xw^)T(y−Xw^).(1)\hat{\bm{w}}^* = \argmin_{\hat{\bm{w}}}(\bm{y} - \mathbf{X}\hat{\bm{w}})^\mathrm{T} (\bm{y} - \mathbf{X}\hat{\bm{w}}). \tag{1}

记

J(w^):=(y−Xw^)T(y−Xw^),(2)J(\hat{\bm{w}}) := (\bm{y} - \mathbf{X}\hat{\bm{w}})^\mathrm{T} (\bm{y} - \mathbf{X}\hat{\bm{w}}), \tag{2}

展开得到

J(w^)=yTy−2w^TXTy+w^TXTXw^.(3)J(\hat{\bm{w}}) = \bm{y}^\mathrm{T}\bm{y} - 2\hat{\bm{w}}^\mathrm{T}\mathbf{X}^\mathrm{T}\bm{y} + \hat{\bm{w}}^\mathrm{T}\mathbf{X}^\mathrm{T}\mathbf{X}\hat{\bm{w}}. \tag{3}

对 w^\hat{\bm{w}} 求导：

∂J(w^)∂w^=−2XTy+2XTXw^,(4)\frac{\partial J(\hat{\bm{w}})}{\partial \hat{\bm{w}}} = -2\mathbf{X}^\mathrm{T}\bm{y} + 2\mathbf{X}^\mathrm{T}\mathbf{X}\hat{\bm{w}}, \tag{4}

令梯度为 0，得

XTXw^=XTy.(5)\mathbf{X}^\mathrm{T}\mathbf{X}\hat{\bm{w}} = \mathbf{X}^\mathrm{T}\bm{y}. \tag{5}

当 XTX\mathbf{X}^\mathrm{T}\mathbf{X} 可逆，即 XTX\mathbf{X}^\mathrm{T}\mathbf{X} 是满秩矩阵或正定矩阵时，

w^∗=(XTX)−1XTy.(6)\boxed{ \hat{\bm{w}}^* = (\mathbf{X}^\mathrm{T}\mathbf{X})^{-1}\mathbf{X}^\mathrm{T}\bm{y}. } \tag{6}