前言

利用 gradient descent 方法來求近似迴歸直線與近似迴歸方程式, 並舉實例說明且預測房價, 請看下文的剖析。

本文

一、 已知 $(x_i,y_i)$, $i=1,2,\ldots,n$, 求迴歸直線

1. 設直線 $y=mx+b$

$$f(m,b)=\frac 12\sum_{i=1}^n [y_i-(mx_i+b)]^2.$$

為了求迴歸直線我們極小化 $f(m,b)=\dfrac 12\sum\limits_{i=1}^n [y_i-(mx_i+b)]^2$。

2.

\begin{align} \dfrac{\partial f}{\partial m}=\displaystyle\sum\limits_{i=1}^n [y_i-(mx_i+b)]\cdot (-x_i)=0$, &\Rightarrow \ -\sum_{i=1}^n x_iy_i+m\Big(\sum_{n=1}^n x_i^2\Big)+b\Big(\sum_{n=1}^n x_i\Big)=0, &\Rightarrow \ m\Big(\sum_{i=1}^n x_i^2\Big)+bn\bar x=\sum_{n=1}^n x_iy_i. \nonumber \\ \end{align}

3.

\begin{align} \dfrac{\partial f}{\partial b}=\displaystyle\sum\limits_{i=1}^n [y_i-(mx_i+b)]\cdot (-1)=0$, &\Rightarrow \ \sum_{i=1}^n y_i=m\Big(\sum_{n=1}^n x_i\Big)+nb\ \Rightarrow \ n\bar y=m(n\bar x)+nb, &\Rightarrow \ \bar y=m\bar x+b. \nonumber \\ \end{align}

4.

\begin{align} m\Big(\sum_{i=1}^n x_i^2\Big)+bn\bar x=\sum_{n=1}^n x_iy_i,\hskip 4.5cm \label{1}\\ \bar y=m\bar x+b.\label{2} \end{align} \begin{align} &\hbox{由 \eqref{2} 式可得 }\Rightarrow \ b=\bar y- m\bar x.\label{3}\\ &\hbox{代入 \eqref{1} 式可得 }m\Big(\sum_{i=1}^n x_i^2\Big)+n\bar x(\bar y-m\bar x)=\sum_{n=1}^n x_iy_i,\nonumber \\ &\Rightarrow \ m\Big[\Big(\frac{\sum\limits_{i=1}^n x_i^2}{n}\Big)-(\bar x)^2\Big]=\frac{\sum\limits_{i=1}^n x_iy_i}{n}-\bar x\cdot\bar y,\nonumber \\ &\Rightarrow \ m=\dfrac{\dfrac{\sum\limits_{i=1}^n x_iy_i}n- \bar x\cdot\bar y}{\dfrac{\sum\limits_{i=1}^n x_i^2}n-(\bar x)^2} \ \Rightarrow \ m=\dfrac{\sum\limits_{i=1}^n x_iy_i- n\bar x\cdot\bar y}{\sum\limits_{i=1}^n x_i^2-{n}(\bar x)^2}.\label{4} \end{align}

5.

\begin{align} \dfrac{\partial^2 f}{\partial m^2}=\sum\limits_{i=1}^n x_i^2, \dfrac{\partial^2 f}{\partial b^2}=\sum\limits_{i=1}^n 1=n, \nonumber \\ \dfrac{\partial^2 f}{\partial m\partial b}=\dfrac{\partial^2 f}{\partial b\partial m}=\sum\limits_{i=1}^n x_i。\nonumber \\ \end{align} \begin{align*} &\hbox{海森 (Hessian) 行列式}=\Delta=\left|\begin{array}{ccc} \dfrac{\partial^2 f}{\partial m^2}& &\dfrac{\partial^2 f}{\partial m\partial b}\\ \dfrac{\partial^2 f}{\partial b\partial m}&&\dfrac{\partial^2 f}{\partial b^2} \end{array}\right|=\left|\begin{array}{ccc} \sum\limits_{i=1}^n x_i^2& &\sum\limits_{i=1}^n x_i\\ \sum\limits_{i=1}^n x_i&&n \end{array}\right|\\ &=n\sum\limits_{i=1}^n x_i^2-\Big(\sum\limits_{i=1}^n x_i\Big)^2=\sum_{1\le i\lt j\le n}(x_i-x_j)^2\gt0 \quad \hbox{且}\quad \dfrac{\partial^2 f}{\partial m^2} =\sum\limits_{i=1}^n x_i^2\gt0. \end{align*}

因此當 $m=\dfrac{\sum\limits_{i=1}^n x_iy_i- n\bar x\cdot\bar y}{\sum\limits_{i=1}^n x_i^2-{n}(\bar x)^2}$ 且 $b=\bar y-m\bar x$ 時 $f(m,b)=\dfrac 12\cdot \sum\limits_{i=1}^n [y_i- (mx_i+b)]^2$ 有極小值。

6. 由1. 到 5. 得知, 當 $(x_i,y_i)$, $i=1,2,\ldots,n$, 所求迴歸直線為 $y=mx+b$ 其中

$$m=\dfrac{\sum\limits_{i=1}^n x_iy_i- n\bar x\cdot\bar y}{\sum\limits_{i=1}^n x_i^2-{n}(\bar x)^2},\quad b=\bar y-m\bar x.$$

二、例子

已知 $(x_1,y_1)=(1,5)$, $(x_2,y_2)=(2,6)$, $(x_3,y_3)=(3,10)$, $(x_4,y_4)=(4,13)$, $(x_5,y_5)=(5,11)$, 求迴歸直線

1. 由一、 得知

當 $(x_i,y_i)$, $i=1,2,\ldots,5$, 所求迴歸直線為 $y=mx+b$ 其中

\begin{align*} &\hskip 1.4cm m=\dfrac{\sum\limits_{i=1}^5 x_iy_i- 5\bar x\cdot\bar y}{\sum\limits_{i=1}^5 x_i^2-{5}(\bar x)^2},\quad b=\bar y-m\bar x.\\ 2. \hskip 1cm &\sum_{i=1}^5 x_i=1+2+3+4+5=15\ \Rightarrow\ \bar x=\frac{15}{5}=3,\\ &\sum_{i=1}^5 y_i=5+6+10+13+11=45\ \Rightarrow\ \bar y=\frac{45}{5}=9,\hskip 5.4cm \\ &\sum_{i=1}^5 x_i^2=1+4+9+16+25=55,\\ &\sum_{i=1}^5 x_iy_i=5+12+30+52+55=154. \end{align*}

代入 \eqref{4} 式 $m=\dfrac{\sum\limits_{i=1}^n x_iy_i- n\bar x\cdot\bar y}{\sum\limits_{i=1}^n x_i^2-{n}(\bar x)^2}$, $n=5$。 得到 $m=\dfrac{\sum\limits_{i=1}^5 x_iy_i- 5\bar x\cdot\bar y}{\sum\limits_{i=1}^5 x_i^2-{5}(\bar x)^2}$

$=\dfrac{154-5(3)(9)}{55-5(3)^2}= \dfrac{154-135}{55-45}=\dfrac{19}{10}=1.9$。 將 $m=1.9$ 代入 \eqref{3} 式

$b=\bar y-m\bar x=9-1.9(3)=9-5.7=3.3$, 因此求得迴歸直線為 $y=1.9x+3.3$。

三、利用 gradient descent 方法來求, 二、 例子 當 $(x_1,y_1)=(1,5)$, $(x_2,y_2)=(2,6)$, $(x_3,y_3)=(3,10)$, $(x_4,y_4)=(4,13)$, $(x_5,y_5)=(5,11)$ 的近似迴歸直線。

1. 起始點 $(m_0,b_0)=(0,0)$。
2. 機器學習率 $\eta=0.001$。
3. $(m_{i+1},b_{i+1})=(m_i,b_i)-\eta\nabla f(m,b)\Big|_{(m_i,b_i)}$

\begin{align*} =\,&(m_i,b_i)-\eta\Big(\frac{\partial f(m,b)}{\partial m}\Big|_{(m_i,b_i)},\frac{\partial f(m,b)}{\partial b}\Big|_{(m_i,b_i)}\Big),\hskip 2.2cm \\ \hbox{其中}\quad &\frac{\partial f(m,b)}{\partial m}\Big|_{(m_i,b_i)}=\sum_{i=1}^5 [y_i-(m_ix_i+b_i)]\cdot(-x_i),\\ &\frac{\partial f(m,b)}{\partial b}\Big|_{(m_i,b_i)}=\sum_{i=1}^5 [y_i-(m_ix_i+b_i)]\cdot(-1). \end{align*}

4. 疊代 10000 次即 $i=0,1,2,\ldots, 9999$

\begin{align*} \hbox{得到}\ (m_1,b_1)=\,&(0.154,0.045),&(m_2,b_2)=&(0.298855,0.087465),\\ (m_3,b_3)=\,&(0.435106,0.127545),&(m_4,b_4)=&(0.563262,0.165381),\\ (m_5,b_5)=\,&(0.683802,0.201105),&(m_6,b_6)=&(0.797176,0.234842),\\ (m_7,b_7)=\,&(0.903809,0.266710),&(m_8,b_8)=&(1.004099,0.296820),\\ (m_9,b_9)=\,&(1.098421,0.325274),&(m_{10},b_{10})=&(1.187129,0.352171),\\[-6pt] & \quad\qquad\vdots&&\\[-6pt] &\hskip -40pt (m_{50},b_{50})=(2.460078,0.796231),&&\\[-6pt] & \quad\qquad\vdots&&\\[-6pt] &\hskip -40pt (m_{100},b_{100})=(2.549805,0.931158),&&\\[-6pt] & \quad\qquad\vdots&&\\[-6pt] &\hskip -40pt (m_{1000},b_{1000})=(2.206322,2.194082),&&\\[-6pt] & \quad\qquad\vdots&&\\[-6pt] &\hskip -40pt (m_{5000},b_{5000})=(1.910405,3.262436),&&\\[-6pt] & \quad\qquad\vdots&&\\[-6pt] &\hskip -40pt (m_{10000},b_{10000})=(1.900152,3.299452).&& \end{align*}

我們求得 $y=m_{10000}x+b_{10000} \ \Rightarrow\ y=1.900152x+3.299452$ 為近似迴歸直線。

四、多元迴歸分析例子

\begin{align*} \hbox{已知當}\ (x_1,t_1,y_1)=\,&(57,8,64),\, (x_2,t_2,y_2)=(59,10,71),\, (x_3,t_3,y_3)=(49,6,53),\\ \ (x_4,t_4,y_4)=\,&(62,11,67),\, (x_5,t_5,y_5)=(51,8,55),\, (x_6,t_6,y_6)=(50,7,58),\\ \ (x_7,t_7,y_7)=\,&(55,10,77),\, (x_8,t_8,y_8)=(48,9,57), \hbox{求迴歸方程式} \ y\!=\!mx\!+\!ut\!+\!b. \end{align*}

1. 設迴歸方程式為 $y\!=\!mx\!+\!ut\!+\!b$, 令

$$f(m,u,b)=\dfrac 12\sum\limits_{i=1}^8[y_i-(mx_i+ut_i+b)]^2,$$

為了求迴歸方程式我們極小化

$$f(m,u,b)=\dfrac 12\sum\limits_{i=1}^8[y_i-(mx_i+ut_i+b)]^2.$$

2. $\dfrac{\partial f}{\partial m}=\displaystyle\sum\limits_{i=1}^8[y_i-(mx_i+ut_i+b)]\cdot(-x_i)=0$,

\begin{align} \Rightarrow\ &-\sum_{i=1}^8 x_iy_i+m\Big(\sum_{i=1}^8 x_i^2\Big)+u\Big(\sum_{i=1}^8 x_it_i\Big)+b\Big(\sum_{i=1}^8 x_i\Big)=0,\hskip 2cm \nonumber \\ \Rightarrow\ &m\Big(\sum_{i=1}^8 x_i^2\Big)+u\Big(\sum_{i=1}^8 x_it_i\Big)+b\Big(\sum_{i=1}^8 x_i\Big)=\sum_{i=1}^8 x_iy_i.\label{5} \end{align}

3. $\dfrac{\partial f}{\partial u}=\displaystyle\sum\limits_{i=1}^8[y_i-(mx_i+ut_i+b)]\cdot(-t_i)=0$,

\begin{align} \Rightarrow\ &-\sum_{i=1}^8 t_iy_i+m\Big(\sum_{i=1}^8 x_it_i\Big)+u\Big(\sum_{i=1}^8 t_i^2\Big)+b\Big(\sum_{i=1}^8 t_i\Big)=0,\hskip 2cm \nonumber \\ \Rightarrow\ &m\Big(\sum_{i=1}^8 x_it_i\Big)+u\Big(\sum_{i=1}^8 t_i^2\Big)+b\Big(\sum_{i=1}^8 t_i\Big)=\sum_{i=1}^8 t_iy_i.\label{6} \end{align}

4. $\dfrac{\partial f}{\partial b}=\displaystyle\sum\limits_{i=1}^8[y_i-(mx_i+ut_i+b)]\cdot(-1)=0$,

\begin{align} \Rightarrow\ &m\Big(\sum_{i=1}^8 x_i\Big)+u\Big(\sum_{i=1}^8 t_i\Big)+8b=\sum_{i=1}^8 y_i.\hskip 4.5cm \label{7} \end{align}

5. 由 \eqref{5} \eqref{6} \eqref{7} 式可得矩陣形式

$$\left[\begin{array}{ccc} 8& \displaystyle\sum\limits_{i=1}^8 x_i &\displaystyle\sum\limits_{i=1}^8 t_i\\ \displaystyle\sum\limits_{i=1}^8 x_i&\displaystyle\sum\limits_{i=1}^8 x_i^2&\displaystyle\sum\limits_{i=1}^8 x_it_i\\ \displaystyle\sum\limits_{i=1}^8 t_i&\displaystyle\sum\limits_{i=1}^8 x_it_i&\displaystyle\sum\limits_{i=1}^8 t_i^2 \end{array}\right] \left[\begin{array}{c} b\\ m\\ u \end{array}\right]=\left[\begin{array}{c} \displaystyle\sum\limits_{i=1}^8 y_i\\ \displaystyle\sum\limits_{i=1}^8 x_iy_i\\ \displaystyle\sum\limits_{i=1}^8 t_iy_i \end{array}\right], \hskip 4cm $$

推得

\begin{align*} &\left[\begin{array}{ccc} 8&431&69\\ 431&23405&3762\\ 69&3762&615 \end{array}\right] \left[\begin{array}{c} b\\ m\\ u \end{array}\right]=\left[\begin{array}{c} 502\\ 27264\\ 4406 \end{array}\right],\\ &\Rightarrow \left[\begin{array}{c} b\\ m\\ u \end{array}\right]=\left[\begin{array}{ccc} 8&431&69\\ 431&23405&3762\\ 69&3762&615 \end{array}\right]^{-1}\left[\begin{array}{c} 502\\ 27264\\ 4406 \end{array}\right],\\ &\Rightarrow \left[\begin{array}{c} b\\ m\\ u \end{array}\right]=\frac 1{13464}\cdot \left[\begin{array}{ccc} 241431&-5487&6477\\ -5487&159&-357\\ 6477&-357&1479 \end{array}\right]\cdot\left[\begin{array}{c} 502\\ 27264\\ 4406 \end{array}\right]=\left[\begin{array}{c} \dfrac{1923}{187}\\[9pt] \dfrac{105}{187}\\[9pt] \dfrac{1445}{561} \end{array}\right], \end{align*}

求得迴歸方程式 $y\!=\!mx\!+\!ut\!+\!b=\dfrac{105}{187}x+ \dfrac{1445}{561}t+\dfrac{1923}{187}$。

利用 gradient descent 方法來求 :

\begin{align*} \hbox{已知當}\ (x_1,t_1,y_1)=\,&(57,8,64),\, (x_2,t_2,y_2)=(59,10,71),\, (x_3,t_3,y_3)=(49,6,53),\\ \ (x_4,t_4,y_4)=\,&(62,11,67),\, (x_5,t_5,y_5)=(51,8,55),\, (x_6,t_6,y_6)=(50,7,58),\\ \ (x_7,t_7,y_7)=\,&(55,10,77),\, (x_8,t_8,y_8)=(48,9,57)\, \hbox{時, 近似迴歸方程式為何？} \end{align*}

設近似迴歸方程式為 $y\!=\!mx\!+\!ut\!+\!b$, $f(m,u,b)=\dfrac 12\sum\limits_{i=1}^8[y_i\!-\!(mx_i\!+\!ut_i\!+\!b)]^2$。

6. 起始點 $(m_0,u_0,b_0)=(0,0,0)$。
7. 機器學習率 $\eta=0.000001$。
8. $(m_{i+1},u_{i+1},b_{i+1})=(m_i,u_i,b_i)-\eta\nabla f(m,u,b)\Big|_{(m_i,u_i,b_i)}$

\begin{align*} &\hskip -30pt =(m_i,u_i,b_i)-\eta\Big(\frac{\partial f(m,u,b)}{\partial m}\Big|_{(m_i,u_i,b_i)},\frac{\partial f(m,u,b)} {\partial u}\Big|_{(m_i,u_i,b_i)},\frac{\partial f(m,u,b)}{\partial b}\Big|_{(m_i,u_i,b_i)}\Big),\\ \hbox{其中}\ &\frac{\partial f(m,u,b)}{\partial m}\Big|_{(m_i,u_i,b_i)}=\sum_{i=1}^8[y_i-(mx_i+ut_i+b)]\cdot(-x_i),\\ &\frac{\partial f(m,u,b)}{\partial u}\Big|_{(m_i,u_i,b_i)}=\sum_{i=1}^8[y_i-(mx_i+ut_i+b)]\cdot(-t_i),\\ &\frac{\partial f(m,u,b)}{\partial b}\Big|_{(m_i,u_i,b_i)}=\sum_{i=1}^8[y_i-(mx_i+ut_i+b)]\cdot(-1). \end{align*}

9. 疊代 200000000 次即 $i\!=\!0,1,2,\ldots,199999999$, 得到 $(m_{200000000},u_{200000000},b_{200000000})$ $=(0.56150,2.57575,10.28327)$, 因此近似迴歸方程式為 $y=m_{200000000}x+u_{200000000}t+b_{200000000}$, 即 $y=0.56150x+2.57575t+10.28327$。

五、房屋價格預測

1. $x_i$ 為第 $i$ 間房子的面積, 單位平方英呎。
$t_i$ 為第 $i$ 間房子的臥房數。
$v_i$ 為第 $i$ 間房子的屋齡。
$y_i$ 為第 $i$ 間房子的價格, 單位美元。

2. 資料

\begin{align*} (x_1,t_1,v_1,y_1)=&(2600,3,20,550000),&(x_2,t_2,v_2,y_2)=&(3000,4,15,565000),\\ (x_3,t_3,v_3,y_3)=&(3200,3,18,610000),&(x_4,t_4,v_4,y_4)=&(3600,3,30,595000),\\ (x_5,t_5,v_5,y_5)=&(4000,5,8,760000).&& \end{align*}

3. 設迴歸方程式為 $y\!=\!mx\!+\!ut\!+\!rv\!+\!b$, 令

$$f(m,u,r,b)=\dfrac 12\sum\limits_{i=1}^5[y_i\!-\!(mx_i\!+\!ut_i\!+\!rv_i\!+\!b)]^2,$$

為了求迴歸方程式我們極小化

$$f(m,u,r,b)=\dfrac 12\sum\limits_{i=1}^5[y_i\!-\!(mx_i\!+\!ut_i\!+\!rv_i\!+\!b)]^2.$$

4. $\dfrac{\partial f}{\partial m}=\displaystyle\sum\limits_{i=1}^5[y_i-(mx_i+ut_i+rv_i+b)]\cdot(-x_i)=0$,

\begin{align} \Rightarrow\ &-\sum_{i=1}^5 x_iy_i+m\Big(\sum_{i=1}^5 x_i^2\Big)+u\Big(\sum_{i=1}^5 x_it_i\Big)+r\Big(\sum_{i=1}^5 x_iv_i\Big) +b\Big(\sum_{i=1}^5 x_i\Big)=0,\hskip .5cm \nonumber \\ \Rightarrow\ &m\Big(\sum_{i=1}^5 x_i^2\Big)+u\Big(\sum_{i=1}^5 x_it_i\Big)+r\Big(\sum_{i=1}^5 x_iv_i\Big) +b\Big(\sum_{i=1}^5 x_i\Big)=\sum_{i=1}^5 x_iy_i.\label{8} \end{align}

5. $\dfrac{\partial f}{\partial u}=\displaystyle\sum\limits_{i=1}^5[y_i-(mx_i+ut_i+rv_i+b)]\cdot(-t_i)=0$,

\begin{align} \Rightarrow\ &-\sum_{i=1}^5 t_iy_i+m\Big(\sum_{i=1}^5 x_it_i\Big)+u\Big(\sum_{i=1}^5 t_i^2\Big)+r\Big(\sum_{i=1}^5 t_iv_i\Big) +b\Big(\sum_{i=1}^5 t_i\Big)=0,\hskip .1cm \nonumber \\ \Rightarrow\ &m\Big(\sum_{i=1}^5 x_it_i\Big)+u\Big(\sum_{i=1}^5 t_i^2\Big)+r\Big(\sum_{i=1}^5 t_iv_i\Big) +b\Big(\sum_{i=1}^5 t_i\Big)=\sum_{i=1}^5 t_iy_i.\label{9} \end{align}

6. $\dfrac{\partial f}{\partial r}=\displaystyle\sum\limits_{i=1}^5[y_i-(mx_i+ut_i+rv_i+b)]\cdot(-v_i)=0$,

\begin{align} \Rightarrow\ &-\sum_{i=1}^5 v_iy_i+m\Big(\sum_{i=1}^5 v_ix_i\Big)+u\Big(\sum_{i=1}^5 v_it_i\Big)+r\Big(\sum_{i=1}^5 v_i^2\Big) +b\Big(\sum_{i=1}^5 v_i\Big)=0,\hskip .1cm \nonumber \\ \Rightarrow\ &m\Big(\sum_{i=1}^5 v_ix_i\Big)+u\Big(\sum_{i=1}^5 v_it_i\Big)+r\Big(\sum_{i=1}^5 v_i^2\Big) +b\Big(\sum_{i=1}^5 v_i\Big)=\sum_{i=1}^5 v_iy_i.\label{10} \end{align}

7. $\dfrac{\partial f}{\partial b}=\displaystyle\sum\limits_{i=1}^5[y_i-(mx_i+ut_i+rv_i+b)]\cdot(-1)=0$,

\begin{align} \Rightarrow\ &-\sum_{i=1}^5 y_i+m\Big(\sum_{i=1}^5 x_i\Big)+u\Big(\sum_{i=1}^5 t_i\Big)+r\Big(\sum_{i=1}^5 v_i\Big)+5b=0, \nonumber \\ \Rightarrow\ &m\Big(\sum_{i=1}^5 x_i\Big)+u\Big(\sum_{i=1}^5 t_i\Big)+r\Big(\sum_{i=1}^5 v_i\Big) +5b=\sum_{i=1}^5 y_i.\hskip 3.5cm \label{11} \end{align}

8. 由 \eqref{8} \eqref{9} \eqref{10} \eqref{11} 式可得矩陣形式

$$\left[\begin{array}{cccc} 5& \displaystyle\sum\limits_{i=1}^5 x_i &\displaystyle\sum\limits_{i=1}^5 t_i&\displaystyle\sum\limits_{i=1}^5 v_i\\ \displaystyle\sum\limits_{i=1}^5 x_i&\displaystyle\sum\limits_{i=1}^5 x_i^2& \displaystyle\sum\limits_{i=1}^5 t_ix_i &\displaystyle\sum\limits_{i=1}^5 v_ix_i\\ \displaystyle\sum\limits_{i=1}^5 t_i& \displaystyle\sum\limits_{i=1}^5 t_ix_i &\displaystyle\sum\limits_{i=1}^5 t_i^2& \displaystyle\sum\limits_{i=1}^5 v_it_i \\ \displaystyle\sum\limits_{i=1}^5 v_i&\displaystyle\sum\limits_{i=1}^5 v_ix_i&\displaystyle\sum\limits_{i=1}^5 v_it_i&\displaystyle\sum\limits_{i=1}^5 v_i^2 \end{array}\right] \left[\begin{array}{c} b\\ m\\ u\\ r \end{array}\right]=\left[\begin{array}{c} \displaystyle\sum\limits_{i=1}^5 y_i\\ \displaystyle\sum\limits_{i=1}^5 x_iy_i\\ \displaystyle\sum\limits_{i=1}^5 t_iy_i\\ \displaystyle\sum\limits_{i=1}^5 v_iy_i \end{array}\right], \hskip 4cm $$

推得

\begin{align*} &\left[\begin{array}{cccc} 5&16400&18&91\\ 16400&54960000&60200&294600\\ 18&60200&68&304\\ 91&294600&304&1913 \end{array}\right] \left[\begin{array}{c} b\\ m\\ u\\ r \end{array}\right]=\left[\begin{array}{c} 3080000\\ 10259000000\\ 11325000\\ 54385000 \end{array}\right],\\ &\Rightarrow \left[\begin{array}{c} b\\ m\\ u\\ r \end{array}\right]=\left[\begin{array}{cccc} 5&16400&18&91\\ 16400&54960000&60200&294600\\ 18&60200&68&304\\ 91&294600&304&1913 \end{array}\right]^{-1}\left[\begin{array}{c} 3080000\\ 10259000000\\ 11325000\\ 54385000 \end{array}\right],\\ &\Rightarrow \left[\begin{array}{c} b\\ m\\ u\\ r \end{array}\right]\!=\!\frac 1{640000000}\cdot \left[\begin{array}{cccc} 18613240000&732400&-3843160000&-387480000\\ 732400&1324&-1031600&-74800\\ -3843160000&-1031600&1424440000&115320000\\ -387480000&-74800&115320000&11960000 \end{array}\right]\\ &\hskip 2cm \cdot\! \left[\begin{array}{c} 3080000\\ 10259000000\\ 11325000\\ 54385000 \end{array}\right] =\left[\begin{array}{c} 383725\\ 137.25\\ -26025\\ -6825 \end{array}\right], \end{align*}

求得迴歸方程式 $y\!=\!mx\!+\!ut\!+\!rv\!+\!b=137.25x-26025t-6825v+383725$。

(1) 將 $(x,t,v)=(3000,3,40)$ 代入迴歸方程式

\begin{align*} y=\,&137.25x-26025t-6825v+383725\\ =\,&137.25(3000)-26025(3)-6825(40)+383725=444400. \end{align*}

我們可以預測當房子的面積 3000 平方英呎, 有 3 間臥房, 屋齡 40 年, 房子的價格為 444400 美元。

(2) 將 $(x,t,v)=(2500,4,5)$ 代入迴歸方程式

\begin{align*} y=\,&137.25x-26025t-6825v+383725\\ =\,&137.25(2500)-26025(4)-6825(5)+383725=588625. \end{align*}

我們可以預測當房子的面積 2500 平方英呎, 有 4 間臥房, 屋齡 5 年, 房子的價格為 588625 美元。

利用 gradient descent 方法來求 :

\begin{align*} \hbox{已知當}\ (x_1,t_1,v_1,y_1)=&(2600,3,20,550000),\ &(x_2,t_2,v_2,y_2)=&(3000,4,15,565000),\\ (x_3,t_3,v_3,y_3)=&(3200,3,18,610000),\ &(x_4,t_4,v_4,y_4)=&(3600,3,30,595000),\\ (x_5,t_5,v_5,y_5)=&(4000,5,8,760000)&& \end{align*}

時, 近似迴歸方程式為何？

設近似迴歸方程式為 $y\!=\!mx\!+\!ut\!+\!rv\!+\!b$, 令

$$f(m,u,r,b)=\dfrac 12\sum\limits_{i=1}^5[y_i\!-\!(mx_i\!+\!ut_i\!+\!rv_i\!+\!b)]^2,$$

1. 起始點 $(m_0,u_0,v_0,b_0)=(0,0,0,0)$。
2. 機器學習率 $\eta=0.01$。
3. $(m_{i+1},u_{i+1},r_{i+1},b_{i+1})=(m_i,u_i,r_i,b_i)-\eta\nabla f(m,u,r,b)\Big|_{(m_i,u_i,r_i,b_i)}$

\begin{align*} &\hskip -30pt =(m_i,u_i,r_i,b_i)-\eta\Big(\frac{\partial f(m,u,r,b)}{\partial m}\Big|_{(m_i,u_i,r_i,b_i)},\frac{\partial f(m,u,r,b)}{\partial u}\Big|_{(m_i,u_i,r_i,b_i)},\\ &\frac{\partial f(m,u,r,b)}{\partial r}\Big|_{(m_i,u_i,r_i,b_i)},\frac{\partial f(m,u,r,b)}{\partial b}\Big|_{(m_i,u_i,r_i,b_i)}\Big),\\ \hbox{其中}\ &\frac{\partial f(m,u,r,b)}{\partial m}\Big|_{(m_i,u_i,r_ib_i)}=\sum_{i=1}^5[y_i-(mx_i+ut_i+rv_i+b)]\cdot(-x_i)\\ &\frac{\partial f(m,u,r,b)}{\partial u}\Big|_{(m_i,u_i,r_i,b_i)}=\sum_{i=1}^5[y_i-(mx_i+ut_i+rv_i+b)]\cdot(-t_i)\\ &\frac{\partial f(m,u,r,b)}{\partial r}\Big|_{(m_i,u_i,r_i,b_i)}=\sum_{i=1}^5[y_i-(mx_i+ut_i+rv_i+b)]\cdot(-v_i)\\ &\frac{\partial f(m,u,r,b)}{\partial b}\Big|_{(m_i,u_i,r_i,b_i)}=\sum_{i=1}^5[y_i-(mx_i+ut_i+rv_i+b)]\cdot(-1). \end{align*}

4. 將 5 組資料

\begin{align*} (x_1,t_1,v_1,y_1)=&(2600,3,20,550000),\ &(x_2,t_2,v_2,y_2)=&(3000,4,15,565000),\\ (x_3,t_3,v_3,y_3)=&(3200,3,18,610000),\ &(x_4,t_4,v_4,y_4)=&(3600,3,30,595000),\\ (x_5,t_5,v_5,y_5)=&(4000,5,8,760000),&\hbox{標準化。}& \end{align*}

5. 疊代 90000 次即 $i\!=\!0,1,2,\ldots,89999$, 得到 $m=0.883772$, $u=-0.277378$, $r=-0.651639$, $b=0.000000$, 因此標準化後近似迴歸方程式為

$$y_{std}=0.883772x_{std}-0.277378t_{std}- 0.651639v_{std}+0.000000.$$

最終回歸參數 (原始尺度), 得到近似迴歸方程式為

$$y=137.25x-26025t-6825v+383725.$$

結論

1. 當 $(x_i,y_i)$, $i=1,2,\ldots,n$ 迴歸直線為 $y=mx+b$ 其中

$$m=\dfrac{\sum\limits_{i=1}^n x_iy_i- n\bar x\cdot\bar y}{\sum\limits_{i=1}^n x_i^2-{n}(\bar x)^2},\qquad b=\bar y=m\bar x.$$

2. $(x_1,y_1)=(1,5)$, $(x_2,y_2)=(2,6)$, $(x_3,y_3)=(3,10)$, $(x_4,y_4)=(4,13)$, $(x_5,y_5)=(5,11)$, 利用公式解, 求得迴歸直線為 $y=1.9x+3.3$。

3. $(x_1,y_1)=(1,5)$, $(x_2,y_2)=(2,6)$, $(x_3,y_3)=(3,10)$, $(x_4,y_4)=(4,13)$, $(x_5,y_5)=(5,11)$, 利用 gradient descent 方法, 起始點 $(m_0,b_0)=(0,0)$, 機器學習率 $\eta=0.001$, $(m_{i+1},b_{i+1})=(m_i,b_i)-\eta\nabla f(m_i,b_i)$, 疊代 10000 次, 求得近似迴歸直線為 $y=m_{10000}x+b_{10000}$ 即 $y=1.900152x+3.299452$。

4. $(x_1,y_1)=(1,5)$, $(x_2,y_2)=(2,6)$, $(x_3,y_3)=(3,10)$, $(x_4,y_4)=(4,13)$, $(x_5,y_5)=(5,11)$, 利用 gradient descent 方法, 起始點 $(m_0,b_0)=(0,0)$, 機器學習率 $\eta=0.001$, $(m_{i+1},b_{i+1})=(m_i,b_i)-\eta\nabla f(m_i,b_i)\Big|_{m_i,b_i}$, 為了求更接近迴歸直線, 我們疊代 20000 次, 有效位數小數點後 6 位, 求得近似迴歸直線為 $y=m_{20000}x+b_{20000}=1.900000x+3.300000$ 與公式解求得迴歸直線 $y=1.9x+3.3$ 相等。

5. 已知當 $(x_1,t_1,y_1)=(57,8,64)$, $(x_2,t_2,y_2)=(59,10,71)$, $(x_3,t_3,y_3)=(49,6,53)$, $(x_4,t_4,y_4)\!=\!(62,11,67)$, $(x_5,t_5,y_5)\!=\!(51,8,55)$, $(x_6,t_6,y_6)\!=\!(50,7,58)$, $(x_7,t_7,y_7)$ $=(55,10,77)$, $(x_8,t_8,y_8)=(48,9,57)$, 求得迴歸方程式為 $y\!=\!mx\!+\!ut\!+\!b=\dfrac{105}{187}x\!+\!\dfrac{1445}{561}t\!+\!\dfrac{1923}{187}$, 利用 gradient descent 方法, 求得近似迴歸方程式為 $y=m_{200000000}x+u_{200000000}t+b_{200000000}$, 即 $y=0.56150x+2.57575t+10.28327$ 其中 $m=\dfrac{105}{187}\approx 0.561497326$, $u=\dfrac{1445}{561}\approx 2.575757576$, $b=\dfrac{1923}{187}\approx 10.283422459$.

6. 給 5 組資料 $(x_1,t_1,v_1,y_1)\!\!=\!\!(2600,3,20,550000),(x_2,t_2,v_2,y_2)\!\!=\!\!(3000,4,25,565000)$, $(x_3,t_3,v_3,y_3)=(3200,3,28,610000)$, $(x_4,t_4,v_4,y_4)=(3600,3,30,595000)$,\\ $(x_5,t_5,v_5,y_5)=(4000,5,8,760000)$, 求得迴歸方程式 $y\!=\!mx\!+\!ut\!+\!rv\!+\!b=137.25x-26025t-6825v+383725$ 並預測 2 組房價。 同時利用 gradient descent 方法, 標準化 5 組資料後求得近似迴歸方程式為

$$y_{std}=0.883772x_{std}-0.277378t_{std}- 0.651639v_{std}+0.000000.$$

最終回歸參數(原始尺度), 得到近似迴歸方程式為

$$y=137.25x-26025t-6825v+383725.$$

參考文獻

本文作者投稿時為國立宜蘭大學人工智慧研究所二年級學生