1.1.2 머신 러닝

 영국 빅토리아 시대의 에이다 러브레이스(Ada Lovelace)는 최초의 기계적 범용 컴퓨터인 해석기관(Analytical Engine)을 발명한 찰스 배비지(Charles Babbage)의 친구이자 동료였습니다. 선견지명이 있어 시대를 많이 앞섰지만 1830년대와 1840년대부터 해석 기관이 범용 컴퓨터란 개념이 아직 정의됮 ㅣ않은 때였습니다. 단지 해석학(mathematical analysis) 분야의 계산을 자동화하기 위해 기계적인 연산을 사용하는 방법이었을 뿐입니다. 그래서 이름이 해석 기관입니다. 1843년 에이다 러브레이스는 이 발명에 대해 다음과 같이 언급했습니다. "해석 기관이 무언가를 새롭게 고안해 내는것은 아닙니다. 우리가 어떤 것을 작동시키기 위해 어떻게 명ㄱ령할지 알고 있다면 이 장치는 무엇이든 할 수 있습니다. ... 이런 능력은 우리가 이미 알고 있는것을 유용하게 사용할 수 있도록 도와줄 것입니다." AI 선구자인 앨런 튜링(Alan Turing)은 1950년에 튜링 테스트(Turing test)와 AI의 주요 개념을 소개한 그의 기념비적인 노문 "Computing Machinery and Intelligence"에서 '러브레이스의 반론(Lady Lovelace's objection)으로 이 논평을 인용했습니다. 튜링은 에이다 러브레이스의 말을 인용했지만 범용 컴퓨터가 학습과 창의력을 가질 수 있는지 숙고한 후 가능한 일이라고 결론을 냈습니다.

머신 러닝은 이런 질문에서부터 시작됩니다. "우리가 어떤 것을 작동히키기 위해 '어떻게 명령할 지 알고 있는것' 이상을 컴퓨터가 처리하는 것이 가능한가? 그리고 특정 작업을 수행하는 법을 스스로 학습할 수 있는가? 컴퓨터가 우리를 놀라게 할 수 있을까? 프로그래머가 직접 만든 데이터 처리 규칙 대신 컴퓨터가 데이터를 보고 자동으로 이런 규칙을 학습할 수 있을까?

이 질문은 새로운 프로그래밍 패러다임의 장을 열었습니다. 전통적인 프로그래밍인 심볼릭 AI의 패러다임에서는 규칙(프로그램)과 이 규칙에 따라 처리될 데이터를 입력하면 해답이 출력됩니다. 머신 러닝에서는 데이터와 이 데이터로 부터 기대되는 해답을 입력하면 규칙이 출력됩니다. 이 규칙을 새로운 데이터에 적용하여 창의적인 답을 만들 수 있습니다.

머신러닝 시스템은 명시적으로 프로그램되는 것이 아니라 훈련(training)됩니다. 작업과 관련 있는 많은 샘플을 제공하면 이 데이터에서 통계적 구조를 찾아 그 작업을 자동화하기 위한 규칙을 만들어 냅니다. 예를 들어 여행 사진을 태깅하는 일을 자동화하고 싶다면, 사람이 이미 태그해 놓은 다수의 사진 샘플을 시스템에 제공해서 특정 사진에 태그를 연과시키기 위한 통계적 규칙을 학습할 수 있을 것 입니다.

먼시 ㄴ러닝은 1990년대 들어와서야 각광을 받기 시작했지만, 고성능 하드웨어와 대량으 ㅣ데이터셋이 가능해지면서 금방 AI에서 가장 인기 있고 성공적인 분야가 되었습니ㅏㄷ. 머신 러닝은 수리 통계와 밀접하게 관련되어 있지만 통계와 다른점이 몇가지 있습니다. 먼저 머신러닝은 통계와 달리 보통 대량의 복잡한 데이터셋(예를 들어 몇 만개의 픽셀로 구성된 이미지가 수백만 개가 있는 데이터셋)을 다루기 때문에 베이지안 분석(Bayesian analysis) 같은 전통적인 통계 분석 방법은 현실적으로 적용하기 힘듭니다. 이런 이유 때문에 머신 러닝, 특히 딥러닝은 수학적 이론이 비교적 부족하고(어쩌면 아주 부족하고) 엔지니어링 지향적입니다. 이런 실천적인 접근 방식 때문에 이론보다는 경험을 바탕으로 아이디어가 증명되는 경우가 많습니다.

1.1.1 인공 지능

 인공 지능은 1950년대에 초기 컴퓨터 과학 분양의 리부 선각자들이 "컴퓨터가 '생각'할 수 있는가?" 라는 질문을 하면서 시작되었습니다. 이 질문의 답은 오늘날에도 여전히 찾고 있습니다. 이 분야에 대한 간결한 정의는 다음과 같습니다. 보통의 사람이 수행하는 지능적인 작업을 자동화하기 위한 연구 활동 입니다. 이처럼 AI 머신러닝과 딥러닝을 포괄하는 종합적인 분야입니다. 또 학습 과정이 전혀 없는 다른 방법도 많이 포함하고 있습니다. 예를 들어 초기 체스 프로그램은 프로그래머가 만든 하드코딩된 규칙만 가지고 있었고 머신 러닝으로 인정받지 못했습니다. 아주 오랜기간 동안 많은 전문가는 프로그래머들이 명시적인 규칙을 충분하게 많이 만들어 지식을 다루면 인간 수준의 인공 지능을 만들 수 있다고 믿었습니다. 이런 접근 방법을 심볼릭 AI(symbolic AI)라고 하며 1950년대부터 1980년대까지  AI 분야의 지배적인 패러다임이었습니다. 1980년대 전문가 시스템(expert system) 의 호황으로 그 인기가 절정에 다다랐습니다.

심볼릭AI가 체스 게임처럼 잘 정의된 논리적인 문제를 푸는데 적합하다는 것이 증명되었지만, 이미지 분류, 음성 인식, 언어 번역 같은 더 복잡하고 불분명한 문제를 해결하기 위한 명확한 규칙을 찾는 것은 아주 어려운 일입니다. 이런 심볼릭 AI를 대체하기 위한 새로운 방법이 등장했는데, 바로 머신 러닝입니다.

2.3 Linear Model in Action

 Let's actually train a single-input linear neuron model using the gradient descent algorithm. First, we need to sample multiple data points. For a toy example with a known model, we directly sample from the specified real model:

y = 1.477x  + 0.0089

01. Sampling data

In order to simulate the observation errors, we add an independent error variable e to the model, where e follows a Gaussian distribution with a mean value of 0 and a standard deviation of 0.01(i.e., variance of 0.01):

y = 1.477x + 0.089 + e, e ~ N(0.01)

2.2 Optimization Method

 Now let's summarize the preceding solution: we need to find the optimal parameters w and b, so that the input and output meet a linear relationship y = wx + b, i ∈ [1,n]. However, due to the existence of observation errors e, it is necessary to sample a data set D = {(x(1),y(1)),(x(2),y(2)), x(3),y(3)...,(x(n),y(n))}, composed of a sufficient number of data samples, to find an optimal set of parameters w and b to minimize the mean squared error L = 1/n(wx(i) + b - y(i))2.

For a single-input neuron model, only two samples are needed to obtain the exact solution of the equations by the elimination method. This exact solution derived by a strict formula is called an analytical solution. However, in the case of multiple data points (n 2), there is probably no analytical solution. We can only use numerical optimization methods to obtain an approximate nuimerical solution. Why is it called optimization? This is because the computer's calculation speed is very fast. We can use the powerful computing power to "search" and "try" multiple times, thereby reducing the error L step by step. The simplest optimization method is brute-force search or random experiment. For example, to find the most suitable w and b, we can randomly sample any w and b from the real number space and calculate the error value L of the corresponding model. Pick out the semallest error L from all the experiments {L}, and its corresponding, w and b are the optimal parameters we are looking for.

This brute-force algorithm is simple and straightforward, but it is extremely inefficient for large-scale, high-dimensional optimization problems. Gradient descent is the most commonly used optimization algorithm in neural network training. With the parallel acceleration capability of powerful graphics processing unit(GPU) chips, it is very suitable for optimizing neural network models with massive data.

Naturaaly it is also suitable for optimizing our simple linear neuron model. Since the gradient descent algorithm is the core algorithm of deep learning, we will first apply the gradient descent algorithm to solve simple nueuron models and then detail ists application in neural network in Chapter 7.

With the concept of derivative, if we want to solve the maximum and minimum values of a function, we can simply set the derivative function to be 0 and find the corresponding independent variable a values, that is, the stagnation point, and then check the stagnation type. Taking the function f(x) = x2.sin(x) as an example, we can plot the functjion and its derivative in the interval x  |-10, 10|, where the blue solid line is f(x) and the yellow dotted line is df(x)/dx as shown in Figure 2-5. It can be seen that the points where the derivative (dashed line) is 0 are the stagnation points, and both the maximum and minimum values of f(x) appear in the stagnation points.

The gradient of a function is defined as a vector of partial derivatives of the function on each independent variable. Considering a three-dimensional function z = f(x,y), the partial derivative of the function with respect to the independent variable x is dz/dx, the partial derivative of the function with respect to the independent variable y is recorded as dz/dy, and the gradient ∇f is a vector (dz/dx, dz/dy). Let's look at a specific function f(x,y) = -(cos2x + cos2y)2. As shown in Figure 2-6, the length of the red arrow in the plane represents the modulus of the gradient vector, and the direction of the arrow represents the direction of the gradient vector. It can be seen that the direction of the arrow always points to the function value increasing direction. The steeper the function surface, the longer the length of the arrow, and the larger the modulus of the gradient.

Through the preceding example, we can intuitively feel that the gradient direction of th efunction always points to the direction in which the function value increases. Then the opposite direction of the gradient should point to the direction in which the function value decreases.

To take advantage of this property, we just need to follow the preceding equation to iteratively update x. Then we can get smaller and smaller function values. n is used to scale the gradient vector, which is known as learning rate and generally set to a smaller value, such as 0.01 or 0.001. In particular, for one-dimensional functions, the preceding vector form can be written into a scalar form:

x' = x -n.dy/dx

By iterating and updating x several times through the preceding formula, the function value y' at x' is always more likely to be smaller than the function value at x.

The method of optimizing parameters by the formula(2.1) is called the gradient descent algorithm. It calculates the gradient f of the function f and iteratively updates the parameters to obtain the optimal numberical solution of the parameters when the function f reaches its minimum value. It should be noted that model input in deep learning is generally represented as x and the parameters to be optimized are generally represented by 0,w, and b.

Now we will apply the gradient descent algorithm to calculate the optimal parameters w' and b in the beginning of this session. Here the mean squared error function is minimized:

The model parameters that need to be optimized are w and b, so we update them iteratively using the following equations:

x

2.1 Neuraon Model

 An adult brain contains about 100 billion neuraons. Each neuraon obtains input signals through dendrites and transmits output signals through axons. The neurons are interconnected to form a huge neural network, thus forming the human brain, the basis of perception and consciousness. Figure 2-1 is a typical biological neuron structure. In 1943, the psychologist Warren McCulloch and mathematical logician Walter Pitts proposed a mathematical model of artificial neural networks to simulate the mechanism of biological neuraons. This research was further developed by the American neurologist Frank Rosenblatt into the perceptron model, which is also the cornerstone of modern deep learning.

Starting from the structure of biological neurons, we will revisit the exploration of scientific pioneers and gradually unveil the mystery of automatic learning machines.

First, we can abstract the neuron model into the mathematical structure as shown in Figure 2-2. The neuron input vector x = [x1, x2, x3,...xn]T maps to y through function f:x->y, where θ represents the parameters in the function f. Consider a simplified case, such as linear transformation: f(x) = wtx + b. The expanded form is

f(x) = w1x1 + w2x2 +.... +wnxn +b

The preceding calculation logic can be intuitively shown in Figure 2-2.

The parameters θ = {w1, w2, w3,...,wn,b} determine the state of the neuron, and the processing logic of this neuron can be determined by fixing those parameters. When the number of input nodes n = 1 (single input), the neuron model can be further simplified as 

y = ws +b

 Then we can plot the change of y as a function of x as shown in Figure 2-3. As the input signal x increases, the output also increases linearly. Here parameter w can be understood as the slope of the straight line, and b is the bias of the straight line.

For a certain neuron, the mapping relationship f between x and y is unknown but fixed. Two pints can determine a straight line. In order to estimate the value of w and b, we only need to sample any two data points(x(1), y(1)) and (x(2), y(2)) from the straight line in Figure 2-3, where the superscript indicates the data point number:

y(1) = wx(1) +b

y(2) = wx(2) + b

If(x(1), y(1))  (x(2), y(2)), we can solve the preceding equations to get the value of w and b. Let's consider a specific example: x(1) = 1, y(1) = 1.567, x(2) = 2, y(2) = 3.043. Substituting the numbers in the preceding formulas gives

1.567 = w.1 + b

3.043 = w.2 + b

This is the system of binary linear equations that we learned in junior or high school. The analytical solution can be easily calculated using the elimination method, that is, w = 1.477, b=0.089.

You can see that we only need two different data points to perfectly solve the parameters of a single-input lineary neuron model. For linear neuron models with N input, we only need to sample N + 1 different data points. It seems thjat the linear neuron models can be perfectly resolved. So what's wrong with the preceding method? Considering that there may be observation errors for any sampling point, we assume that the observation error variable e follows a normal distribution N(μ, σ2) with μ as mean and σ2 as variance. Then the samples follow:

y = wx + b + e, e - N(μ, σ2)

Once the observation error is introduced, event if it is as simple as a linear model, if only two data  ppoints are smapled, it may bring a large estimation bias. As shown in Figure 2-4, the data points all have observation errors. IF the estimatino is based on the two blue rectangular data points, the estimatied blue dotted line woould have a large deviation from the true orange straight line. In order to reduce the estimation bias introduced by observation errors, we can sample multiple data points D = {(x(1), y(1)), (x(2),y(2)), (x(3),y(3))...,(x(n),y(n))} and then find a "best" straight line, so that it minimizes the sum of errors between all sampling points and the straight line.

Due to the existence of observation errors, there may not be a straight line that perfectly passes through all the sampling points D. Therefore, we hope to find a "good" straight line close to all sampling points. How to measure "good" and "bad"? A natural idea is to use the mean squared error (MSE) between the predicted vaslue wx(i) + b and the true value y(i) at all sampling points as the total error, that is

Then search a set of parameters w and b to minimize the total error L. The straight line corresponding to the minimal total error is the optimal straight line we are looking for, that is

Here n represents the number of sampling points.

CHATER 2 Regression

 Some people worry that artificaial intelligence will make us feel inferior, but then, anybody in his right mind should have an inferiority complex every time he looks at a flower. -Alan Kay

1.6.4 Common Editor Installation

 There are many ways to write programs in Python. You can use IPython or Jupyter Notebook to write code interactively. You can also use Sublime Text, PyCharm, and VS Code to develop medium and large projects. This book recommends using PyCharm to write and debug code and using VSCode for interactive project development. Both of them are free. Users can download and install them by themselves.

Next, let's start the deep learning journey!