1.1.1 인공 지능

 인공 지능은 1950년대에 초기 컴퓨터 과학 분양의 리부 선각자들이 "컴퓨터가 '생각'할 수 있는가?" 라는 질문을 하면서 시작되었습니다. 이 질문의 답은 오늘날에도 여전히 찾고 있습니다. 이 분야에 대한 간결한 정의는 다음과 같습니다. 보통의 사람이 수행하는 지능적인 작업을 자동화하기 위한 연구 활동 입니다. 이처럼 AI 머신러닝과 딥러닝을 포괄하는 종합적인 분야입니다. 또 학습 과정이 전혀 없는 다른 방법도 많이 포함하고 있습니다. 예를 들어 초기 체스 프로그램은 프로그래머가 만든 하드코딩된 규칙만 가지고 있었고 머신 러닝으로 인정받지 못했습니다. 아주 오랜기간 동안 많은 전문가는 프로그래머들이 명시적인 규칙을 충분하게 많이 만들어 지식을 다루면 인간 수준의 인공 지능을 만들 수 있다고 믿었습니다. 이런 접근 방법을 심볼릭 AI(symbolic AI)라고 하며 1950년대부터 1980년대까지  AI 분야의 지배적인 패러다임이었습니다. 1980년대 전문가 시스템(expert system) 의 호황으로 그 인기가 절정에 다다랐습니다.

심볼릭AI가 체스 게임처럼 잘 정의된 논리적인 문제를 푸는데 적합하다는 것이 증명되었지만, 이미지 분류, 음성 인식, 언어 번역 같은 더 복잡하고 불분명한 문제를 해결하기 위한 명확한 규칙을 찾는 것은 아주 어려운 일입니다. 이런 심볼릭 AI를 대체하기 위한 새로운 방법이 등장했는데, 바로 머신 러닝입니다.

2.3 Linear Model in Action

 Let's actually train a single-input linear neuron model using the gradient descent algorithm. First, we need to sample multiple data points. For a toy example with a known model, we directly sample from the specified real model:

y = 1.477x  + 0.0089

01. Sampling data

In order to simulate the observation errors, we add an independent error variable e to the model, where e follows a Gaussian distribution with a mean value of 0 and a standard deviation of 0.01(i.e., variance of 0.01):

y = 1.477x + 0.089 + e, e ~ N(0.01)

2.2 Optimization Method

 Now let's summarize the preceding solution: we need to find the optimal parameters w and b, so that the input and output meet a linear relationship y = wx + b, i ∈ [1,n]. However, due to the existence of observation errors e, it is necessary to sample a data set D = {(x(1),y(1)),(x(2),y(2)), x(3),y(3)...,(x(n),y(n))}, composed of a sufficient number of data samples, to find an optimal set of parameters w and b to minimize the mean squared error L = 1/n(wx(i) + b - y(i))2.

For a single-input neuron model, only two samples are needed to obtain the exact solution of the equations by the elimination method. This exact solution derived by a strict formula is called an analytical solution. However, in the case of multiple data points (n 2), there is probably no analytical solution. We can only use numerical optimization methods to obtain an approximate nuimerical solution. Why is it called optimization? This is because the computer's calculation speed is very fast. We can use the powerful computing power to "search" and "try" multiple times, thereby reducing the error L step by step. The simplest optimization method is brute-force search or random experiment. For example, to find the most suitable w and b, we can randomly sample any w and b from the real number space and calculate the error value L of the corresponding model. Pick out the semallest error L from all the experiments {L}, and its corresponding, w and b are the optimal parameters we are looking for.

This brute-force algorithm is simple and straightforward, but it is extremely inefficient for large-scale, high-dimensional optimization problems. Gradient descent is the most commonly used optimization algorithm in neural network training. With the parallel acceleration capability of powerful graphics processing unit(GPU) chips, it is very suitable for optimizing neural network models with massive data.

Naturaaly it is also suitable for optimizing our simple linear neuron model. Since the gradient descent algorithm is the core algorithm of deep learning, we will first apply the gradient descent algorithm to solve simple nueuron models and then detail ists application in neural network in Chapter 7.

With the concept of derivative, if we want to solve the maximum and minimum values of a function, we can simply set the derivative function to be 0 and find the corresponding independent variable a values, that is, the stagnation point, and then check the stagnation type. Taking the function f(x) = x2.sin(x) as an example, we can plot the functjion and its derivative in the interval x  |-10, 10|, where the blue solid line is f(x) and the yellow dotted line is df(x)/dx as shown in Figure 2-5. It can be seen that the points where the derivative (dashed line) is 0 are the stagnation points, and both the maximum and minimum values of f(x) appear in the stagnation points.

The gradient of a function is defined as a vector of partial derivatives of the function on each independent variable. Considering a three-dimensional function z = f(x,y), the partial derivative of the function with respect to the independent variable x is dz/dx, the partial derivative of the function with respect to the independent variable y is recorded as dz/dy, and the gradient ∇f is a vector (dz/dx, dz/dy). Let's look at a specific function f(x,y) = -(cos2x + cos2y)2. As shown in Figure 2-6, the length of the red arrow in the plane represents the modulus of the gradient vector, and the direction of the arrow represents the direction of the gradient vector. It can be seen that the direction of the arrow always points to the function value increasing direction. The steeper the function surface, the longer the length of the arrow, and the larger the modulus of the gradient.

Through the preceding example, we can intuitively feel that the gradient direction of th efunction always points to the direction in which the function value increases. Then the opposite direction of the gradient should point to the direction in which the function value decreases.

To take advantage of this property, we just need to follow the preceding equation to iteratively update x. Then we can get smaller and smaller function values. n is used to scale the gradient vector, which is known as learning rate and generally set to a smaller value, such as 0.01 or 0.001. In particular, for one-dimensional functions, the preceding vector form can be written into a scalar form:

x' = x -n.dy/dx

By iterating and updating x several times through the preceding formula, the function value y' at x' is always more likely to be smaller than the function value at x.

The method of optimizing parameters by the formula(2.1) is called the gradient descent algorithm. It calculates the gradient f of the function f and iteratively updates the parameters to obtain the optimal numberical solution of the parameters when the function f reaches its minimum value. It should be noted that model input in deep learning is generally represented as x and the parameters to be optimized are generally represented by 0,w, and b.

Now we will apply the gradient descent algorithm to calculate the optimal parameters w' and b in the beginning of this session. Here the mean squared error function is minimized:

The model parameters that need to be optimized are w and b, so we update them iteratively using the following equations:

x

2.1 Neuraon Model

 An adult brain contains about 100 billion neuraons. Each neuraon obtains input signals through dendrites and transmits output signals through axons. The neurons are interconnected to form a huge neural network, thus forming the human brain, the basis of perception and consciousness. Figure 2-1 is a typical biological neuron structure. In 1943, the psychologist Warren McCulloch and mathematical logician Walter Pitts proposed a mathematical model of artificial neural networks to simulate the mechanism of biological neuraons. This research was further developed by the American neurologist Frank Rosenblatt into the perceptron model, which is also the cornerstone of modern deep learning.

Starting from the structure of biological neurons, we will revisit the exploration of scientific pioneers and gradually unveil the mystery of automatic learning machines.

First, we can abstract the neuron model into the mathematical structure as shown in Figure 2-2. The neuron input vector x = [x1, x2, x3,...xn]T maps to y through function f:x->y, where θ represents the parameters in the function f. Consider a simplified case, such as linear transformation: f(x) = wtx + b. The expanded form is

f(x) = w1x1 + w2x2 +.... +wnxn +b

The preceding calculation logic can be intuitively shown in Figure 2-2.

The parameters θ = {w1, w2, w3,...,wn,b} determine the state of the neuron, and the processing logic of this neuron can be determined by fixing those parameters. When the number of input nodes n = 1 (single input), the neuron model can be further simplified as 

y = ws +b

 Then we can plot the change of y as a function of x as shown in Figure 2-3. As the input signal x increases, the output also increases linearly. Here parameter w can be understood as the slope of the straight line, and b is the bias of the straight line.

For a certain neuron, the mapping relationship f between x and y is unknown but fixed. Two pints can determine a straight line. In order to estimate the value of w and b, we only need to sample any two data points(x(1), y(1)) and (x(2), y(2)) from the straight line in Figure 2-3, where the superscript indicates the data point number:

y(1) = wx(1) +b

y(2) = wx(2) + b

If(x(1), y(1))  (x(2), y(2)), we can solve the preceding equations to get the value of w and b. Let's consider a specific example: x(1) = 1, y(1) = 1.567, x(2) = 2, y(2) = 3.043. Substituting the numbers in the preceding formulas gives

1.567 = w.1 + b

3.043 = w.2 + b

This is the system of binary linear equations that we learned in junior or high school. The analytical solution can be easily calculated using the elimination method, that is, w = 1.477, b=0.089.

You can see that we only need two different data points to perfectly solve the parameters of a single-input lineary neuron model. For linear neuron models with N input, we only need to sample N + 1 different data points. It seems thjat the linear neuron models can be perfectly resolved. So what's wrong with the preceding method? Considering that there may be observation errors for any sampling point, we assume that the observation error variable e follows a normal distribution N(μ, σ2) with μ as mean and σ2 as variance. Then the samples follow:

y = wx + b + e, e - N(μ, σ2)

Once the observation error is introduced, event if it is as simple as a linear model, if only two data  ppoints are smapled, it may bring a large estimation bias. As shown in Figure 2-4, the data points all have observation errors. IF the estimatino is based on the two blue rectangular data points, the estimatied blue dotted line woould have a large deviation from the true orange straight line. In order to reduce the estimation bias introduced by observation errors, we can sample multiple data points D = {(x(1), y(1)), (x(2),y(2)), (x(3),y(3))...,(x(n),y(n))} and then find a "best" straight line, so that it minimizes the sum of errors between all sampling points and the straight line.

Due to the existence of observation errors, there may not be a straight line that perfectly passes through all the sampling points D. Therefore, we hope to find a "good" straight line close to all sampling points. How to measure "good" and "bad"? A natural idea is to use the mean squared error (MSE) between the predicted vaslue wx(i) + b and the true value y(i) at all sampling points as the total error, that is

Then search a set of parameters w and b to minimize the total error L. The straight line corresponding to the minimal total error is the optimal straight line we are looking for, that is

Here n represents the number of sampling points.

CHATER 2 Regression

 Some people worry that artificaial intelligence will make us feel inferior, but then, anybody in his right mind should have an inferiority complex every time he looks at a flower. -Alan Kay

1.6.4 Common Editor Installation

 There are many ways to write programs in Python. You can use IPython or Jupyter Notebook to write code interactively. You can also use Sublime Text, PyCharm, and VS Code to develop medium and large projects. This book recommends using PyCharm to write and debug code and using VSCode for interactive project development. Both of them are free. Users can download and install them by themselves.

Next, let's start the deep learning journey!

1.6.3 TensorFlow Installation

 TensorFlow, like other Python libraries, can be installed using the Python package management tool "pip install" command. When installing TensorFlow, you need to determine whether to install a more powerful GPU version or a general-performance CPU version based on whether your omputer has an NVIDA GPU graphics card

# Install numpy

pip install numpy

With the preceding command, you should be able to automatically download and install the numpy library. Now let's install the latest GPU verison of TensorFlow. The command is as follows:

# Install TensorFlow GPU version

pip install -U tensorflow

The preceding command should automatically download and install the TensorFlow GPU version, which is currently the official version of TensorFlow 2.x. The "-U" parameter secifies that if this package is installed, the upgrade command is executed.

Now let's test whether the GPU version of TensorFlow is successfully installed. Enter "ipython" on the "cmd" command line to enter the ipython interactive terminal, and thenm enter the "import tensorflow as tf" command. If no errors occur, continue to enter "tf.test.is_gpu_available()" to test whether the GPU is available. This command will print a series of information. The information beginning with "I"(Information) contains information about the available GPU graphics devices and will return "True" or "False" at the end, indicating whether the GPU device is available, as shown in Figure 1-35. If True, the TensorFlow GPU version is successfully installed; if False, the installation fails.

You may need to check the steps of CUDA, cuDNN, and environment variable configuration again or copy the error and seek help from the search engine.

If you don't have GPU, you can install the CPU version. The CPU version cannot use the GPU to accelerate calculations, and the conputational seed is relatively slow. However, because the models introduced as learning purposes in this book are generally not omputationally expensive, the CPU version can also be used. If it also possible to add the NVIDA GPU device after having better understanding of deep learning in the future. If the installation of the TensorFlow GPU version fails, we can also use the CPU version directly. The command to install the CPU version is

# Install TensorFlow CPU version

pip install -U tensorflow-cpu

After installation, enter the "import tensorflow as tf" command in the ipython terminal to verify that the CPU version is successfully installed. Afeter TensorFlow is installed, you can view the version number through "tf._version_". Figure 1-36 shows an example. Note that even the code works for all TensorFlow 2.x versions.

The preceding manaual process of installing CUDA and cuDNN, configuring the Path environment variable, and installing TensorFlow is the standard installation method. Although the steps are tedious, it is of great help to understand the functional role of each library. In fact, for the novice, you can complete the preceding steps by two commands as follows:

# Create virtual environment tf2 with tensorflow-gpu setup required

# to automatically install CUDA, cuDNN, and TensorFlow GPU

conda create -n tf2 tensorflow-gpu

#Activate tf2 environment

conda activate tf2

This quick installation method is called the minimal installation method. This is also the convenience of using the Anaconda distribution.

TensorFlow installed though the minimal version requires activation of the corresponding vertual environment before use, which needs to be distinguished from the standard version. The standard version is installed in Anaconda's default environment base and generally does not require manual activation of the base environment.

Common Python libraries can also be installed by default. The command is as follows:

# Install common python libraries

pip install -U ipython numpy matplotilib pillow pandas

When TensorFlow is running, it will consume all GPU resources by default, which is very computationally unfiendly, especially when the computer has multiple users or programs using GPU resources at the same time. Occuping all GOU resources will make other programs unable to run. Therefore, it is generally recommeded to set the GPU memory usage of TensorFlow to the growth mode, that is, to apply for GPU memory resources based on the actual model size. The code implementation is as follows:

# Set GPU resource usage method

# Get GPU device list

gpus = tf.config.experimental.list_physical_devices('GPU')

if gpus:

    try:

        # Set GPU usage to growth mode

        for gpu in gpus:

            tf.config.experimental.set_memory_growth(gpu, True)

    except RuntimeError as e:

        # print error

        print(e)