1.6.3 TensorFlow Installation

 TensorFlow, like other Python libraries, can be installed using the Python package management tool "pip install" command. When installing TensorFlow, you need to determine whether to install a more powerful GPU version or a general-performance CPU version based on whether your omputer has an NVIDA GPU graphics card

# Install numpy

pip install numpy

With the preceding command, you should be able to automatically download and install the numpy library. Now let's install the latest GPU verison of TensorFlow. The command is as follows:

# Install TensorFlow GPU version

pip install -U tensorflow

The preceding command should automatically download and install the TensorFlow GPU version, which is currently the official version of TensorFlow 2.x. The "-U" parameter secifies that if this package is installed, the upgrade command is executed.

Now let's test whether the GPU version of TensorFlow is successfully installed. Enter "ipython" on the "cmd" command line to enter the ipython interactive terminal, and thenm enter the "import tensorflow as tf" command. If no errors occur, continue to enter "tf.test.is_gpu_available()" to test whether the GPU is available. This command will print a series of information. The information beginning with "I"(Information) contains information about the available GPU graphics devices and will return "True" or "False" at the end, indicating whether the GPU device is available, as shown in Figure 1-35. If True, the TensorFlow GPU version is successfully installed; if False, the installation fails.

You may need to check the steps of CUDA, cuDNN, and environment variable configuration again or copy the error and seek help from the search engine.

If you don't have GPU, you can install the CPU version. The CPU version cannot use the GPU to accelerate calculations, and the conputational seed is relatively slow. However, because the models introduced as learning purposes in this book are generally not omputationally expensive, the CPU version can also be used. If it also possible to add the NVIDA GPU device after having better understanding of deep learning in the future. If the installation of the TensorFlow GPU version fails, we can also use the CPU version directly. The command to install the CPU version is

# Install TensorFlow CPU version

pip install -U tensorflow-cpu

After installation, enter the "import tensorflow as tf" command in the ipython terminal to verify that the CPU version is successfully installed. Afeter TensorFlow is installed, you can view the version number through "tf._version_". Figure 1-36 shows an example. Note that even the code works for all TensorFlow 2.x versions.

The preceding manaual process of installing CUDA and cuDNN, configuring the Path environment variable, and installing TensorFlow is the standard installation method. Although the steps are tedious, it is of great help to understand the functional role of each library. In fact, for the novice, you can complete the preceding steps by two commands as follows:

# Create virtual environment tf2 with tensorflow-gpu setup required

# to automatically install CUDA, cuDNN, and TensorFlow GPU

conda create -n tf2 tensorflow-gpu

#Activate tf2 environment

conda activate tf2

This quick installation method is called the minimal installation method. This is also the convenience of using the Anaconda distribution.

TensorFlow installed though the minimal version requires activation of the corresponding vertual environment before use, which needs to be distinguished from the standard version. The standard version is installed in Anaconda's default environment base and generally does not require manual activation of the base environment.

Common Python libraries can also be installed by default. The command is as follows:

# Install common python libraries

pip install -U ipython numpy matplotilib pillow pandas

When TensorFlow is running, it will consume all GPU resources by default, which is very computationally unfiendly, especially when the computer has multiple users or programs using GPU resources at the same time. Occuping all GOU resources will make other programs unable to run. Therefore, it is generally recommeded to set the GPU memory usage of TensorFlow to the growth mode, that is, to apply for GPU memory resources based on the actual model size. The code implementation is as follows:

# Set GPU resource usage method

# Get GPU device list

gpus = tf.config.experimental.list_physical_devices('GPU')

if gpus:

    try:

        # Set GPU usage to growth mode

        for gpu in gpus:

            tf.config.experimental.set_memory_growth(gpu, True)

    except RuntimeError as e:

        # print error

        print(e)

1.6.2 CUDA Installation

 Most of the current deep learning frameworks are based on NVIDIA's GPU graphics card for accelerated calculations, so you need to install the GPU acceleration library CUDA provided by NVIDIA. Before installing CUDA, make suer your computer has an NVIDIA graphics device that supports the CUDA program. If you computer does not have an NVIDIA grahics card-for example, some computer graphics card manufactures are AMD or Intel - the CUDA program won't work, and you can skip this step and directly install the TensorFlow CPU version.

The installation of CUDA is divided into three steps: CUDA software installation, cuDNN deep neural network acceleration library installation, and environment variable configuration. The installation process is a bit tedious. We will go through them step by step using the Windows 10 system as an example.

CUDA Sotfware Installation Open the official downloading website of the CUDA program: https://developer.nvidia.com/duca-10.0-download-archive. Here we use CUDA 10.0 version: select thw Windows platform, x86_64 architecture, 10 system, and exe (local) installation package and then select "Download" to download the CUDA installation software. After the download is cimplete, open the software. As shown in Figure 1-25, select the "Custom" option and click the "NEXT" button to enter the installation program selection list as shown in Figure 1-26. Here you can select the components that need to be installed and unselect those that do not need to be installed. Under the "CUDA" category, unselect the "Visual Studio Integration" item. Under the "Driver components" category compare the version number of "Current Version" is greater than "New Version," you need to uncheck the "Display Driver." If "Current Version is less than or equal to "New Version," leave "Display Driver." If Current Version" is less than or equal to "New Version," leave "Display Driver" checked, as shown in Figure 1-27. After the setup is complete, you can click "NEXT" and follow the instructions to install.

After the installation is complete, let's test whether the CUDA software is successfully installed. Open the "cmd" terminal and enter "nvcc-V" to print th ecurrent CUDA version information, as shown in Figure 1-28. If the command is not recongnized, the installation has filed. We can find the "nvcc.exe" program from the CUDA installation path "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\V10.0\bin", as shown in Figure 1-29.

cuDNN Neural Network Acceleration Library Installation. CUDA is not a special GPU acceleration library for neural networks; it is designed for a variety of applications that require paralled computing. If you want to accelerate for neural network applications, you need to install an additional cuDNN library. It should be noted that the cuDNN library is not an executable program. You only need to download and decompress the cuDNN file and configure the Path environment variable.

Open the website https://developer.nvidia.com/cudnn and select "Download cuDNN." Due to NVIDIA regulations, users need to log in or create a new user to continue downloading. After logging in, enter the cuDNN download interface and check "I Agree To the Terms of the cuDNN Software License Agreement," and the cuDNN version download option will pop up, Select the cuDNN version that matches CUDA 10.0, and click the "cuDNN Library for Windows 10" link to download the cuDNN file, as shown in Figure 1-30. It should be noted that cuDNN itself has a version number, and it also needs to match the CUDA version number.

After downloading the cuDNN file, unzip it and rename the folder "cuda" to "cudnn765". Then copy the "cudnn765" folder to the CUDA installation path "C\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0". A dialog box that requires adminstrator rights mayu pop up here. Select continue to paste.

Environment Variable Configuration. We have completed the installation of cuDNN, but in order for the system to be aware of the location of the cuDNN file, we need to configure the Path environment variable as follows. Open the file brower, right-click "My Computer," select "Properties," select "Advanced system settings," and select "Environment Variables," as shown in  Figure 1-32. Select the "Path" environment variable in the "System variables" column and select "Edit," as shown in Figure 1-33. Select "New," enter the cuDNN installation path "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\cudnn756\bin", and use the "Move up" button to move this item to the top.

After the CUDA installation is complete, the environment variables should include "C:\Program File\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin," "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\libnvvp", and "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\cudnn756\bin". The preceding path may differ slightly according to the actual path, as shown in Figure 1-34. After confirmation, click "OK" to close all dialog boxes.

1.6.1 Anaconda installation

 The Pythonb interpreter is the bridge that allows code written in Python to be executed by CPU and is the core software of the Python language. Users candownload the appropriate version(Python 3.7 is used here) of the interpreter form www.python.org/. After the installation is completed, you can call the python.exe program to execute the source code file written in Python(.py files).

Here we choose to install Anaconda software that integrates a series of auxiliary functions such as the Python interpreter, package management, and virtual environment. We can download Andacoda from www.anaconda.com/distribution/#download-section and select the latest version of Python to download and install. As shown in Figure 1-22, check the "Add Anaconda to my PATH environment variable" option, so that you can call the Anacondat program through the command line. As shown in Figure 1-23, the installer asks whether to install the VS code software together. Select Skip. The entire installation process lasts about 5 minutes, and the specific time depends on the computer performance.

After the installation is complete, how can we verify that Anaconda was successfully installed? Pressing the Windows+R key combination on the keyboard, you can bring up the running program dialog box, enter "cmd," and press Enter to open the command-line program "cmd.exe" that comes with Windows. Or click the Start menu and enter "cmd" to find the "cmd.exe" program and open it. Enter the "conda list" command to view the installed libraries in the Python environment. If it is a newly installed Python environment, the listed libraries are all libraries that come with Anaconda, as shown in Figure 1-24. If the "conda list" can pop up a series of library list information normally, the Anaconda software installation is successful. Otherwise, the installation failed, and you need to reinstall.

1.6 Development Environment Installation

 After knwing the convenience brought by the deep learning fraework, we are now ready to install the latest version of TensorFlow in the local desktopo. TensorFlow supports a variety of ocmmon operating systems, such as Windows 10, Ubuntu 18.04, and Mac OS.  It supports both GPU version running on NVIDIA GPU and CPU version that uses only the CPU to do calculations. We take the most common operating system, Windows 10, NVIDIA GPU,and Python as examples to introduce how to install the TensorFlow tramework and other development software.

Generally speaking, the development environment installation is divided into four major steps: th ePyuthon iterpreter Anaconda, the CUDA acceleration library, the TensorFlow framework, and commonly used editors.

1.5.3 Demo

 The core of deep learning is the design idea of algorithms, and deep learning frameworks are just our tools for implementing algorithms. In the following, we will demonstrate the three core functions of the TensorFlow deep learning framework to help us understand the role of frameworks in algorithm design.

a) Accelerated Calculation

The neural network is essentially comoposed of a large number of basic mathematical operations such as matrix multiplication and addition. One important function of TensorFlow is to use the GPU to conveniently implement parallel computing acceleration functions. In order to demonstrate the acceleration feffect of GPU, we can compare mean running time for multiple matirx multiplications on CPU and GPU as follows.

We create two matrices A and B with spahe [1,n] and [n,1], separately. The size of the matrices can be adjusted using parameter n. The code is as follows:

# Create two matriees running on CPU

with tf.device('/cpu:0'):

    cpu_a = tf.random.normal([1,n])

    cpu_b =tf.random.normal([n,1])

    print(cpu_a.device, cpu_b.device)

#Create two matrices running on GPU

with tf.device('/gpu:0'):

    gpu_a = tf.random.normal([1,n])

    gpu_b = tf.random.normal([n,1])

    print(gpu_a.device, gpu_b.device)

Let's implement the ufnctions of the CPU and GPU operations and measuer the computation time of the two functions through the imeit. itmeit() function. It should be noted that additional environment initialization work is generally required for the first calculation, so this time cannot be counted. We remove this time through the warm-up session and then measuer the calculation time as follows:

def cpu_run(): # CPU function

    with tf.device('/cpu:0'):

        c = tf.matmul(cpu_a, cpu_b)

    retun c

def gpu_run(): # GPU function

    with tf.device('/gpu:0'):

        c = tf.matmul(gpu_a, gpu_b)

    return c

#First calcualtion needs warm-up

cpu_time = timeit.timeit(cpu_rn, number=10)

gpu_time = timeit.timeit(gpu_run, number=10)

print('warmup:', cpu_time, gputime)

# Calculate and print mean running time

cpu_time = timeit.timeit(cpu_run, number=10)

gpu_time = timeit.timeit(gpu_run, number=10)

print('run time:', cpu_time, gpu_time)

We plot the computation time under CPU and GPU environments at different matrix sizes as shown in Figure 1-21. It can be seen that when the matrix size is small, the CPU and GPU times are almost the same, which does not reflect the advantages of GPUY parallel computing. When the matrix size is larger, the CPU computing time significantly increases, and the GPU takes full advantage of paralled computing without almost any change of computation time.

b) Automatic Gradient Calculation

When using TensorFlow to construct the forward caculation process, in addition to being able to obtain numberical results, TensorFlow also automatically builds a computational graph. TensorFow provides automatic differentiation that can calculate the derivative of the output on network parameters without manual derivation. Consider the expression of the following function:

y  = aw2 + bw +c

The derivative relationship of the output y th the variable w is 

dy/dw = 2aw +b

Consider the derivative at (a,b,c,w) = (1,2,3,4). We cat get dy/dw = 2*1*4 + 2 = 10

With TensorFlow, we can directly calculate the derivative given the expression of a function without manaully deriving the expression of the derivatives. TensorFlow can automatically derive it. The code is implemented as follows:

import tensorflow as tf

# Create 4 tensors

a = tf.constant(1.)

b = tf.constant(2.)

c = tf.constant(3.)

w = tf.constant(4.)

with tf.GradientTape() as tape:# Track derivative tape.watch([w]) # Add w to derivative watch list

# Design the function

y = a * w w**2 _ b * w + c

# Auto derivative calculation

[dy_dw] = tape.gradient(y, [w])

print(dy_dw) # print the derivative

The result of the program is 

tf.Tensor(10.0, shape=(), dtype=float32)

It con be seen that the result of TensorFlow's automatic differentiation is consistent with the result of manual calculation.

c) Common Neural Network Interface

In addition to the underlying mathematical functions such as matrix multiplication and addition, TensorFlow also has a series of convenient functions for deep learning systems such as commonly used neural network operation functions, commonly used network layers, network training, model saving, loading, and deployment. Using TensorFlow, you can easily use thes functions to complete common production processes, which is efficient and stable.

1.5.2 TnsorFlow 2 and 1.x

 TensorFlow 2 is a completely different fraework from TensorFlow 1.x in terms of user experience. TensorFlow 2 is not compatible with TensorFlow 1.x code. At the same time, it is very different in programming style and functional interface design. TensorFlow 1.x code needs to rely on artificial migration, and automated migration methods are not reliable. Google is about to stop updating TensorFlow 1.x. It is not recommended to learn TensorFlow 1.x now.

TensorFlow 2 supports the dynamic graph priority mode. You can obtain both the computational graph and the numerical results during the calculation. You can debug the code and print the data in real  time. The network is built like a building block, stacked layer by layer, which is in line with software development thinking.

Taking simple addition 2.0 + 4.0 as an example, in TensorFlow 1.x, we need to create a calculation graph first as follows:

import tensorflow as tf

# 1. Create computaition graph with tf 1.x

# Create 2 input variables with fixed name and type

a_ph = tf.placeholder(tf.float32, name='variable_a')

b_ph = tf.placeholder(tf.loat32, name='variable_b')

# Create output operation and name

c_op = tf.add(a_ph, b_ph, name='variable_c')

The process of creating acomputational graph is analogous to the process of establishing a formula c=a_b through symbols. It only records the computational steps of the formula and does not actually caculate the numerical results. The numberical results can only be obtained by running the output c and assigning values a = 2.0 and b = 4.0 as follows:

# 2. Run computational graph with tf 1.x

# Create running environment

sess = tf.InteractiveSession()

#Initialization

init = tf.global_variables_initializer()

sess.run(init) # Run the initialization

# Run the computation graph and return value to c_numpy

c_numpy = sess.run(c_op, feed_dict={a_ph: 2., b_ph: 4.})

#print out the output

print('a+b', c_numpy)

It can be seen that it is so tedious to perform simple addition operations in TensorFlow 1, let alone to create complex neural network algorithms. This programming method of creating a computational graph and then running it later is called symbolic programming.

Next, we use TensorFlow 2 to complete the same operation as follows:

import tensorflow as tf

# Use TensorFlow 2 to run

# 1.Create and initalize variable

a = tf.constant (2.)

b = tf.constant(4.)

# 2. Run and get result directly

print('a_b=', a_b)

As you can see, the calculation process is very simple, and there are not extra calcuation steps.

The method of getting both computation graphs and numerical results at the same time is called jimperative programming, also known as dynamic graph mode. TensorFlow 2 and PyTorch are both developed using dynamic graph priority mode, which is easy to debug. In general, the dynamic graph mode is highly efficient for development, but it may not be as efficient as the static graph mode for running. TensorFlow 2 also supports converting the dynamic graph mode to the static graph mode through tf.function, achieving a win-win situation of both development and operating efficiency. In the remaining part of this book, we use TensorFlow to represent TensorFlow 2 in general.

1.5.1 Major Frameworks

 - Theano is one of the earliest deep learning frameworks. It was developd by Yoshua Bengio and Ian Goodfellow. It is a Python-based ocmputing library for positioning low-level operations. Theano supports both GPU and CPU operations. Due to Theano's low development efficiency, long model compilation time, and developers switching to TensorFlow, Theano ahs now stopped maintenace.

- Scikit-learn is a complete computing library for machine learning algorithms. It has builit-in support for common traditional machine learning algorithms, and it ahs rich documentation and examples. However, scikit-learn is not specifically designed for neural networks. It does not support GPU acceleration, and the implementation of neural network-related layers is also lacking.

- Caffe was developed by Jia Yangqing in 2013. It is mainly used for applications using convolutional neural networks and is not suitable for other types of neural networks. Caffe's main development language is C++, and it also provides interfaces for other languages such as Python. It also supports GPU and CPU. Due to the earlier developement time and higher visibility in the industry, in 2017 Facebook launched an upgraded fversion of Caffe, Caffe2. Caffe2 has now been integrated into the PyTorch library.

- Torch is a very good scientfic computing library, developed based on the less popular programming language Lua, Torch is highly flexible, and it is also an excellent gene inherited by PyTorch. However, due to the small number of Lua language users, Torch has been unable to obtain mainstream applications.

- MXNet was developed by Chen Tianqi and Li Mu and is the official deep learning framework of Amazon. It adopts a mixed method of imperative programming and symbolic programming, which has high flexibility fast running speed, and rich documentation and examples.

-PyTorch is a deep learning framework launched by Facebook based on the original Torch framework using Python as the main development language. PyTorch borrowed the design style of Chainer and adopted imperative programming, which made it very convenient to build and debug the network. Although PyTorch was only released in 2017, due to its sophisticated and compact interface design, PyTorch After the 1.0 version, the original PyTorch and Cafrfe2 were merged to make up for PyTorch's deficiencies in industrial deployment. Overall, PyTorch is an excellent deeop learning framework.

- Keras is a high-level framework implemented based on the underlying operations provided by frameworks such as Theano and TensorFlow. It provides a large number of high-level interfaces for rapid training and testing. For common applications, developing with Keras is very efficient. But because there is no low-level implementation, the underlyuing framework needs to be abstracted, so the operation efficiency isnot high, and the flexibility is average.

- TensorFlow is a deep learning framework released by Google in 2015. The initial version only supported symbolic programming. Thanks to its earlier release and Google's influence in the field of deep learning, TensorFlow quickly became the most popular deep learning framework. However, due to frequent changes in the interface design, redundant functional design, and difficulty in symbolic programming development and debuygging, TensorFlow 1.x was once criticized by the industry. In 2019, Google launched the  official version of TensorFlow 2, which runs in dynamic graph priority mode and can avoid many defects of the TensorFlow 1.x version. TensorFlow 2 has been widely recognized by the industry.

At present, TensorFlow and PyTorch are the two most widley used deep learning frameworks in industry. TensoirFlow has a complete solution and user base in the industry. Thanks to its streamlined and flexible interface design, PyTorch can quickly build and debug entworks, which has received ravee reviews in academia. After TensorFlow 2 was released, it makes it easier for users to learn TensorFlow and seamlessly deploy moduels to production. This book users TensorFlow2 as the main framework to implement deep learning algorithms.

Here are the connections and differences between TensorFlow and Keras.  Keras can be understood as a set of high-level API design specifications. Keras itself has an official implementaion fo thespecifications. The same specifications are also implemented in TensorFlwo, which is called the tf.keras module, and tf.keras will be used as the unique high-level interface to avoid interface redundancy,. Unless otherwise specified, Keras in this book refers to tf.keras.