The MNIST database

 In developing the DBN model, we will use a dataset that we have discussed before - the MNIST database, which contains digital images of hand-drawn digits from 0 to 9. This database is combination of two sets of earlier images from the National Institute of Standards and Technology(NIST): Special Database 1(digits written by US high school students) and Special Database 3(written by US Census Bureau employees), the sum of which is split into 60,000 training images and 10,000 test images.

The original images in the dataset were all block and white, while the modified dataset normalized them to fit into a 20*20-pixel bounding box and removed jagged edges using anti-aliasing, leading to intermediary grayscale values in cleaned images; they are padded for a final resolution of 28*28 pixels.

In the original NIST dataset, all the training images came from bureau employees, while the test dataset came from high school students, and the modified version mixes the two groups in the training and test sets to provide a less biased population for training machine learning algorithms.

An early application of Support Vector Machines(SMVs) to this dataset yielded an error rate of 0.8%, while the latest deep learning models have shown error rates as low as 0.23%. You should note that these figures were obtained due to not only the discrimination algorithms used but also "data augmentation" tricks such as creating additional translated images where the digit has been shifted by several pixels, thus increasing the number of data examples for the algorithm to learn from. Because of its wide availability, this dataset has become a benchmark for many machine learning models, including Deep Neural Networks.

The dataset was also the benchmark for a breakthrough in training multi-layer neural networks in 2006, in which an error rate of 1.25% was achieved(without image translation, as in the preceding examples). In this chapter, we will example in detail how this breakthrough was achieved using a generative model, and explore how to build our own DBN that can generate MNIST digits.