Restricted Boltzmann Machines-Based Image Feature Encoding Techniques and Its Application to Deep Networks

Abstract

Image classification is the process of classifying a given image into one of several distinct classes according to its visual content. One of the main issues in designing a classification system is to develop an appropriate feature encoding method for image representation. This dissertation presents two novel restricted Boltzmann machines (RBMs) designed for the image feature encoding, and how the RBMs can be applied to deep networks such as deep belief network (DBN) and deep convolutional neural network (DCNN) for image classification tasks.

The first designed RBM named as discriminative group-wise Beta-Bernoulli process restricted Boltzmann machine (DG-BBP RBM) is developed by modifying the traditional BBP RBM. This model has the capability to learn class-specific features. Here, the class-specific feature designates the discriminative feature of each class that shares little with those of other classes. For image classification task, a new DBN structure is formed by stacking newly designed DG-BBP RBMs. With this DBN, we can obtain not only a hierarchical set of class-specific mid-level features but also relevant semantic concepts such as parts of an object in upper levels. In experiments on two image classification tasks, our method shows much better results than did the traditional BBP RBM and other related methods and captures semantic attributes that can be used to discriminate between classes.

The second designed RBM named as SVM-driven sparse restricted Boltzmann machines (SVM-SRBM) employs the information obtained from the support vector machine (SVM) for discrimination between classes, and the sparsity modeling for sparseness of features. This RBM is used to customize features extracted from the pre-trained DCNN for target classification tasks, which enables DCNNs to apply to the tasks with limited amount of training data. In this dissertation, the target task is to analyze the photo aesthetics. Experiment results show that our method outperforms the current state-of-the-art methods in automatic photo quality assessment, and also gives reliable ratings of aesthetic attributes that can be useful in photo editing. In addition to the classification tasks for the photo aesthetic analysis, our method is applied to the general object classification task. In this experiment, our method shows much better performance than did the baseline (i.e., directly using features extracted from the pre-trained DCNN) and other related methods for this work. Thus, we demonstrate that this method can be used flexibly for various classification tasks.