Master's Thesis Presentation
“Self-Supervised Learning: Pre-Training with Denoising Autoencoders Using Layer-wise Learning”
Misha Sohan
Tuesday, July 25, 2023, at 9:00 AM
Zoom Meeting
Abstract
Many deep learning methods are dependent on access to large quantities of well- labeled data. Though the amount of data generated worldwide has greatly increased over the years, labeling remains a major bottleneck. In recent years, several unsupervised learning methods have been introduced to target this issue. We implement a self-supervised computer vision representation model for image classification. Specifically, we first pre-train using a denoising autoencoder with a siamese structure, where two different versions of each image are generated via different augmentations. This pre-training generates a low-dimensional representation of the images. In particular, in the pre-training, we examine the question of whether the autoencoder layers in a deep network can be learned layer by layer sequentially and still maintain the high performance of a network where layers are learned jointly. Shallow networks are easier to interpret and optimize compared to deep networks but do not perform as well. Thus, building a deep network layer by layer could be beneficial in combining the benefits of both shallow and deep networks. We evaluate the performance of the representations learned via pre-training with a downstream image classification task that uses a standard linear evaluation procedure. We achieve comparably good performance with both joint and layer-wise training, with accuracy in the 69% to 72% range using the CIFAR-10 training and test datasets.
