Video: https://www.youtube.com/watch?v=7l6fttRJzeU
Slides: https://nips.cc/media/neurips-2021/Slides/21895.pdf
Self-Supervised Learning
– Self-Prediction and Contrastive Learning
What is self-supervised learning and why we need it?
the idea of constructing supervised learning tasks out of unsupervised datasets
Why?
✅ Data labeling is expensive and thus high-quality dataset is limited
✅ Learning good representation makes it easier to transfer useful information to a variety of downstream tasks ⇒ \Rightarrow ⇒ e.g. Few-shot learning / Zero-shot transfer to new tasks
Self-supervised learning tasks are also known as pretext tasks
Video Colorization (Vondrick et al 2018)
a self-supervised learning method
resulting in a rich representation
can be used for video segmentation + unlabelled visual region tracking, without extra fine-tuning
just label the first frame
Zero-shot CLIP (Radford et al. 2021)
Despite of not training on supervised labels
Zero-shot CLIP classifier achieve great performance on challenging image-to-text classification tasks
Precursors 先驱者 to recent self-supervised approaches
Some ideas:
Restricted Boltzmann Machines
Autoencoders
Word2Vec
Autogressive Modeling
Siamese networks
Multiple Instance / Metric Learning
a special case of markov random field
consisting of visible units and hidden units
has connections between any pair across visible and hidden units, but not within each group
you can observe linear substructures in the embedding space where the lines connecting comparable concepts such as the corresponding masculine and feminine words appear in roughly parallel lines
Autoregressive model:
Autoregressive (AR) models are a class of time series models in which the value at a given time step is modeled as a linear function of previous values
NADE: Neural Autogressive Distribution Estimator
Autogressive model also has been a basis for many self-supervised methods such as gpt
Many contrastive self-supervised learning methods use a pair of neural networks and learned from their difference
– this idea can be tracked back to Siamese Networks
if you believe that one network F can well encode x and get a good representation f(x)
then, 对于两个不同的输入x1和x2，their distance can be d(x1,x2) = L(f(x1),f(x2))
the idea of running two identical CNN on two different inputs and then comparing them —— a Siamese network
Train by:
✅ If xi and xj are the same person, ∣ ∣ f ( x i ) − f ( x j ) ||f(xi)-f(xj) ∣∣f(xi)−f(xj) is small
✅ If xi and xj are the different person, ∣ ∣ f ( x i ) − f ( x j ) ||f(xi)-f(xj) ∣∣f(xi)−f(xj) is large
Predecessors of the predetestors of the recent contrastive learning techniques : multiple instance learning and metric learning
deviate frome the typical framework of empirical risk minimization
ealy work:
metric learning:
contrastive Loss:
Triplet loss
N-pair loss:
The part to be predicted pretends to be missing
relationship: can be based on inner logics within data
✅ such as different camera views of the same scene
✅ or create multiple augmented version of the same sample
The multiple samples can be selected from the dataset based on some known logics (e.g., the order of words / sentences), or fabricated by altering the original version
即 we know the true relationship between samples but pretend to not know it
Self-prediction construct prediction tasks within every individual data sample
to predict a part of the data from the rest while pretending we don’t know that part
The following figure: demonstrate how flexible and diverse the options we have for constructing self-prediction learning tasks
✅ can mask any dimensions
分类：
The autoregressive model predicts future behavior based on past behavior
Examples :
mask a random portion of information and pretend it is missiing, irrespective of the natural sequence
e.g.,
Examples :
Some transformation (e.g., segmentation, rotation) of one data samples should maintain the original information of follow the desired innate logic
Examples
Order of image patches
✅ e.g., shuffle the patches
✅ e.g., relative position, jigsaw puzzle
Image rotation
Counting features across patches
Hybrid Self-Prediction Models: Combines different type of generation modeling
VQGAN (Esser & Rombach et al. 2021)
VQ-VAE: to learn a discrete code book of context rich visual parts
A transformer model: trained to auto-aggressively modeling the color combination of this code book
Goal:
To learn such an embedding space in which similar sample pairs stay close to each other while dissimilar ones are far apart
对比学习 can be applied to both supervised and unsupervised settings
Category
Inter-sample classification
