Powered by OpenAIRE graph
Found an issue? Give us feedback
image/svg+xml Jakob Voss, based on art designer at PLoS, modified by Wikipedia users Nina and Beao Closed Access logo, derived from PLoS Open Access logo. This version with transparent background. http://commons.wikimedia.org/wiki/File:Closed_Access_logo_transparent.svg Jakob Voss, based on art designer at PLoS, modified by Wikipedia users Nina and Beao ZENODOarrow_drop_down
image/svg+xml Jakob Voss, based on art designer at PLoS, modified by Wikipedia users Nina and Beao Closed Access logo, derived from PLoS Open Access logo. This version with transparent background. http://commons.wikimedia.org/wiki/File:Closed_Access_logo_transparent.svg Jakob Voss, based on art designer at PLoS, modified by Wikipedia users Nina and Beao
ZENODO
Article . 2022
License: CC BY
Data sources: ZENODO
ZENODO
Article . 2022
License: CC BY
Data sources: Datacite
ZENODO
Article . 2022
License: CC BY
Data sources: Datacite
versions View all 2 versions
addClaim

Emerging Properties in Self-Supervised Vision Transformers for ML.

Authors: Pushpa G; Smithu B S;

Emerging Properties in Self-Supervised Vision Transformers for ML.

Abstract

The convergence of Vision Transformers (ViTs) and self-supervised learning (SSL) has catalyzed a new paradigm in computer vision. While supervised ViTs often require massive, hand-labeled datasets, their self-supervised counterparts, particularly models trained with methods like DINO or MAE, have demonstrated an uncanny ability to learn rich, semantically meaningful representations from unlabeled images. This article surveys the remarkable "emerging properties" of these models, which go beyond simple image classification accuracy. We delve into phenomena such as the emergence of explicit scene segmentation within attention maps, the discovery of object boundaries, and the creation of highly transferable, linear-separable feature spaces. We analyze the mechanisms that give rise to these properties, including self-distillation and contrastive learning, and discuss their profound implications for both research and application in machine learning, particularly in data-scarce domains.

Keywords

Self-Supervised Learning, Computer Vision, DINO, Vision Transformer

  • BIP!
    Impact byBIP!
    selected citations
    These citations are derived from selected sources.
    This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
    0
    popularity
    This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.
    Average
    influence
    This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
    Average
    impulse
    This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.
    Average
Powered by OpenAIRE graph
Found an issue? Give us feedback
selected citations
These citations are derived from selected sources.
This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
BIP!Citations provided by BIP!
popularity
This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.
BIP!Popularity provided by BIP!
influence
This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
BIP!Influence provided by BIP!
impulse
This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.
BIP!Impulse provided by BIP!
0
Average
Average
Average
Upload OA version
Are you the author of this publication? Upload your Open Access version to Zenodo!
It’s fast and easy, just two clicks!