Powered by OpenAIRE graph
Found an issue? Give us feedback
image/svg+xml art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos Open Access logo, converted into svg, designed by PLoS. This version with transparent background. http://commons.wikimedia.org/wiki/File:Open_Access_logo_PLoS_white.svg art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos http://www.plos.org/ ZENODOarrow_drop_down
image/svg+xml art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos Open Access logo, converted into svg, designed by PLoS. This version with transparent background. http://commons.wikimedia.org/wiki/File:Open_Access_logo_PLoS_white.svg art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos http://www.plos.org/
ZENODO
Part of book or chapter of book . 2025
License: CC BY
Data sources: ZENODO
ZENODO
Part of book or chapter of book . 2025
License: CC BY
Data sources: Datacite
ZENODO
Part of book or chapter of book . 2025
License: CC BY
Data sources: Datacite
versions View all 2 versions
addClaim

A Holistic Approach to Transformer-Based Models: Architecture, Applications, and Ethical Layers

Authors: DEMİR, Söz Tuana; GÖKÇE NARİN, Nida;

A Holistic Approach to Transformer-Based Models: Architecture, Applications, and Ethical Layers

Abstract

This section presents a detailed description of transformer-based models, which have revolutionized the field of artificial intelligence, particularly in natural language processing, vision, speech, and multimodal applications, among others. This chapter begins with a description of the limitations of older architectures, such as Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) networks, which were widely used before the advent of transformers, and how transformers addressed these deficiencies. The Vaswani et al. transformer model, introduced in 2017, offers a successful solution to such concerns, particularly through its parallel processing and self-attention mechanisms. The chapter covers the essential components of transformers, such as the multi-headed attention mechanism, positional encoding, residual connections, and the encoder-decoder model. Then it focuses on current leading models, such as BERT, GPT (through GPT-4), DALL·E, Claude 3, Gemini, LLaMA 3, Mistral, Whisper, and Sora. Concerns such as how these systems are powered, where they excel, and the type of output they produce are presented with examples. In addition to the technical data, the social, environmental, and ethical problems for which transformers provide solutions also come into focus. Serious issues, such as artificial intelligence sometimes producing content contrary to reality and not being grounded in reality, i.e., situations referred to as hallucinations; producing skewed results; inducing copyright infringement; posing a threat to personal data; and the detrimental effects of massive models on the environment, are being addressed. For example, a machine learning algorithm would generate a news title reporting accurately on a non-existent incident, demonstrating just how dangerous the threat of hallucinations can be when it comes to critical applications. Here, what is emphasized is that technology professionals must not only be capable of innovating but also behave by values such as transparency, fairness, ethics, and sustainability. Furthermore, the section offers practical advice on which models might be more suitable for other applications, such as creative text writing, classification, multi-modal tasks, or speech content generation. Specific key points that need to be remembered to minimize risks are also conveyed to the readers. Finally, the importance of collaboration among developers, decision-makers, and society to facilitate this growth in a manner that benefits society is highlighted.

Related Organizations
Keywords

Multi-modal tasks, Natural language processing, Ethical artificial intelligence, Generative artificial intelligence, Transformer architecture

  • BIP!
    Impact byBIP!
    selected citations
    These citations are derived from selected sources.
    This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
    0
    popularity
    This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.
    Average
    influence
    This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
    Average
    impulse
    This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.
    Average
Powered by OpenAIRE graph
Found an issue? Give us feedback
selected citations
These citations are derived from selected sources.
This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
BIP!Citations provided by BIP!
popularity
This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.
BIP!Popularity provided by BIP!
influence
This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
BIP!Influence provided by BIP!
impulse
This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.
BIP!Impulse provided by BIP!
0
Average
Average
Average
Green