Powered by OpenAIRE graph
Found an issue? Give us feedback
image/svg+xml art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos Open Access logo, converted into svg, designed by PLoS. This version with transparent background. http://commons.wikimedia.org/wiki/File:Open_Access_logo_PLoS_white.svg art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos http://www.plos.org/ ZENODOarrow_drop_down
image/svg+xml art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos Open Access logo, converted into svg, designed by PLoS. This version with transparent background. http://commons.wikimedia.org/wiki/File:Open_Access_logo_PLoS_white.svg art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos http://www.plos.org/
ZENODO
Other literature type . 2025
License: CC BY
Data sources: ZENODO
image/svg+xml art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos Open Access logo, converted into svg, designed by PLoS. This version with transparent background. http://commons.wikimedia.org/wiki/File:Open_Access_logo_PLoS_white.svg art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos http://www.plos.org/
ZENODO
Other literature type . 2025
License: CC BY
Data sources: ZENODO
image/svg+xml art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos Open Access logo, converted into svg, designed by PLoS. This version with transparent background. http://commons.wikimedia.org/wiki/File:Open_Access_logo_PLoS_white.svg art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos http://www.plos.org/
ZENODO
Other literature type . 2025
License: CC BY
Data sources: ZENODO
image/svg+xml art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos Open Access logo, converted into svg, designed by PLoS. This version with transparent background. http://commons.wikimedia.org/wiki/File:Open_Access_logo_PLoS_white.svg art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos http://www.plos.org/
ZENODO
Other literature type . 2025
License: CC BY
Data sources: ZENODO
image/svg+xml art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos Open Access logo, converted into svg, designed by PLoS. This version with transparent background. http://commons.wikimedia.org/wiki/File:Open_Access_logo_PLoS_white.svg art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos http://www.plos.org/
ZENODO
Other literature type . 2025
License: CC BY
Data sources: ZENODO
ZENODO
Other literature type . 2025
License: CC BY
Data sources: Datacite
ZENODO
Other literature type . 2025
License: CC BY
Data sources: Datacite
ZENODO
Other literature type . 2025
License: CC BY
Data sources: Datacite
ZENODO
Other literature type . 2025
License: CC BY
Data sources: Datacite
ZENODO
Other literature type . 2025
License: CC BY
Data sources: Datacite
ZENODO
Other literature type . 2025
License: CC BY
Data sources: Datacite
versions View all 6 versions
addClaim

This Research product is the result of merged Research products in OpenAIRE.

You have already added 0 works in your ORCID record related to the merged Research product.

Semantic Axis Decomposition of Transformer Embeddings

Authors: Schetnikov, Aleksey Sergeevich;

Semantic Axis Decomposition of Transformer Embeddings

Abstract

This work introduces a novel method for interpreting sentence-transformer embeddings via semantic axis decomposition, capsule layering, and recursive reasoning. Top-N dimensions are selected using Random Forests and assigned human-interpretable meanings such as “emotionality”, “scientificness”, and “question intent”. We go beyond scalar embeddings by stacking multi-layered capsule sub-coordinates and defining recursive semantic floors. Each floor is traversed using two parameters: alpha (influence magnitude) and theta (semantic shift direction), enabling low-cost, interpretable reasoning paths. A new mechanism—floor pruning—discards semantically weak layers to retain only impactful shifts. Our proposed structure was validated in the Capsule AI Evolution experiment, achieving a +4.45% accuracy gain over baseline without fine-tuning the transformer. This confirms that semantic floor traversal enables dynamic, modular reasoning and performance improvement through capsule-space navigation alone. This is a conceptual and visual demonstration. Code and GUI tools are released separately. Keywords: transformer, embeddings, interpretability, semantic floors, latent space, capsule layers, XAI, sentence-transformers, recursive reasoning Interactive prototype:https://github.com/kexi-bq/embedding-explainer Experiment and evaluation code:https://github.com/kexi-bq/capsule-ai-evolution Capsule Selection In this framework, capsule selection (e.g., dim_5.3) is not fixed in advance but is discovered through an evolutionary optimization process driven by Capsule AI.The system applies random shifts along selected capsule dimensions and evaluates their impact on downstream task performance (e.g., classification accuracy).If a particular capsule configuration improves results, it is rewarded and retained; otherwise, it is discarded. Through multiple iterations, Capsule AI gradually identifies and accumulates effective capsule directions — allowing the embedding space to be optimized without fine-tuning the transformer model itself. Post-Publication Experiment: Capsule Reactivation of Neutral Dimensions We conducted a follow-up experiment targeting neutral embedding dimensions — those that showed <0.001 change in classification accuracy when zeroed out. Task:Multi-class classification of 150 text phrases into 4 categories:emotional, scientific, neutral, question(see: Semantic_Test_Dataset.csv) Baseline setup: Embeddings: all-MiniLM-L6-v2 Classifier: LogisticRegression Train/test split: 70/30 Procedure: Identified 383 “neutral” dims using accuracy-preserving zeroing Applied Capsule Shift (dim += alpha) individually to each dim in test data Measured accuracy impact per shift Results: 87 out of 383 dims (22.7%) produced accuracy improvements Best delta: +2.22% (from 93.33% to 95.56%) This confirms that many “dead” dimensions are semantically reactivatable through capsule-based manipulation — even without retraining the model. For correspondence: Aleksey Schetnikov Email: alex21259alex@gmail.com Telegram: @Alex_larinov

Keywords

transformer, embeddings, interpretability, XAI, AI

  • BIP!
    Impact byBIP!
    citations
    This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
    0
    popularity
    This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.
    Average
    influence
    This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
    Average
    impulse
    This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.
    Average
Powered by OpenAIRE graph
Found an issue? Give us feedback
citations
This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
BIP!Citations provided by BIP!
popularity
This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.
BIP!Popularity provided by BIP!
influence
This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
BIP!Influence provided by BIP!
impulse
This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.
BIP!Impulse provided by BIP!
0
Average
Average
Average
Green