Upcoming data from the Rubin Observatory offer unprecedented opportunities for astronomical data analysis, but also methodological challenges: how can we automatically detect new (sub)classes of events? Can we detect rare or anomalous events? We investigate here the Rotary Masked Auto-Encoder (RoMAE), a recent development of Transformers to process irregularly-sampled multivariate time-series. Self-supervised pretraining has been shown to significantly improve downstream tasks, suggesting that these models can extract and encode high-level information in their embeddings. However, it remains unclear how (and which) information can be retrieved and disentangled in a setup like Rubin’s data. Thus, we explore the properties of RoMAE’s embeddings in different synthetic scenarios using ELASTiCC.v2 and investigate the ability of the embeddings to identify different types of transients and anomalies. We also tentatively explore the application of our approach on Rubin’s Data Preview 1.