Powered by OpenAIRE graph
Found an issue? Give us feedback
ZENODOarrow_drop_down
ZENODO
Dataset . 2024
License: CC BY
Data sources: Datacite
ZENODO
Dataset . 2024
License: CC BY
Data sources: Datacite
versions View all 2 versions
addClaim

This Research product is the result of merged Research products in OpenAIRE.

You have already added 0 works in your ORCID record related to the merged Research product.

Code and data for "ActivityGen: Extracting Enabled Activities from Screenshots"

Authors: Manuel, Sovin; Beyel, Harry H.; van der Aalst, Wil M. P.;

Code and data for "ActivityGen: Extracting Enabled Activities from Screenshots"

Abstract

The code and data for the paper "ActivityGen: Extracting Enabled Activities from Screenshots" is provided here. Abstract Many tasks in organizations are performed in a desktop environment. It is possible to record users' interactions in a desktop environment by taking screenshots when an action happens. The result is an interaction log. By considering the associated images of a record, it is possible to detect which activity was performed and which activities were enabled. This information can be extracted, resulting in a translucent event log. Such a translucent event log is valuable and can be used as input for dedicated process-mining techniques. The results can be used to analyze human-computer interactions or create bots for robotic process automation. However, current techniques for extracting information on enabled activities rely on template matching, which is rigid and sensitive to variations. To solve this issue, we present our modular framework, ActivityGen. ActivityGen detects and labels graphical user interface elements by also considering additional information. ActivityGen uses more advanced techniques to overcome the limitations of previous approaches and can extract information without a user's input. Furthermore, it can be adjusted to a user's needs. It detects graphical user interface elements more accurately than state-of-the-art techniques and labels them faster, more robust, and more domain-oriented than state-of-the-art techniques. Data ReDraw_CLS and ReDraw_ViSM are specified in the work. The basis for ReDraw_CLS is the ReDraw dataset. We focus on the following components: Button, CheckBox, EditText, Image, ImageButton (which we refer to as icon), RadioButton, and Switch. We noticed that the examples of ImageView and ImageButton are similar, primarily consisting of icon images. Therefore, we removed the ImageView class and introduced an Image class instead. The Image class contains images from the validation set of the Coco validation set 2017 and the YouTube Thumbnails dataset, enabling the detection of general website images. ReDraw_ViSM iterates add 6,000 synthetically created buttons to the former dataset by distributing them in the same ratio into train, test, and validation sets. lm_basic and lm_extended contain the text training for the language models. Code The code allows for the execution of ActivityGen. Moreover, we provide our evaluation scripts. However, the models do not have to be trained. The models' weights are provided in the model folder.

Related Organizations
  • BIP!
    Impact byBIP!
    citations
    This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
    0
    popularity
    This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.
    Average
    influence
    This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
    Average
    impulse
    This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.
    Average
Powered by OpenAIRE graph
Found an issue? Give us feedback
citations
This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
BIP!Citations provided by BIP!
popularity
This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.
BIP!Popularity provided by BIP!
influence
This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
BIP!Influence provided by BIP!
impulse
This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.
BIP!Impulse provided by BIP!
0
Average
Average
Average