• Research softwareclose
• Other research productsclose
• Other ORP type close
• DE close
• Digital Humanities and Cultural Heritage close

Finding and amending contradictions in a financial report is crucial for the publishing company and its financial auditors. To automate this process, we introduce a novel approach that incorporates informed pre-training into its transformer-based architecture to infuse this model with additional Part-Of-Speech knowledge. Furthermore, we fine-tune the model on the public Stanford Natural Language Inference Corpus and our proprietary financial contradiction dataset. It achieves an exceptional contradiction detection F1 score of 89.55% on our real-world financial contradiction dataset, beating our several baselines by a considerable margin. During the model selection process we also test various financial-document-specific transformer models and find that they underperform the more general embedding approaches.

In computer science, robustness can be thought of as the ability of a system to handle erroneous or nonstandard input during execution. This thesis studies the robustness of the methods that extract structured information from unstructured documents containing human language texts. Unfortunately, these methods usually suffer from various problems that prevent achieving robustness to the nonstandard inputs encountered during system execution in real-world scenarios. Throughout the thesis, the key components of the information extraction workflow are analyzed and several novel techniques and enhancements that lead to improved robustness of this process are presented. Firstly, a deep learning-based text recognition method, which can be trained almost exclusively using synthetically generated documents, and a novel data augmentation technique, which improves the accuracy of text recognition on low-quality documents, are presented. Moreover, a novel noise-aware training method that encourages neural network models to build a noise-resistant latent representation of the input is introduced. This approach is shown to improve the accuracy of sequence labeling performed on misrecognized and mistyped text. Further improvements in robustness are achieved by applying noisy language modeling to learn a meaningful representation of misrecognized and mistyped natural language tokens. Furthermore, for the restoration of structural information from documents, a holistic table extraction system is presented. It exhibits high recognition accuracy in a scenario, where raw documents are used as input and the target information is contained in tables. Finally, this thesis introduces a novel evaluation method of the table recognition process that works in a scenario, where the exact location of table objects on a page is not available in the ground-truth annotations. Experimental results are presented on optical character recognition, named entity recognition, part-of-speech tagging, syntactic chunking, table recognition, and interpretation, demonstrating the advantages and the utility of the presented approaches. Moreover, the code and the resources from most of the experiments have been made publicly available to facilitate future research on improving the robustness of information extraction systems.

While most approaches individually exploit unstructured data from the biomedical literature or structured data from biomedical knowledge graphs, their union can better exploit the advantages of such approaches, ultimately improving representations of biology. Using multimodal transformers for such purposes can improve performance on context dependent classification tasks, as demonstrated by our previous model, the Sophisticated Transformer Trained on Biomedical Text and Knowledge Graphs (STonKGs). In this work, we introduce ProtSTonKGs, a transformer aimed at learning all-encompassing representations of protein-protein interactions. ProtSTonKGs presents an extension to our previous work by adding textual protein descriptions and amino acid sequences (i.e., structural information) to the text- and knowledge graph-based input sequence used in STonKGs. We benchmark ProtSTonKGs against STonKGs, resulting in improved F1 scores by up to 0.066 (i.e., from 0.204 to 0.270) in several tasks such as predicting protein interactions in several contexts. Our work demonstrates how multimodal transformers can be used to integrate heterogeneous sources of information, paving the foundation for future approaches that use multiple modalities for biomedical applications.

Emerging modern technologies including Mixed Reality (MR) are gradually becoming an inherent component of higher education. Studies concerning the effectiveness of their use as a didactic tool have been conducted in several fields of research. While the benefits of spatial representation through MR have already been employed in the fields of cultural heritage and museums, hardly any research is dedicated to higher archaeological education. In this paper, we present the first results of the ongoing development of the MARBLE-App of the research programme »Mixed and Augmented Reality in Blended Learning Environments« on the integration of MR into everyday archaeological higher education. The results are presented on the basis of learners' feedback from a focus group study carried out with archaeology students on understanding the spatial complexity of an archaeological structure. They show an increase in the students' impression, their self-reflected learning gains and their engagement towards spatial-related topics in the field of archaeology.That said, we perceive a high potential yet to be explored for the use of this technology in archaeological spatial education.

Verifying the identity of a person (sitter) portrayed in a historical painting is often a challenging but critical task in art historian research. In many cases, this information has been lost due to time or other circumstances and today there are only speculations of art historians about which person it could be. Art historians often use subjective factors for this purpose and then infer from the identity information about the person depicted in terms of his or her life, status, and era. On the other hand, automated face recognition has achieved a high level of accuracy, especially on photographs, and considers objective factors to determine the identity or verify a suspected identity. The limited amount of data, as well as the domain-specific challenges, make the use of automated face recognition methods in the domain of historic paintings difficult. We propose a specialized, likelihood-based fusion method to enable deep learning-based face recognition on historic portrait paintings. We additionally propose a method to accurately determine the confidence of the made decision to assist art historians in their research. For this purpose, we used a model trained on common photographs and adapted it to the domain of historical paintings through transfer learning. By using an underlying challenge dataset, we compute the likelihood for the assumed identity against reference images of the identity and fuse them to utilize as much information as possible. From these results of the likelihoods fusion, we then derive decision confidence to make statements to determine the certainty of the model’s decision. The experiments were carried out in a leave-one-out evaluation scenario on our created database, the largest authentic database of historic portrait paintings to date, consisting of over 760 portrait paintings of 210 different sitters by over 250 different artists. The experiments demonstrated, that a) the proposed approach outperforms pure face recognition solutions, b) the fusion approach effectively combines the sitter information towards a higher verification accuracy, and c) the proposed confidence estimation approach is highly successful in capturing the estimated accuracy of the decision. The meta-information of the used historic face images can be found at https://github.com/marcohuber/HistoricalFaces.

Artwork and objects of cultural heritage undergo unpreventable stress due to environmental conditions or other external influences over their often centuries-long lifetime. Such influences may leave the optically visible parts of the objects intact, while invisible damages can develop invisible under the surface and endanger the structural integrity of irreplaceable works of art. One example are frescos and mural paintings, where the mortar-based preparation layers on walls are prone to developing cracks due to constant mechanical stress on the building's structure. Non-destructive testing (NDT) by terahertz measurements can help reveal such defects and yield valuable information of the layer structure of the paintings to support the work of art historians in restoration and conservation. In this contribution, we present structural investigations, which we performed on Leonardo da Vinci's famous mural painting "The Last Supper" using terahertz FMCW and TDS measurements techniques.

Modern data and information systems usually contain considerable amounts of data and documents and thus provide a large amount of information. The automatic extraction of domain-specific information is all the more important in order to improve work with such systems. If information is available as free text information, machine processing can prove to be a difficult technical hurdle. State-of-the-art approaches use modern Natural Language Processing (NLP) methods to solve such tasks. In this paper, we want to introduce a data-driven approach, applying an XML data model to an application-specific scenario, using different NLP methods, which are combined into a multidimensional pipeline. It is important to understand how certain NLP methods can be used and what their limitations are. Individual modern NLP methods are often not sufficient and resilient enough to solve complex information extraction tasks. Therefore, it has to be examined how such problems can be alleviated or circumvented by a combination of different NLP methods. As a distinction to categorical grammar models, all cases considered here should be available as free text. The approach presented in this paper is still a work in progress, yet first evaluation results will be given.

As part of the unprecedented wealth of data available nowadays, semi-formal reports in the domain of remote sensing can convey information important for decision making in structured and unstructured text parts. For such reports, often kept in large data management systems, targeted information retrieval remains difficult, e.g., the extraction of texts parts relevant to a question posed via natural language. The work presented in this paper therefore aims at finding the relevant documents in data management systems and extracting their relevant content parts based on natural language questions. For this purpose, an approach for semantic information retrieval based on Abstract Meaning Representation (AMR) is adapted, extended and evaluated for the considered domain of remote sensing and image exploitation. In detail, two different metrics used in AMR, Smatch and SemBleu, are compared for their suitability in an AMR-based search. The first results presented in this paper are promising. In addition, more detailed experiments regarding the performance of the metrics under differently formulated yet semantically equivalent questions reveal interesting insights into their ability for semantic comparison.

Since their invention during the renaissance, copperplate engravings have been an important art form and a useful tool to create reproducible images. They appear stylistically similar to pen-and-ink drawings, but are cut into copper plates and printed instead of drawn directly onto the paper. This cutting process requires much effort and a high degree of skill, but results in pictures with extremely clean, precise lines. These long, smoothly curved lines are used to fill the page with many shades of hatching and set engravings apart from other line-based art forms. Stylized Rendering is a field of computer graphics that aims to render 3D scenes in a variety of non-photorealistic styles. While there are a number of works about creating images with line drawing and hatching, the specifics of the engravings style pose a unique challenge to stylized rendering. This is because the very long, smooth, and locally parallel hatching lines that define the engraving style are a poor match for previous methods. These have often been focused on the types of hatching found in pen-and-ink or pencil drawings, where lines are much shorter and have less variation. With this thesis, I present a novel approach to recreate the style of copperplate engravings in 3D rendering. It is especially focused on faithfully recreating the specific hatching style of engravings and applying it to dynamic scenes in a temporally coherent manner. This is achieved by creating and rendering individual hatching strokes that are continually updated to conform to the motion of the scene. My contribution lies in a new scheme for updating the shape and connectivity of hatching lines to ensure temporal coherence. This scheme adapts a number of steps from a previous approach for temporal continuity of contour lines and changes them to be applied to hatching.

Asset Administration Shells (AAS) have emerged as a main concept to implement the Industry 4.0 paradigm. As part of a series of projects targeted at small and medium enterprises (SMEs), we implemented a system where the AAS of a machine can communicate via instant messenger with staff e.g. in maintenance, when certain process parameters are out of bounds. The staff can react to the alarm and request additional information in near-natural language from the company’s document archive. The system uses the BaSyx middleware, Matrix Instant Messaging Protocol, and the Maubot framework. This paper contributes to the further development of Industry 4.0 solutions for SMEs by reporting on the implementation efforts of AAS’s in a novel conjunction with a communication platform that offers additional services, therefor pushing for improvements and new solutions.