ESNLIR: Expanding Spanish NLI Benchmarks with Multi-Genre and Causal Annotation This is the complete code, model and datasets for the preprint ESNLIR: A Spanish Multi-Genre Dataset with Causal Relationships. THE FINAL PAPER WILL in a ICAI by Springer CCIS volume of 2025. Installation This repository is a poetry project, which means that it can be installed easily by executing the following command from a shell in the repository folder: poetry install As this repository is script based, the README.md file contains all the commands executed to generate the dataset and train models. ---------------------------------------------------------------------------------------------- Core code The core code used for all the experiments is in the folder auto-nli and all the calls to the core code with the parameters requested are found in README.md ---------------------------------------------------------------------------------------------- Parameters All the parameters to create datasets and train models with the core code are found in the folder parameters. ---------------------------------------------------------------------------------------------- Models Model types For BERT based models, all in pytorch, there are two types of models from huggingfaces that were used for training and also are required to load a dataset because of the tokenizer: RoBERTa (BERTIN): https://huggingface.co/bertin-project/bertin-roberta-base-spanish XLMRoBERTa: https://huggingface.co/FacebookAI/xlm-roberta-base Model folder The model folder contains all the trained models for the paper. There are three types of models: baseline: An XGBoost model that can be loaded with pickle. roberta: BERTIN based models in pytorch. You can load them with the model_path /model/roberta/model xlmroberta: XLMRoBERTa based models in pytorch. You can load them with the model_path /model/xlmroberta/model Models with the suffix _annot are models trained with the premise (first sentence) only. Apart from the pytorch model folder, each model result folder (ex: /model/xlmroberta/) contains the test results for the test set and the stress test sets (ex: /model/xlmroberta/test) Load model Models are found in the folder model and all of them are pytorch models which can be loaded with the huggingface interface: from transformers import AutoModel model = AutoModel.from_pretrained('',local_files_only=True) ---------------------------------------------------------------------------------------------- Dataset labeled_final_dataset.jsonl This file is included outside the ZIP containing all other files, and it contains the final test dataset with 974 examples selected by human majority label matching the original linking phrase label. Other datasets: The datasets can be found in the folder data that is divided in the following folders: base_dataset The splits to train, validate and test the models. splits_data Splits of train-val-test extracted for each corpora. They are used to generate base_dataset. sentence_data Pairs of sentences found in each corpus. They are used to generate splits_data. Dataset dictionary This repository contains the splits that resulted from the research project "ESNLIR: A Spanish Multi-Genre Dataset with Causal Relationships". All the splits are in JSONL format and have the same fields per example: sentence_1: First sentence of the pair. sentence_2: Second sentence of the pair. connector: Linking phrase used to extract pair. connector_type: NLI label, between "contrasting", "entailment", "reasoning" or "neutral" extraction_strategy: "linking_phrase" for "contrasting", "entailment", "reasoning" and "none" for neutral. distance: How many sentences before the connector is the sentence_1 sentence_1_position: Number of sentence for sentence_1 in the source document sentence_1_paragraph: Number of paragraph for sentence_1 in the source document sentence_2_position: Number of sentence for sentence_2 in the source document sentence_2_paragraph: Number of paragraph for sentence_2 in the source document id: Unique identifier for the example dataset: Source corpus of the pair. Metadata of corpus, including source can be found in dataset_metadata.xlsx. genre: Writing genre of the dataset. domain: Domain genre of the dataset. Example: {"sentence_1":"sefior Bcajavides no es moderado, tampoco lo convertirse e\u00f1 declarada divergencia de miras polileido en griego","sentence_2":"era mayor claricomentarios, as\u00ed de los peri\u00f3dicos como de los homes dado \u00e1 la voluntad de los hombres, sin que sobreticas","connector":"por consiguiente,","connector_type":"reasoning","extraction_strategy":"linking_phrase","distance":1.0,"sentence_1_paragraph":4,"sentence_1_position":86,"sentence_2_paragraph":4,"sentence_2_position":87,"id":"esnews__spanish_pd_news__531537","dataset":"esnews__spanish_pd_news","genre":"news","domain":"spanish_public_domain_news"} Dataset load To load a dataset/split as a pytorch object used to train-validate-test models you must use the custom class dataset from auto_nli.model.bert_based.dataset import BERTDataset dataset = BERTDataset( os.path.join(dataset_folder, ), max_len=, model_type= , only_premise=, max_samples=) ---------------------------------------------------------------------------------------------- Notebooks The folder notebooks contains a collection of jupyter notebooks used to preprocess datasets and visualize results.